We are only temporary conservators of the stamps in our collections. We are not mighty and noble institutions like the National Postal Museum in Washington DC or the Spellman Museum of Stamps and Postal History. Those institutions will be standing long after we are gone, their collections intact and there for many generations to enjoy.
But as an individual you must give thought to what later generations will make of your collection. Will they look upon it (as long as it is left intact; if you can't get your children interested, that might be a vanishingly brief period) and think that the collector who put it together did so with intelligence and care? Or will they be looking at a collection -- perhaps little more than an accumulation -- of a casual weekend collector?
You can say "What will it matter if I am gone?" That may be true, but every knowledgeable stamp collector knows that another collector will eventually benefit - or not - from his legacy. Whether it is a positive or negative one is up to you.