We had a really big vessel. It was only the two of us with 50' on deck and 40,000 lbs. I found it no more difficult handling the larger boat than I did my 36 foot boat. Everything was larger but their wasn't any more to deal with except when we needed a slip. When cruising and hanging out at anchor there was no difference between the two. What was different was the motion of the boat. 42 feet of waterline and 40,000 lbs made a tremendous difference in comfort from a 29' waterline and 16,000 lbs. Yes we had more room to stow stuff and never had cabin fever as a plus but the motion of the boat was the big difference.
I go on a smaller vessel with less weight and a shorter waterline and the motion is erratic. More weight and waterline makes for a more comfortable ride. Not only a more comfortable ride but a safer ride. Going to the bow on a small boat in a pitching sea is dangerous. In the same sea on our larger vessel the smoother motion is just safer for us old farts. In Dashews 'Surviving The Storm' from past survival stories given that two vessels are of the same quality build larger is safer.
I do admit that I've gained more appreciation of smaller vessels in a 'traveling nest' mentality but as you said as you get older you want more comfort. The motion of a bigger boat is just more comfortable and safer.