When I was boat shopping for my first boat (and only boat so far) I was given some advice (from a person not involved in the sale) - Buy the boat you want and can afford now. Don't buy the 28ft 'starter' boat, because you will only outgrow it and then go through the hassle of selling it and buying another. So I ended up buying a 38' Ericson.
After a year or so I had to contact the selling broker for some paperwork. He asked how I liked the boat. I told him I liked it a lot, but if I had it do do over I probably would have bought a smaller boat, probably a 32' or so. (I singlehand a lot).
His face slumped and he told me that he always told his customers that he had NEVER had anyone come back and say they wished they had bought a smaller boat, and that I just ruined that for him...
Now, after 13 years, I am very happy with the 38. But I probably would still be happy with a 32. It's being out there that counts, not the boat you do it in.