Chris -- SG's suggestion about Sag Harbor is a good one, and if you go there I'd encourage you to explore that entire area around Shelter Island. Gardeners Bay has delightful day sailing and it's no more than a few hours between 3-Mile Harbor, Coecles Harbor, Greenport, Orient Harbor, Dehring Harbor, West Neck Harbor, Noyack Bay, etc.

I've anchored in all of them and they each have their appeal. You could spend a week or two exploring that area. It can really kick up a bad chop around Gardiners Island if the wind is from the west.