It's for the sport-fishermen and gold-chain crowd, and the harbor lacks charm. Go instead into Gardeners Bay (always beautiful sailing in that bay) and visit any of the beautiful and quiet harbors on Shelter Island (each with their own flavor), or go to Greenport and Sag Harbor if you wants restaurants and provisioning. You'll have a much different impression.

Three Mile Harbor is also a pleasant anchorage but not as nice as Shelter Island's 3 choices -- and 3-mile isn't in walking distance to anything, but you can dinghy to the marinas which have overpriced restaurants where diners pay extra for the view you get from your own cockpit.

Greenport is more down-to-earth and Sag Harbor is where the super yachts and movie stars go, and a "beater" is a Beemer that's more than 2 years old. Both are very boater-friendly towns with excellent shopping and a wide variety of restaurants a few steps from the docks. Good dining also in Dering Harbor (on Shelter Island) at Sweet Tomato's.