My former home port is Sag Harbor, one town over. I have spend many a lovely evening in Three Mile Harbor. There is no problem getting in and out (unless you do it at low time in a Swan that draws 8+') and it is largely empty. Dinner at East Hampton Point, the closest restaurant you can dinghy to, is pricy, but the food is excellent and the people are beautiful...it is the Hamptons after all.