You should allow time for the Gardiner's Bay area. Sag harbor, 3 Mile harbor, Greenport, and the three harbors on Shelter Island are all worth a stop. Orient Harbor too, is beautiful and quiet in settled conditions and a good place to kayak or explore the shallows with your dinghy, but the bottom is largely made of cobblestones and course gravel so it's not good holding.