It would probably be a mistake not to anchor in there.

As for the island. The main town is "boardwalky". The ferries generally come into the "old harbor" (east side of the island). T-Shirt shops, etc. The bars and restaurants off of the Salt Pond tended to be places you went-to when you were is a "college bear hall" frame of mind. The houses on other portions of the island are pretty new.

The island has a lot of history from the "colonisation" period. It was a Tory stronghold, pretty independant, the land was owned by a couple or several families for the most part until only a few years ago. It's disappearing as Long Island Sound and the ocean claims its soft basis.

We got a LOCAL cab driver who had lived on the island for a lot years to take us on a tour of the island. Maybe less than an hour and a half. He gave us some of the history of the island. It was about $25 about 10 years ago. (It's the short season now; and, I bet they'd love to take you around for the equivalent of a couple of normal fares plus a decent tip.)

Anyway, enjoy the trip. You're going in the best time of the year.