The channel is well marked.

You'll want to stop in Essex -- great restaurants and nice shops, including some art and antiques. The Sailor's Supply store (hard to find, although you'll see the sign out by the street) shares space with the ice cream shop. I recommend you take a mooring for $32.50 at Brewer Dauntless... they have a launch service, but it only runs until 6 so you might want to launch your dinghy if you plan dinner or a late shower ashore. Definitely visit the CT River Museum in Essex -- $8.50 admission and a LOT of history on display. Very interesting and educational, especially if you take the time to listen to the recordings by each exhibit.

I advise against anchoring in the river overnight or leaving your boat at anchor, due to reversing currents. The opposing wind and current can do some strange things to your boat.

Image below is looking north, between Old Lyme and Essex.