The food was OK but the service was awful and prices were as you say... Well, of course you pay for the "overhead" and view but it's not a place we'd return to.
Someone told us we would be walking distance to town from the southern-most marina, so we landed our dinghy there and walked quite a way without seeing anything that indicated we were getting to a town, so we turned around after having a good walk. So to sum up, our experience there was that there didn't seem to be much ashore. However it is a nice harbor and relatively undeveloped, and would be a nice stop when looking for someplace to avoid the crush elsewhere on a holiday weekend.
OTOH, Sag Harbor has lots of shore-side action right at the dinghy dock and is where we provision in that area (or Greenport). Sag Harbor is a very boater-friendly town.