If you went to Cuttyhunk on any September or June week day, it would be nearly deserted. These businesses in places line MV, Cape Cod, Nantucket or BI that seem to have amazing money flowing in July/August really need to cover the whole year's costs in 2 months.
A restaurant only has so many tables, and can only turn them over so many times per day/night. Food prep and service is fairly labor intensive, top chefs are pricey, and you can only charge a limited premium for location.