But I assumed it was because the bay has too much fresh water up as far as Blue Hill?
For whatever reason, you're right. The buoys thin out an hour or so above Jericho Bay, then disappear all together a few miles short of Blue Hill Harbor. We usually find the inner harbor is warm enough to swim comfortably. But despite this being one hot summer, the water was a little cool up there this season. Mid 60's as opposed to the 70 degrees we found last year.