The colder water also coincides with bigger surf and poor visibility. So I don't venture out swimming into the ocean at that time of year. My ocean swim season pretty much coincides with the time changes... within about a week or two.
Now on the flip side, we do get some rare incredible clear calm days sometimes in the "dead" of winter. Then I slip on a thin 3 mil long wet suit, thick hood, gloves and fins and go snorkel diving (or "free diving"). I go bare foot, but the rest of me is well covered. (the bare foot deal has to do with the fins I prefer)
I don't focus on the activity as a swim, but as a dive. I have taken some beautiful pictures on such clear cold days.
I do have a swimming wet suit... it has farmer john type arm openings and the arm holes are sewn and sealed. I don't much like it though, as it changes my buoyancy and feels odd, it gives me more of a head down position as there is more suit covering the lower part of my body than the upper. I bought it years ago in an attempt to extend my season, and I never really enjoyed it.
Usually when I restart my season in the spring, I wear a thin rash guard and a hoodie and some short fins, as the water does take a bit of getting used to, at least for the first couple of weeks. But in late autumn, I push the season as long as I can.
The thing about ocean swimming is that one can get into a great rhythm and hold it "forever," and there is hardly the aspect of running into another swimmer as there is in a pool (although I have... oddly enough). Plus you set your own pace, whereas in a pool, you have to match the pace of the circle or side you are swimming... and the darn flip turns... oy.
In the ocean, there is also the beauty of watching other critters in the water, and the feeling of the motion of the ocean... the latter part I am sure is something to which sailors can totally relate.