the antenna is pretty good, on top of the mizzen puts it nearly 40 feet above the water. But we used the smaller coax for the lead, and there may be something suspicious about the connections. How important is it to use the good low loss coax in this application?

Since this receiver is no longer supported, I have been thinking about getting the Raymarine Class B transceiver. That could use the masthead VHF antenna which has low loss cable, but would have to share it with VHF through the internal Raymarine mux. Going back and stringing the large RG8U through the mizzen would be a lengthy chore. I think it has RG58U in there, 4.9db/100 ft vs. 2.2 for the RG8U (@100MHz)