I guess over the years I've tried about every approach. I often will do it without masking, but I'm pretty sloppy at times and there are a few tight spots that I'm sure to slop up. Often I don't notice it for six months until it starts turning yellow. Also, I often have a few bad spots that need to go down to bare wood and get lots of coats. These I mask off or after a couple of coats I loose track of where they were. (of course, after a couple of coats, I've lost track of how many coats I've done)
I bought a roll of frog tape to give a try. It said it was good for 21 day removal and I thought that was good, but when I got home and read the fine print, it said 7 days in the sun. It doesn't sound any better than good old 3M Scotch Blue.
I also got a roll of 3M 2080 to try, but It seems like I've tried that in the past and it didn't adhere well.
The bottom line is, it doesn't look like I'll be able to improve on what I've been doing. If I don't get done in a week, I'll pull it up and start over.
My varnishing will never win any awards, but it doesn't look too bad to a casual observer and is as good as I'm willing to put the effort in to do. To really do a first class job, I would have to strip it all down and start over. Maybe in my next life. Same with gelcoat, I just spent 4 days buffing and waxing my hull. It looks nice, not real nice, but I'm not sure it looks much better than I could have got with a good once over with cleaner-wax.