Do you have dual take-off from the external regulator? What determines the charging output of your Balmar?

The issue for the wire, in terms of size isn't really the volts only -- it's the amperage too (ultimately the watts). The higher the voltage, the greater the amperage capacity before you need to upsize.

Your windlass does draw a lot for a period of time. It really is much more than just a cold-cranking, engine starting battery -- but I think that should work most of the time.

I wouldn't be raising the anchor without the engine running, unless you don't have a choice.

IF you take the amp hours that you are discharging the battery under load with the windkass, then let the battery settle down, then measure the change in voltage, we could probably get a better idea of what's going on.