According to the user manual (http://www.lofrans.com/ajax/download.php?documents[]=3), max lift working load is 418lbs. You're good there.
According to the user manual, the Tigres 12V, 1500W version has #2 wire recommended for less than 25m distance. Your good here - make sure the *SUM* of the positive and negative wire lengths are less than 25m.
According to the user manual, the Tigres 12V, 1500W version can draw ~130A at working load, so the breaker seems to be on the edge a bit.

With 3m distance and #2 wire (x2 for + and -), you are right at the edge of the 3% drop curve at 100+ amps. This means that your voltage drop due to cable alone will be ~0.4V @14V battery.

An important question of me is - did it ever work? If so, then suspicion is contact corrosion, and/or solenoid problems. There is a short cleaning primer here: http://www.sailfeed.com/2014/01/how-to-fix-your-electric-windlass-9-out-of-10-times/ If not, then it's a different installation/design problem.

I would measure the voltage drop with the system unloaded and measure the voltage drop through the cable/connections (not "calculated") and see if it is reasonable. 9.5V at motor sounds like a big drop - at 130A, this is not even providing the rated 1500W to the motor.

Also, check that the voltage drop on EACH of the motor's power posts are equal when going in both directions, unloaded. A big difference will point to the relay/contactor or motor issues, not cabling of your system.

Just some thoughts....