Check for resistance/voltage drop across ALL wiring in the starter primary circuit ---- keyswitch, pushbutton, all the connectors, and especially the engine mounted fuse holder. You can easily 'test' by bypassing the entire primary circuit by jumping from the started 'hot' connection to the solenoid primary connection. If that instantly starts the starter + engine ... you have a 'voltage drop problem' with primary circuit ... and 'usually' its the damn fuse holder with corroded wire terminals.

One would think that Yanmar would use 'tinned' wiring and components in their panels .... but noooooo just vulnearble 'automotive' stuff.