to allow for expansion as it soaks up water.

In our old boat I allowed 0.010" clearance in a 2" diameter rudder pintle, and it became VERY stiff after a few months. I opened it out with a rasp to about 1/16" clearance, (rough) and sailed that way for 8 years. Since it was just an outboard pintle, there was no issue with a seal. (The boat design did not need a seal, which is the best approach when possible)

In your kind of boat, something more sophisticated is needed, since you have to hold the shaft in good enough alignment for the seal to work.

Sorry that I cannot give a precise suggestion on clearance. It will depend on the material and the diameter of the shaft. I agree 100% with Evans suggestion to use a low-expansion plastic.

Peerhaps another denizen can offer recommendations. I machine shop used to doing work on boats shou know.