Finally ended up removing it. I was on a small 2 cylinder Yanmar so I gave it the benefit of the doubt and assumed that the engine just vibrated too much. The instructions do say to expect some leakage during the run in period.