You have "short stack", which may not be hot enough, and may be a induced back draft from various wind effects.

I've experienced wood fireplace chimney's in houses or cabins that get intermittent back drafts in strong winds -- even after they are heated-up. Those had exhaust point that were feet, not inches above a deck. The house also wasn't heeled over ;^))))