If you are in a area with shifting channels due to the nature of the bottom (mud or sand), post storm blow-outs (or volcanos ;^))) ), then I'd tend to be very careful if you're not sure what is going on.

If you are in an area with pretty fixed bottom conditions (like Maine or similar areas with rock/granite), then I'd be more concerned with hitting a mark that you wouldn't be expecting to have shifted as you move along. I'd check your GPS coordinates, bearings from various landmarks, etc. You should be especially careful in the fog, rain, or at night because you might hit a mark that is now in what otherwise was safe water.

(In my case, I try NOT to head from mark-to-mark in the fog because there are too many others who are either on reciprocal courses or coming up your stern. I will often alter the course to move on the safer "wide" side of a route. If I'm concerned, when I get "close" to where a mark should be, I "duck" over to check-out the number on the mark.)