If you center drill (or pilot drill) the hole, a good twist drill will not wander, its the operator of the drill motor that wanders! Your eye is remarkably bad at lining up a drill you are holding, you cannot depend on it. The best quick system is to have two buddies site it from several feet away as you drill, one from each axis. They point which way to correct your aim as you go, you look at them, not the drill. If you don't have two friends , find some way to temporarily fasten a long rod next to the hole which you can check for alignment with squares, levels, whatever, then use that to help your eye guide the drill.

If you are worried about hitting the hole pattern coming from the outside, best way is to start on the inside with a small pilot drill in a close-quarters drill motor, then come back through with the clearance bit. However that is thin enough that you could mark the pattern and drill from the outside, as long as you keep the alignment close as described above.