Sounds like an old Shirley Temple movie title.

The millwrights I worked around did it like this:

Tighten the setscrew to get a mark on the shaft.
Remove the screw and arm/sprocket, etc. and centerpunch the location.
Use a drill the same size as the screw. 3/8" drill for 3/8" screw.
Drill only until the outer shoulder of the bit breaks the surface. If you go deeper then the bending force on the screw might come into play.
One last thing they would do is to tighten the screw, then tap in in and retighten a couple of times. This works better with square head screws but can be done with allen socket heads with the right size punch. I use Locktite.