@robin

How about using a circular cutter, cut a circle in a piece of plywood of a similar depth of the cutter to use as a guide. Clamp the plywood guide to fretboard and it might stop the cutter from wandering.

I had something like that in mind ...

I was going to make an mdf (or ply) template, marked with the fret positions.  Mark the template with the dot centres, and use the centres to drill a 5mm or 6mm (or whatever) hole.  Then fix the template over the fretboard, lined up carefully and use the holes in the template to hold the circular cutter in place.