A good quality shop can support track and gaps on any layer of;
- 3 mil (0.12mm) in std quality and
- 2 mil (0.08mm) on advanced ($) build.
So it is not a process limitation but a contaminant limitation on the effect of breakdown voltages.
Here inner layers can be contaminated by air voids which cause a lower dielectric constant of air 1 vs FR4 >=4. This results in a higher E field and charge buildup in a void which can cause early breakdown.
We call it Partial Discharge (PD) and it behaves like a injunction oscillator, until more rapid and external then it is called corona.
Even though FR4 is rated >=500V per 0.1mm a, void can breakdown in about 1/4 of this. Since a full breakdown voltage thru the dielectric between high voltage conductors is usually followed by extreme high current from storage capacitance of low impedance AC sources, I believe IPC has implemented at least a 2:1 safety factor for internal gaps between HV conductors.
For the outer surfaces it is worse. Even though the solder mask is an excellent insulator like FR4 epoxy, the surface can be bridged by dust and humidity and creates partial discharges that lead to a full breakdown depending on then thickness of dust, humidity and salt spray so,even more safety margin is required for dielectric breakdown on the surface which is equivalent to 100V/0.4mm or 250V/mm about 1/10th of the breakdown of flat surfaces in clean air or 1/4 of sharp edges in clean air..
The key requirement here is safety and flammability under high voltage follow on current. If one is designing a custom board for microcurrent HV levels and has a somewhat sealed environment, they may deviate from the said IPC guidelines. But dust and humidity as well as epoxy voids both degrade the arc flash thresholds of clean pure dielectrics.