Short Answer: Reduce R1 from 10 to 1 Ohm and add 10:1 divider on input or as req'd.
Your Op Amp Output High or Ioh current limits being close to your operating limit of 100mA imposes severe voltage drop internally on Vds
Also for stability it may be difficult to calculate as it "saturates" the output losing negative feedback and thus linear gain drops to zero.
However the impedance of the LED is dynamic so it never reaches 100mA and the ouput can oscillate in and out of "full rail" mode ( although technically only called saturation in BJT's ) from full linear gain to zero gain. A very unstable condition.
The Iout min > 100mA due to RdsOn = 35 Ω @ 5V, so 100mA will result in a Vds drop of 3.5V!! or Vout=1.5V Meanwhile the current rises to 100mA only if there is no drop on the current sense R1 which is an additional 10*0.1A= 1V so this adds up to 5.9V Vdd required to make it work.!!
So the initial solution is try 1 Ohm then analyze worst case tolerances for temp., LED and supply voltage.

simulate this circuit – Schematic created using CircuitLab
OPA354
The stability at lower currents is dictated by the Coss of the MOSFETs and C of the LED which results in a mismatched impedance especially when there is inadequate headroom for Vds internally.
Speculation
It may require increasing the Vdd to 5.5 to meet the stability at maximum current.
There is no gaurantee this will work at all temperatures, but it is close.
This chip however may dissipate 360mW which must be dissipated and may require a board copper heat heatink of case heatsink. Rθja=~90'C/W
optimal improvements

simulate this circuit
Layout to all input and output pins must be done carefully to minimize load capacitance ( from a ground plane ) and perhaps using guard tracks to isolate feedback capacitance. If there is more positive feedback capacitance than negative feedback by virtue of track or jumper geometry (even by 1 pF) it will induce spurious ringing. Therefore this solution was intended to keep all capacitance to an absolute minimum for fastest risetime yet balanced to eliminate spurious ringing. The 1pF would have to be selected based on layout.
Feedback R value if too small will result is wild relaxation oscillations and too large reduces rise time, so impedance matching is critical for 30 to 50MHz BW at high current and there is more work to be done here.