All USB hosts must however be able to provide 100mA by default, so in terms of current you should have no issues.
However, fans are inductive and could produce back-EMF when the power switches off. This is basically a high voltage spike that will feed back into the USB port. Depending on how well protected the port is, this shouldn't be an issue. I would personally place a diode in anti-parallel with the fan (anode to GND). This will add some additional protection.
Another potential issue is that if the fan stalls, you may get a current surge as a result because the fan is basically just a DC motor, and current could rise rapidly under stall condition. This is likely not an issue with higher quality fans. Placing something like a 15Ω resistor in series between the 5V and the fan will limit the maximum current that can be drawn in the event of a stall. It will cause ~0.5V drop under normal conditions, but this shouldn't be an issue for the fan.
As a final point, the fan will generate electrical noise. I would personally place something like a 10μF capacitor across the USB +5V/GND pins. This in combination with the 15Ω resistor will act as a low pass filter, limiting the amount of noise that gets back into the device from the motor.
Something like this should suffice:

simulate this circuit – Schematic created using CircuitLab
Investigation - Akasa AK-183-L2B 12V 120mm Fan
So I though I would do a quick investigation with a branded (£8) fan, to see if my points about noise and filtering, quote "... are unfounded" (@AliChen)
So here is my PSU. I'm loading it with a ~33mA pure resistive load, which is an equivalent current draw to the fan:

Not bad I think you would agree. Regulating nicely, with ~40mV ripple/noise.
Now lets connect the fan. The fan draws roughly 33mA from the PSU, and is spinning happily.

There are now period spikes pulsing up to ~5.5V, with a large amount of ripple and noise (~200mVpp). I personally don't want that anywhere near a USB port.
Now lets put a capacitor across it:

Not bad, filters out a lot of the mess.
Now with a series resistor as well, same circuit as in the main answer above (minus the diode). I'm probing on the power supply side.

Even better. The spikes are cleaned up to ~70mVpp, and the noise is pretty clean.
As another point of interest, this particular fan draws about 55mA in a stall condition, so not too bad.