Is it because it can exceed the reverse breakdown voltage of semiconductors?
That's one of the reasons, but you'll have to look at the device as something much more complex. For example, think of a transistor that would normally be "off", but negative biasing puts it into conducting mode.
In that mode, it might carry a much higher current than it was designed for, leading to the destruction of the device.
Then you've got protection diodes, whose job is to break down in case of ESD, or in case you suddenly stop supplying current to a motor, and that causes, by the inductive properties of that motor, a negative voltage spike of very large amplitude (but only for a very short duration), which you need to "short away" as quickly as possible. Those diodes would normally be in "non-conducting" mode, but if you wire them up reversely, they will constantly conduct electricity, overheat and die.
More examples:
In classical switch mode power designs, there's often a diode antiparallel to the excited coil; in normal bias without the switch mode controller active, it doesn't let any current through. If you plug that circuit in in reverse, the diode will constantly carry high current and evaporate.
Electrolytic Capacitors (typically, the round cans) are voltage sensitive and will quickly be destroyed by reverse voltage, reducing their oxide dielectric layer, leading to them gassing, high currents flowing, liquid blowing up, fires, the four apocalyptic horseman and more Justin Bieber albums.