As a first approximation, there isn't a difference. A similar question might be "what's the difference between a DC motor and a generator?" Either will function in either capacity, but each is optimized to maximize certain qualities at the expense of others, according to the intended application.
Let's compare the internal schematic for a common op-amp, TL072, and a common comparator, LM339:
TL072

LM339

Things in common:
- differential pair input
- very high gain
Differences:
The TL072 uses JFETs for the input. This is because JFETs provide extremely high input impedance, which is desirable for an op-amp. Most op-amp analysis assumes that the bias currents (the currents moving through the inputs) are zero, but this is only true to the extent that the input impedance is infinite. For a comparator, you do want high input impedance to avoid loading the source excessively, but it's not so important that input impedance is very high.
The LM339 has an open collector output. For a comparator, this is good, because it allows it to interface with any number of output voltages through a pull-up resistor, or trivially implement a wired-or bus. You wouldn't want this for an op-amp, because usually, you want an op-amp to be as good at sourcing current as sinking it, so that your output waveform is symmetrical. Notice the NPN-PNP push-pull pair output of TL072.
The LM339 can pull the output as low as 0.2V or as high as \$V_{cc}\$, by virtue of its open-collector output. TL072 specifies an output voltage swing of \$\pm 10V\$ when \$Vcc = 15V\$ when the load is less than \$2k\Omega\$.
An experienced IC designer could probably point out more differences, from the schematic alone. I'm not one of those, but I can see the differences in the datasheets. For example, I don't see a common-mode or power-supply rejection ratio, harmonic distortion, or noise figure specified at all for LM339. These could be measured for a comparator, and you will find them in every op-amp datasheet, but for a comparator application, these parameters aren't especially relevant, so they aren't specified, and if they were, would likely be very poor.
So, in every case, you could use an op-amp as a comparator, or a comparator as an op-amp, if your requirements are not very demanding. Given the difference in how the parts are specified, it may not be possible to know from the specifications how it will even perform.