But, if the potential at the two inputs is equal, then shouldn't the
output of the op-amp be 0 (or very close to it)
Yes, this is very true of any op-amp except ideal ones. It creates an error that sometimes can be annoyingly large. Consider an op-amp with an open-loop gain of 10,000 in a unity gain non-inverting circuit: -

Image from wiki.
If Vin is (say) 3 volts, we would expect the output to be 3 volts theoretically but, to get 3 volts at the output we need an input differential voltage of 3/10000 or 0.3 mV.
Because it's a unity gain amplifier, this means that the true output is not 3 volts but 2.9997 volts. That's at DC. At AC, the open-loop op-amp gain drops almost proportionately with frequency above the 3 dB point something like this: -

Image from here.
And, the impact of this is that the gain error gets larger with frequency. So, choose your op-amp carefully for your application and do an error analysis. Note also that for op-amp circuits with signal gain, you get an even bigger error.
For instance, at DC, if the open-loop gain is 10,000 and you require a circuit gain of (say) 10, if you fed 300 mV at the input (hoping to get 3 volts on the output), the actual output voltage would be 2.997 volts (previously 2.9997 volts for a unity gain configuration): -
