If the distances between measurement point and the measurement circuit are short (i.e. there is no distance/cable involved that could be threatened by an external noise source) AND there is a shared ground between the measurement point and the measurement circuit, then sure, you can use a single ended amplifier.
The problem arises when you have noise threats on the connection between source and measuring circuit OR when there is ground bounce between measurement point and measuring circuit. For instance, you might think that this circuit is fine: -

However, if you analysed how noise might react with the cable and the unbalanced impedances of both wires then you would get this: -

With a single-ended receiving amplifier, one input is firmly tied to local ground and will be largely unaffected by the interference on the cable. Sure, there'll be an injection of energy but, that energy will be "soaked-up" by the local ground producing very little surge artefacts.
However, the other input has an impedance to local ground governed by the input resistor, \$R\$ and, it will develop a surge voltage that will be amplified by the single-ended amplifier.
This can only be solved by a differential amplifier with a balanced input impedance: -

Because both inputs into the differential receiving amplifier have the same impedance to local ground, the interference waveform is seen equally at both input nodes and, the effect of the differential amplifier is to cancel those signals out.
If the measurement point were locally grounded it creates even bigger problems for the single-ended amplifier but can even create problems for the differential/balanced amplifier if the source has output impedance: -

This can only be solved by balancing the input source with respect to its impedances: -

Images and some text from here