You're conflating two concepts, potential, and potential difference.
It's probably easiest to go right back to the definition of potential to see what's happening.
A potential field is one where the potential energy of an item that feels that field is dependent only on its position. For instance, in a gravity field, the energy of a test mass is given by its height alone. The change of energy in moving from height 1 to 2 is the same as that released by moving from 2 to 1. This energy is measured in Newton.meters (Nm) aka Joules.
In an electric field, you use a test charge instead of a test mass. It needs energy to move it to a higher potential, and releases the same amount of energy when moved to a lower potential. This energy is measured in charge.volts. In atomic physics, we use eV, electron.volts, but in electrical engineering, we tend to use coulomb.volts, aka Joules, aka watt.seconds.
What a multimeter does is allow us a short-cut way to estimate the energy changes that would happen as we move this test charge around. If we connect the probes across a battery, it will read directly what the voltage difference is between two points. If we took one coulomb of charge, and moved it from one terminal of a 12v battery to the other, it would absorb (if we were charging) or deliver (if the battery was discharging) 12 coulomb.volts, that is, 12 Joules.
Now let's push our test charge out from one battery terminal to a third point. The energy it took to do that is the potential difference of the third point to that at which we started. Now let's bring it back to the other terminal. The difference in energy for the whole trip will be 12 charge.volts. It doesn't matter (conceptually) where that third point is, at mains ground, on the moon, at infinity, the round trip journey still has the same change in energy.
As the only thing we can measure is the difference between two potentials, we can assign an arbitrary reference point. We could measure the potentials of several objects, then add the same constant to each measure, and the potential differences would stay the same.
For instance, in a gravity field, there are several common 'zero potential' reference levels. There's the floor level, local ground level, mean sea level, and deep space, depending on what your profession and purpose is.
In an electric field, electrical engineers tend to use earth potential or local chassis as zero, where physicists might sometimes prefer to use infinity. Either reference can be translated into the other by simply adding a constant to all potentials.
So 'zero potential at infinity' is a definition, not a measurement.
Let's see what potential to ground can look like under one measurement condition. This is a picture of a battery with 10pF capacity to ground (reasonable), a person with 100pF capacity to ground (reasonable, taken from the Human Body model for ESD testing), with 1pF capacity to the battery (equivalent to reaching out a hand, and holding it 100mm from the battery), and a battery represented by an internal capacity of 100kF (the battery voltage changes by 1 volt (12.8v down to 11.8v) when we pull 30Ah of charge from it, do the C=Q/dV sums).

simulate this circuit – Schematic created using CircuitLab
The first thing to notice is the huge dynamic range, 100kF/1pF = \$10^{17}\$.
The idea is that initially the battery was at ground potential, but then you walked across a carpet, getting charged to 11kV in the process, and then held out a hand and waved it 100mm from the battery.
You can see that 1pF capacity to the battery gets charged to about 10kV, moving a charge of CV = 10nC.
As that charge also charges the battery capacity to ground, the battery case gets charged to 1kV (Q/C = V). See how easy it is to alter the potential of things when they're not grounded.
That charge also flows through the battery (in the configuration I've drawn it). If we do V = Q/C again, we find the battery voltage has changed by 100fV (0.1pV), which is not very much.