In your board layout, the voltage terminals are sensing the resistance made up of the resistor component, the solder down to the board, and an ill-defined short length of copper of odd geometry.
If the resistor was a few ohms, you could probably neglect the solder and the copper. With a few milli-ohms, it is almost certainly significant.
The solder joint may vary from run to run, and will vary if you change the pad size. Any time you change the pad size or geometry, the copper factor may change slightly. You would have to characterise these factors yourself, and repeat that any time you changed something.
However, if you use a 4 terminal resistor, then the resistor manufacturer has measured his resistor in the way you are going to connect it, and is guarranteeing the resistance in that configuration. You can change pad size, geometry, solder conditions, and your measured 4 terminal resistance will stay the same.
That is the whole point of the 4 terminal connection, repeatability in the face of variable connections to the device itself.