That selectable output format is probably just meant as convenience to the user.
- Of course, if you used less decimals than your sensor had resolution, you'd lose information in that quantization step.
- Your 0.1° resolution at accurracies worse than 1° is of course also only meaningful if you have some information on how the error between reported angle and actual angle is distributed, in which you can use that to implement an estimator for the actual angle based on multiple observations.
- To that end, it might be more useful to have more digits, even if they're just the result of noise. Or it might not, if your estimation method boils down to be mathematically equivalent to first rounding all numbers.
We might need to talk about what "resolution" is: that's easy if you have say simply a piece of plastic mounted on a rotatable platform that you hold into the wind, it rotates so that it's in the direction of the wind, and you've got some optical encoder linked to the platform that gives you one out of 360 possible angles. 1° resolution, easy. The measurement noise will be simply the quantization noise (i.e. assuming wind directions are equiprobable, you'll be off by ¼° on average, simply because of your resolution).
This device is way more complex. It's something that measures the direction of wind by estimating the doppler shift in orthogonal directions and then calculating a likely direction of wind that led to these estimates of doppler shift.
Doing that math, you can often get arbitrarily close to the real value, if you can observe the same phenomenon (wind direction) just for long enough (for details on what's possible with an unbiased estimator: see Cramer-Rao Lower Bound).
However, in a real system, you have a limited amount of observations, which limits the amount of noise due to actual physical uncertainty (wind isn't 100% constant), measurement errors (amplifier temperature, for example), digitization, doppler estimation (that's spectral estimation, there's libraries full of books on that), and from that, direction estimation. All these steps can introduce new errors!
So, if you're designing a system, you'd go through all these things and say, OK, in the end, my estimate of direction will have a variance \$\sigma^2\$, i.e. a mean squared deviaion from its mean, and that gives us an interval that we can be pretty sure about being in. In a normally distributed variable, more than 95% of values fall into \$2\sigma\$ around the mean!
So, probably, they used some statistics to say (for example) something like
if we don't promise we're more accurate than \$2\sigma\$, which is 0.1°, then our measurement is right 95% of time! Let's call that the resolution.
I don't know what you're measuring, but 0.1° is pretty impressive in angular resolution for anything that isn't solid.