Conversion Precision

[davidkappelt]

When converting certain values such as 328.999mV to V (which is a base unit), the answer adds a bit of error resulting in 0.32899900000000004.

Looking at the code I am a bit confused as to why this occurs. Any clarification would be very helpful.

[Brunni]

This is caused by the normal double precision: https://en.wikipedia.org/wiki/Double-precision_floating-point_format

If you write 328.999d / 1000 in a normal c# code you will get the same error. This is just how floating point in computers work.

To avoid this in c# code it is possible to use decimal instead: 328.999m / 1000