vb.net - FormatNumber replacing number with 0

Question

Not understanding this: Number returned from DataReader: 185549633.66000035

We have a requirement to maintain the number of decimal places per a User Choice.

For example: maintain 7 places.

We are using:

FormatNumber(dr.Item("Field"), 7, TriState.false, , TriState.True)

The result is: 185,549,633.6600000. We would like to maintain the 3 (or 35) at the end.

When subtracting two numbers from the resulting query we are getting a delta but trying to show these two numbers out to 6,7,8 digits is not working thus indicating a false delta to the user.

Any advice would be appreciated.

Answer 1

Based on my testing, you must be working with Double values rather than Decimal. Not surprisingly, the solution to your problem can be found in the documentation.

For a start, you should not be using FormatNumber. We're not in VB6 anymore ToTo. To format a number in VB.NET, call ToString on that number. I tested this:

Dim dbl = 185549633.66000035R
Dim dec = 185549633.66000035D

Dim dblString = dbl.ToString("n7")
Dim decString = dec.ToString("n7")

Console.WriteLine(dblString)
Console.WriteLine(decString)

and I saw the behaviour you describe, i.e. the output was:

185,549,633.6600000
185,549,633.6600004

I read the documentation for the Double.ToString method (note that FormatNumber would be calling ToString internally) and this is what it says:

By default, the return value only contains 15 digits of precision although a maximum of 17 digits is maintained internally. If the value of this instance has greater than 15 digits, ToString returns PositiveInfinitySymbol or NegativeInfinitySymbol instead of the expected number. If you require more precision, specify format with the "G17" format specification, which always returns 17 digits of precision, or "R", which returns 15 digits if the number can be represented with that precision or 17 digits if the number can only be represented with maximum precision.

I then tested this:

Dim dbl = 185549633.66000035R

Dim dblString16 = dbl.ToString("G16")
Dim dblString17 = dbl.ToString("G17")

Console.WriteLine(dblString16)
Console.WriteLine(dblString17)

and the result was:

185549633.6600004
185549633.66000035