Software Engineering Stack Exchange
Software Engineering Stack Exchange

Questions tagged [numeric-precision]

34 questions
68
votes
7 answers

What causes floating point rounding errors?

I am aware that floating point arithmetic has precision problems. I usually overcome them by switching to a fixed decimal representation of the number, or simply by neglecting the error. However, I do not know what are the causes of this inaccuracy.…
nmat
  • 813
  • 1
  • 7
  • 6
47
votes
2 answers

Why do some languages round to the nearest EVEN integer?

Programming languages like Scheme (R5RS) and Python (see this Question) round towards the nearest even integer when value is exactly between the surrounding integers. What is the reasoning behind this? Is there a mathematical idea that makes…
Profpatsch
  • 969
  • 8
  • 13
35
votes
9 answers

Why are floating point numbers used often in Science/Engineering?

While investigating the accuracy of floating point numbers, I've seen in a few places a statement similar to "float and double are (designed for/used often in) engineering and scientific calculation" From my understanding, the strength of floats…
DoubleDouble
  • 603
  • 1
  • 5
  • 11
35
votes
11 answers

Why do we still use floats?

I understand why floats served a purpose in the past. And I think I can see why they're useful in some simulation examples today. But I think those example are more exceptional than common. So I don't understand why floats are more prevalent in…
anon
26
votes
6 answers

Why don't computers store decimal numbers as a second whole number?

Computers have trouble storing fractional numbers where the denominator is something other than a solution to 2^x. This is because the first digit after the decimal is worth 1/2, the second 1/4 (or 1/(2^1) and 1/(2^2)) etc. Why deal with all sorts…
SomeKittens
  • 4,220
  • 6
  • 31
  • 38
23
votes
5 answers

Solutions for floating point rounding errors

In building an application that deals with a lot of mathematical calculations, I have encountered the problem that certain numbers cause rounding errors. While I understand that floating point is not exact, the problem is how do I deal with exact…
JNL
  • 904
  • 2
  • 8
  • 18
16
votes
4 answers

How to identify unstable floating point computations?

In numerics, it is very important to be able to identify unstable schemes and to improve their stability. How to identify unstable floating point computations? I am working on a very complex simulation where many numerical schemes work together and…
user40989
  • 2,860
  • 18
  • 35
16
votes
4 answers

Handling extremely large numbers in a language which can't?

I'm trying to think about how I would go about doing calculations on extremely large numbers (to infinitum - intergers no floats) if the language construct is incapable of handling numbers larger than a certain value. I am sure I am not the first…
Mallow
  • 579
  • 1
  • 4
  • 15
9
votes
3 answers

Is order of arguments in an arithmetic expression important to achieve as most exact result as possible (speed is not necessary)?

Actually, in this question I do not ask about particular language or architecture, however I understand there might be some differences. In physics / engineering it is usually better to handle greater numbers than smaller ones, because of better…
Voitcus
  • 201
  • 2
  • 5
9
votes
1 answer

Implied precision of floating-point functions

While reviewing another programmer's implementation of a function to calculate the normal distribution CDF, I made a suggestion to either replace the entire implementation with Python's built-in functions or use SciPy, a common scientific…
200_success
  • 1,568
  • 11
  • 20
8
votes
5 answers

How to handle large numbers?

Well, I've just started doing puzzles and it's so annoying to see puzzles which are easy to do but also need to handle very large numbers. That's the problem. Say I have to deal with numbers like 10 ^ 6 / 2147483647 / etc. Need to do arithmetic…
0cool
  • 81
  • 1
  • 1
  • 3
7
votes
5 answers

Why SQL numeric precision is specified as the number of digits?

I understand that this might be a loaded question. For example, in Sql Server 2008, doing DECLARE @someNum NUMERIC(10); means that the range of the variable will be limited by 10 digits. In other languages such as C, Java, etc. the standards for…
Job
  • 6,459
  • 3
  • 32
  • 54
7
votes
6 answers

When are equality comparisons between IEEE 754 floating point values acceptable?

My understanding is that it's rarely a good idea to compare two floating point numbers due to inherent inaccuracies. However, my understanding of those inaccuracies are that they are deterministic and will always show up the same way given two…
user361757
7
votes
1 answer

Why normalization improves numerical precision?

I was reading the following article: Polynomial interpolation of GPS satellite coordinates, Milan Horemuz and Johan Vium Andersson 2006 and it states the following: "The estimation procedure of the ai coefficients is done with the Matlab function…
RandomGuy
  • 222
  • 1
  • 8
5
votes
2 answers

Trivially Sortable Encoding for Arbitrary-Precision Decimals

I'm looking for a reasonably(*) space-efficient way to encode arbitrary-precision decimals (e.g. BigDecimal), such that when sorting the bit-pattern of the encodings lexicographically, the numbers will be sorted numerically. This is related to this…
Dexter
  • 293
  • 1
  • 7
1
2 3