Floating-Point Arithmetic

Floating-point arithmetic represents numbers with finitely many bits and rounds most arithmetic results. A small error in each operation can accumulate, or an expression can amplify uncertainty already present in its inputs. Algebraically equivalent formulas can therefore give very different numerical answers.

This page develops the mathematical mechanisms. Precision diagnostics in quantum calculations applies them to energy splittings, wavefunction norms, tiny probabilities, and refinement studies. Conditioning and stability distinguishes sensitivity of a problem from the behavior of a chosen algorithm.

Helpful background. Norms compare error scales, and inner products give examples of sums with cancellation.

Finite binary representation

A normalized binary floating-point number has the form

$$x=\pm m\,2^e,$$

where the significand $m$ has finitely many binary digits and the exponent $e$ lies in a finite range. With $p$ significant bits, adjacent normal numbers in $[2^e,2^{e+1})$ differ by $2^{e-p+1}$. Absolute spacing grows with magnitude, while relative spacing stays within a factor of two.

Some simple decimal numbers, including $0.1$, have no finite binary expansion. Rounding is therefore needed even before arithmetic begins. Adding a correction smaller than half the local spacing can leave a stored number unchanged.

Precision and range are different. The significand limits resolution; the exponent limits the largest and smallest magnitudes. Subnormal numbers extend the range near zero with reduced relative precision. Overflow, underflow, infinities, and NaNs require separate treatment from the normal-number error model below.

Machine epsilon and unit roundoff

For IEEE binary64, commonly called double precision, the spacing between $1$ and the next larger representable number is

$$\epsilon_{\mathrm{mach}}=2^{-52} \approx2.22\times10^{-16}.$$

Under rounding to nearest, the usual relative-error bound for one normal result uses the unit roundoff

$$u=2^{-53}\approx1.11\times10^{-16}.$$

Some sources call $u$ rather than $2u$ “machine epsilon.” State the convention before using either in an estimate. Neither promises sixteen correct decimal digits in a final result after cancellation or error amplification.

A local rounding model

For a basic real arithmetic operation $\circ$, the standard model is

$$\operatorname{fl}(x\circ y)=(x\circ y)(1+\delta), \qquad |\delta|\le u.$$

This assumes rounding to nearest, a finite defined exact result, and no overflow or underflow that invalidates the relative bound. It describes the rounding of the operation, not errors already present in $x$ and $y$. Applying many individually accurate operations does not automatically make the final answer accurate.

Absolute and relative errors answer different questions:

$$|\widetilde x-x|, \qquad \frac{|\widetilde x-x|}{|x|}\quad(x\ne0).$$

Near a zero target, an absolute tolerance tied to the problem’s scale is needed. Relative error alone is undefined at zero. Error estimates explains how to combine different sources of uncertainty.

Cancellation and inherited input error

Suppose $s=a-b\ne0$, while the supplied operands are $a+\delta a$ and $b+\delta b$. If $|\delta a|\le\varepsilon|a|$ and $|\delta b|\le\varepsilon|b|$, their propagated error satisfies

$$\frac{|\delta a-\delta b|}{|a-b|} \le\varepsilon\frac{|a|+|b|}{|a-b|}.$$

The subtraction condition factor

$$\kappa_{\mathrm{sub}}=\frac{|a|+|b|}{|a-b|}$$

is large when the desired difference is small compared with the operands. The subtraction itself may be exactly representable; the lost accuracy can come from uncertainty inherited from earlier steps. This distinction matters when deciding whether to reformulate an expression or improve the input data.

Worked example: subtracting nearby square roots

For small positive $x$, the expression $f(x)=\sqrt{1+x}-1$ subtracts nearly equal numbers. Rationalizing gives

$$f(x)=\frac{x}{\sqrt{1+x}+1}.$$

At $x=10^{-16}$, binary64 rounding to nearest makes $1+x$ equal to $1$. The direct expression returns zero, whereas the rationalized expression gives approximately $5\times10^{-17}$, consistent with $f(x)=x/2+O(x^2)$.

The function is well conditioned near zero with respect to relative perturbations in nonzero $x$; the first evaluation path loses the small answer. Specialized evaluations of $e^x-1$ and $\log(1+x)$ use the same principle of retaining a small quantity directly instead of forming it by subtraction.

Summation order and cancellation

Floating-point addition is not associative. For example, in binary64 rounding to nearest, with $a=10^{16}$, $b=-10^{16}$, and $c=1$,

$$\operatorname{fl}(\operatorname{fl}(a+b)+c)=1, \qquad \operatorname{fl}(a+\operatorname{fl}(b+c))=0.$$

The inner addition in the second expression loses the unit correction. Mixed signs or complex phases can make a sum much smaller than the sum of the term magnitudes. Pairwise or compensated summation can reduce accumulated rounding error, but cannot recover information already missing from the terms.

For complex inner products, conjugation is a separate requirement from accurate summation. A missing conjugate changes the mathematical quantity, even in exact arithmetic. Quantitative summation bounds and safe norm evaluation are developed in precision diagnostics.

Exercises

Two meanings of epsilon. Distinguish binary64 spacing immediately above $1$ from unit roundoff under rounding to nearest.

Solution

The spacing is $2^{-52}$. Rounding to the nearest representable value incurs at most half that spacing near $1$, giving the standard relative bound $u=2^{-53}$. The term “machine epsilon” is ambiguous unless its convention is specified.

A stable small difference. Derive the rationalized square-root expression and explain why its denominator is harmless for small positive $x$.

Solution

Multiply numerator and denominator by $\sqrt{1+x}+1$. The numerator becomes $(1+x)-1=x$ algebraically, so it can be supplied directly from the input. The denominator approaches $2$ and contains no subtraction of nearby quantities. This avoids the cancellation in the original evaluation path.

Amplitudes and probabilities. Why can summing complex amplitudes be more sensitive to rounding than summing nonnegative probabilities?

Solution

Terms of comparable magnitude can cancel through their phases, leaving an amplitude much smaller than the sum of their magnitudes. Errors in individual terms or accumulation can then produce a large relative error in the result. Nonnegative sums do not have this cancellation mechanism, although they still suffer rounding and range limitations.

The quantum precision exercises continue with energy variance, grid refinement, and deciding whether a numerical constraint repair changes the physical model.

References

• Goldberg, D., “What Every Computer Scientist Should Know About Floating-Point Arithmetic”, ACM Computing Surveys 23, 5–48, 1991 — representation, rounding, cancellation, and exceptional values.
• Higham, N. J., Accuracy and Stability of Numerical Algorithms, 2nd ed., SIAM, 2002 — local error models, conditioning, and summation stability.
• IEEE Computer Society, IEEE Standard for Floating-Point Arithmetic, IEEE Std 754-2019, 2019 — binary formats and arithmetic semantics.
• Trefethen, L. N., and D. Bau III, Numerical Linear Algebra, SIAM, 1997 — backward error and the relation between conditioning and stability.