stillwater-sc / universal / 34780406903

86%

master: 84%

Build Type

push

github

Committed by web-flow

Commit Message

fix(dfloat): stop add, subtract and divide overflowing the significand (#1488)

dfloat computes on significand_t, 4*ndigits + 8 bits or about 1.2*ndigits + 2
decimal digits, but two operations needed up to 2*ndigits digits:

- Addition aligns exponents by scaling the higher-exponent significand by
  10^shift. Unpacked significands are not normalized to ndigits (parse strips
  trailing zeros into the exponent), so the "shift >= ndigits" shortcut did not
  bound that. decimal64 3.812837151747335e16 - 4304.704077704107 gave
  -1298121021.550523.
- Long division ran ndigits steps whatever the divisor, so the quotient grew to
  about lhs_sig * 10^ndigits / rhs_sig. decimal32 1234567 / 1 gave 8664.995, and
  9999999 / 0.001 gave 1315134.

The overflowed significand was packed as is, so the result was garbage in both
encodings, and different garbage in BID and DPD.

Two more defects sat in the same code:

- The shortcut compared exponents, not magnitudes. dfloat<5> -88925 + 5.1e7
  returned 5.1e7, because 51e6 has exponent 6 but only 2 digits. And for opposite
  signs it absorbed operands that do change the result when it falls below a power
  of ten: decimal64 1e16 - 6 returned 1e16, although 9999999999999994 is
  representable, and decimal32 1 - 1e-7 returned 1.
- Division kept fewer than ndigits digits when the dividend's significand was the
  smaller one: decimal32 1 / 1234567 gave 8e-7, not 8.100005e-7.

Addition now aligns and sums in wide_significand_t (the double width multiply
already uses), then truncates to ndigits in a new normalize_wide_and_pack(). The
shortcut decides on the gap between the operands' leading digits. With the same
sign it absorbs a smaller operand ndigits or more digits below, which truncation
leaves unchanged. With opposite signs it absorbs only one more than ndigits + 1
digits below. That returns the nearest value, as before; truncation elsewhere is
the separate question #1487. Division runs the same ndigits steps in the doubl... (continued)

Coverage Stats

37 of 37 new or added lines in 1 file covered. (100.0%)

3 existing lines in 1 file now uncovered.

43596 of 50861 relevant lines covered (85.72%)

7437765.09 hits per line