petl-developers / petl / 32295565417

92%

push

github

Fix avro decimal precision and scale inferred from Decimal values (#706)

precision_and_scale() derives the decimal logicalType of a column from a
sample of its values. It is a copy of fastavro's bytes-decimal encoder with
the `delta = exp + scale` rescale hardcoded to 1 and the digit count taken
from the rescaled integer with ceil(log10(...)), which leaves four defects:

* any zero aborts toavro() with "ValueError: math domain error", since
  log10(0) is undefined and every zero has (0,) as its digits;
* a value with more than 8 fractional digits gets a scale larger than the
  precision, which the spec forbids ("Scale must be zero or a positive
  integer less than or equal to the precision") and fastavro rejects;
* a positive exponent is read as fractional digits, so Decimal('1E+2') is
  declared with a scale of 2 instead of 0;
* the precision of a column is the greatest precision of its values, but
  writing rescales each value to the scale of the schema, which widens its
  unscaled integer beyond the declared precision.

Count the digits from the coefficient and the exponent instead, and combine
the sample on the integer part and the scale separately. Non finite values
have no decimal representation at all, so they now raise a clear error
instead of failing inside the digit count.

99 of 101 new or added lines in 2 files covered. (98.02%)

14098 of 15341 relevant lines covered (91.9%)

0.92 hits per line