nebkat / dart-bjdata / 34783141233

85%

Build Type

push

github

Committed by nebkat

Commit Message

feat: Choose type markers by size rather than by Dart type

Adds BjdataConfig.compactTypes, on by default, letting the encoder write a
value as whatever type stores it in the fewest bytes instead of the one its
Dart type implies. Values are always preserved exactly; what can change is the
Dart type they decode back to, which is what the setting turns off.

A List<int> of small numbers is written as a uint8 array, halving it. A
Uint32List whose values all fit a byte is written as uint8, quartering it. A
Float64List whose values all survive float16 is written as float16, and so is
a scalar double, which took nine bytes to say 1.0.

Which form wins is genuinely data-dependent rather than a rule of thumb: a
strong type must be wide enough for its largest value and pays that width
throughout, while a generic array stores each value at its own width. So both
are measured for plain lists, and [1, 2, 3, 1000000] stays generic because a
uint32 array of it would be larger. Typed data is only ever narrowed, never
turned generic: passing a Uint32List is itself a request for a typed array, and
a generic array beats one only on a handful of elements.

Nothing is re-typed unless that saves bytes. A positive Int64List is left as
int64 rather than swapped for the same-width uint64, a tie leaves the encoding
alone, an empty buffer keeps its type, and byte is never re-chosen since it is
already the narrowest width and the specification gives it a meaning of its own.

Floats narrow only where every value survives the narrower type unchanged, so
a column of real measurements stays float64 while a column of halves does not.

Packed table columns and N-dimensional arrays narrow too. A column has no Dart
type implying a width, so it is always as narrow as its values allow, as it
already was for integers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Coverage Stats

167 of 172 new or added lines in 6 files covered. (97.09%)

96 existing lines in 4 files now uncovered.

1139 of 1343 relevant lines covered (84.81%)

1.55 hits per line