unicode-org / icu4x / 11852546259

73%

Build Type

push

github

Committed by web-flow

Commit Message

Split numbering systems out of decimal data (#5822)

Fixes https://github.com/unicode-org/icu4x/issues/5818


Before:

```
 decimal/symbols@2, <lookup>, 1316B, 252 identifiers
 decimal/symbols@2, <total>, 4308B, 2436B, 49 unique payloads
```

After:


```
 decimal/digits@1, <lookup>, 207B, 27 identifiers
 decimal/digits@1, <total>, 1080B, 1060B, 27 unique payloads
 decimal/symbols@2, <lookup>, 804B, 184 identifiers
 decimal/symbols@2, <total>, 1749B, 881B, 31 unique payloads

```

Saving ~1.5kB, a good third of the data size. A lot of the wins are just
in deduplication.

I'm also going to try moving the tinystr into the VarZeroVec and seeing
what happens.

I may also try and store the digits more compactly as an `enum {
Sequential(char), Many(ZeroVec<char>) }`. A downside of this is that the
Sequential case would need UTF8 validation every time, though we could
make it so that that's just the wire format and we expand to a digit
array on data load.


Todo: add configurability for this.

@sffc In the long run CompactDecimal / etc should also be using this
data. In that case, should we just always generate all known decimal
systems? How would the unification work across keys?

Run Details

66 of 103 new or added lines in 12 files covered. (64.08%)

7 existing lines in 6 files now uncovered.

54461 of 72260 relevant lines covered (75.37%)

656916.29 hits per line