NaturalHistoryMuseum / splitgill / #107

95%

main: 95%

push

coveralls-python

feat: implement a series of arc indices instead of a fixed number

BREAKING CHANGE: implement a series of arc indices instead of a fixed number

Documents are now put into arc indices which have a fixed size (at the moment 2 million documents). When an arc fills, the arc index is increased and a new index is used (so the first arc is arc-0, then the next once that has 2 million documents in it is arc-1 and so on). The latest index stays the same. This should give us more long term scalability with massive datasets and/or datasets with lots of versions. There are also template tweaks and now there are distinct templates for the arcs and the latest index which is also handy for setting sharding, compression, and other future purposes.

57 of 57 new or added lines in 4 files covered. (100.0%)

1209 of 1274 relevant lines covered (94.9%)

0.95 hits per line