JuliaIO / Zarr.jl / 12200200130

89%

push

github

Add more filters (#154)

* Document the Filter interface

* Move filters to a folder

Same rationale as the other changes :D - just for cleanliness and clarity.

* Factor out variable-length filters to a new file

* Add docstrings to filter API functions

* Add a Fletcher32 filter and test

* re-add the dictionary entries for the vlen filters

* Semi-working fixed scale offset filter

* Add FixedScaleOffset tests

* Add shuffle filter (buggy in the last few bytes, indexing issues)

* WIP quantize filter

* ShuffleFilter working and tested

* Semi working quantize filter

* Format tests better

* Complete interface and test quantize

* Uncomment the FixedScaleOffset tests

* fix getfilter syntax

* Add delta filter

* Adapt for Kerchunk playing fast and loose with the spec

- Kerchunk often encodes the compressor as the last filter, so we check that the compressor isn't hiding in the filters array if the compressor is null.
- Similarly, the dtype is often unknown in this case, or the transform is not encoded correctly, so we ensure that the datatypes of `data` and `a2` remain the same by reinterpreting.

* Fix the delta and quantize JSON.lower

* Change the tests to be more sensible/Julian and avoid truncation errors

* Fix the FixedScaleOffset filter materializer

* Fix decoding for fill values to use `reinterpret` on unsigned -> integer

* If `getfilter` fails, show the filter name and then throw an error

* Apply reinterpret before multiplication in fixed-scale-offset filter

* Only reinterpret negative integers when decoding fill values to unsigned

* Revert "Only reinterpret negative integers when decoding fill values to unsigned"

This reverts commit 24a68e6e3.

* let Fletcher32 operate on n-dimensional arrays

not just vectors, as it was previously constrained to

* fix FixedScaleOffset in many ways

- Never use reinterpret
- use array comprehensions... (continued)

126 of 160 new or added lines in 8 files covered. (78.75%)

908 of 1054 relevant lines covered (86.15%)

14286.37 hits per line