JuliaIO / Zarr.jl / 26688103130

90%

Build Type

push

github

Committed by web-flow

Commit Message

Performance improvements (v2 only for now) (#280)

* use bulk copy in zcompress! fallback to avoid per-byte growth

`zcompress(data, NoCompressor())` returns a lazy `reinterpret(UInt8, data)`
view. The old `empty!` + `append!` fallback walked that view element by
element through `_growend!` / `push!`, materialising bytes one at a time —
roughly 67% of CPU time on uncompressed full-chunk V2 writes in profiling.

Replace with `resize!` + `copyto!` so the same path issues a single SIMD /
memcpy bulk copy. Real compressors (Blosc, Zlib, Zstd) are unaffected: they
already return a freshly-allocated `Vector{UInt8}` and `copyto!` handles
that case identically.

Bytes-on-disk are bit-identical; full test suite (2499 tests) passes.

Measured on Apple M2 Ultra, Julia 1.12.6, V2 + NoCompressor + chunks=(Nx,Ny,Nz,1):

  | size                | baseline | patched  | speed-up |
  |---------------------|---------:|---------:|---------:|
  |  128×128×16×50  50M | 702 MB/s | 995 MB/s | 1.42×    |
  |  256×256×32×50 400M | 799 MB/s |1530 MB/s | 1.92×    |
  |  512×512×32×50 1.5G | 776 MB/s |1104 MB/s | 1.42×    |

Reads are unchanged (this fallback is not on the read path).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* use bulk copy in NoCompressor zuncompress! to skip lazy-view walk

The generic `zuncompress!` fallback at the top of `Compressors.jl`
ends up at `copyto!(::Array{T}, ::ReinterpretArray)` when
`c isa NoCompressor`, since `zuncompress(bytes, ::NoCompressor, T)`
returns a lazy `reinterpret(T, bytes)` view. That `copyto!` walks
element by element at ~17 GB/s on Apple silicon vs. ~85 GB/s for
`unsafe_copyto!` on the same memory — a 5× gap. End-to-end V2 reads
absorb most of it via DiskArrays slicing, channel ferry, and storage
I/O, leaving ~25-99% throughput improvement depending on chunk size.

Add a `::NoCompressor`-dispatched `zuncompress!` method alongside
the existing per-compressor versions for Zstd / Zlib / Blosc. Mirror
imag... (continued)

Coverage Stats

42 of 42 new or added lines in 2 files covered. (100.0%)

4 existing lines in 3 files now uncovered.

1641 of 1834 relevant lines covered (89.48%)

18922.87 hits per line