• Home
  • Features
  • Pricing
  • Docs
  • Announcements
  • Sign In

vortex-data / vortex / 16299238102
88%

Build:
DEFAULT BRANCH: develop
Ran 15 Jul 2025 04:50PM UTC
Jobs 2
Files 592
Run time 2min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

15 Jul 2025 04:45PM UTC coverage: 81.638% (+0.2%) from 81.486%
16299238102

push

github

web-flow
Compact Compression (#3765)

Added a CompactCompression strategy that generally uses Zstd for
binary/string data and Pco for numeric data types.

Size comparison, using the NYC Taxi dataset and running e.g. `cargo run
--release -p vortex-tui convert fhvhv_tripdata_2023-04.parquet
--strategy compact` to produce files:

```
433M fhvhv_tripdata_2023-04_btrblocks.vortex
334M fhvhv_tripdata_2023-04_compact_8192.vortex
321M fhvhv_tripdata_2023-04_compact_inf.vortex
469M fhvhv_tripdata_2023-04.parquet (zstd compressed)
```

Here the two compact strategies are using up to 8192 values per page
versus "inf", as many as possible. Using 8192 (the default I put in the
code) slightly increases size but allows for faster access into slices
and can (in the non-null, non-list case) line up with statistics nicely
for potential pushdown filters.

NOT HAPPENING IN THIS PR (leaving this to future work):

* compression for bool arrays
* compression for decimal array 
* dict encoding for variable-length types

Other changes this incurred:

* Added --strategy arg to convert command (and simplified flags stuff a
bit)
* Added VarBinView support for Zstd encoding
* Added unit tests for all the new functionality

Fixes https://github.com/vortex-data/vortex/issues/3611 .

---------

Signed-off-by: mwlon <m.w.loncaric@gmail.com>

448 of 506 new or added lines in 7 files covered. (88.54%)

1 existing line in 1 file now uncovered.

46727 of 57237 relevant lines covered (81.64%)

144167.36 hits per line

New Missed Lines in Diff

Lines Coverage ∆ File
1
0.0
0.0% encodings/zstd/src/serde.rs
28
83.72
vortex-layout/src/layouts/compact.rs
29
89.27
-3.73% encodings/zstd/src/array.rs

Uncovered Existing Lines

Lines Coverage ∆ File
1
0.0
0.0% encodings/zstd/src/serde.rs
Jobs
ID Job ID Ran Files Coverage
1 run-tests - 16299238102.1 15 Jul 2025 04:50PM UTC 592
78.94
GitHub Action Run
2 run-tpc-h - 16299238102.2 15 Jul 2025 04:50PM UTC 554
48.78
GitHub Action Run
Source Files on build 16299238102
  • Tree
  • List 592
  • Changed 11
  • Source Changed 6
  • Coverage Changed 11
Coverage ∆ File Lines Relevant Covered Missed Hits/Line
  • Back to Repo
  • Github Actions Build #16299238102
  • 677f3eec on github
  • Prev Build on develop (#16295000145)
  • Next Build on develop (#16299277854)
STATUS · Troubleshooting · Open an Issue · Sales · Support · CAREERS · ENTERPRISE · START FREE · SCHEDULE DEMO
ANNOUNCEMENTS · TWITTER · TOS & SLA · Supported CI Services · What's a CI service? · Automated Testing

© 2025 Coveralls, Inc