sunchao / parquet-rs / 347
96%

DEFAULT BRANCH: master
Ran
Jobs 1
Files 25
Run time 2s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
347

Pull #80

travis-ci

web-flow 
Read batch of values for RLE, DELTA_BINARY_PACKED and PLAIN decoding

This adds a `get_batch` method to `BitReader`, which is used by
RLE, DELTA_BINARY_PACKED, and PLAIN decoding. It is also used indirectly
by DICT encoding which uses RLE encoding.

This is done similarly to PARQUET-671, where we use fast unpacking in
https://github.com/lemire/FrameOfReference to unpack bit-packed integers
(each patch has 32 values). This is then used in:

1. PLAIN encoding, when decoding boolean types
2. DELTA_BINARY_PACKED encoding, when decoding deltas for each mini-block
3. RLE encoding, when bit-packing is used
3. Level decoder, when bit-packing is used

Test results show this can improve performance a lot:

Before:

test delta_bit_pack_i32_1k_128 ... bench:      46,039 ns/iter (+/- 9,908) = 31 MB/s
test delta_bit_pack_i32_1k_32  ... bench:      49,225 ns/iter (+/- 56,843) = 29 MB/s
test delta_bit_pack_i32_1k_64  ... bench:      47,451 ns/iter (+/- 26,391) = 30 MB/s
test delta_bit_pack_i32_1m_128 ... bench:  45,701,937 ns/iter (+/- 6,000,442) = 32 MB/s
test delta_bit_pack_i32_1m_32  ... bench:  46,565,152 ns/iter (+/- 11,740,361) = 32 MB/s
test delta_bit_pack_i32_1m_64  ... bench:  44,812,203 ns/iter (+/- 4,263,573) = 33 MB/s
test dict_i32_1k_128           ... bench:      36,773 ns/iter (+/- 3,072) = 34 MB/s
test dict_i32_1k_32            ... bench:      37,843 ns/iter (+/- 14,263) = 33 MB/s
test dict_i32_1k_64            ... bench:      37,140 ns/iter (+/- 3,701) = 34 MB/s
test dict_i32_1m_128           ... bench:  37,544,476 ns/iter (+/- 2,914,721) = 34 MB/s
test dict_i32_1m_32            ... bench:  37,173,955 ns/iter (+/- 1,527,749) = 35 MB/s
test dict_i32_1m_64            ... bench:  36,951,548 ns/iter (+/- 1,694,937) = 35 MB/s
test dict_str_1m_128           ... bench:  39,279,048 ns/iter (+/- 4,481,945) = 13 MB/s

After:

test delta_bit_pack_i32_1k_128 ... bench:       6,274 ns/iter (+/- 989) = 233 MB/s
test delta_bit_pack_i32_1k_32  ... bench:       6,295 ... (continued)
Pull Request #80: Read batch of values for RLE, DELTA_BINARY_PACKED and PLAIN decoding

9065 of 9564 relevant lines covered (94.78%)

0.95 hits per line

Uncovered Existing Lines

Lines Coverage ∆ File
1
97.53
 0.35% encodings/levels.rs
8
95.88
 -0.21% encodings/rle.rs
11
96.2
 0.35% util/bit_util.rs
20
95.15
 -0.23% encodings/decoding.rs
Jobs
ID Job ID Ran Files Coverage
1 347.1 25
94.78
 Travis Job 347.1
Source Files on build 347
Coverage ∆ File Lines Relevant Covered Missed Hits/Line