apache / parquet-cpp / 1017

push

travis-ci

PARQUET-816: Workaround for incorrect column chunk metadata in parquet-mr <= 1.2.8

This turned up in reading of old data files generated by parquet-mr in 2013. There's a bug in parquet-mr 1.2.8 and lower in which the column chunk metadata in the Parquet file is incorrect. Impala inserted an explicit workaround for this (see See https://github.com/apache/incubator-impala/blob/88448d1d4/be/src/exec/hdfs-parquet-scanner.cc#L1227).

In this particular file, the dictionary page header is 15 bytes, and the entire column chunk is: 15 (dict page header) + 277 (dictionary) + 17 (data page header) + 28 (data page) bytes, making 337 bytes.

But the metadata says the column chunk is only 322 bytes – the dict page header size got dropped from the accounting.

Author: Wes McKinney <wes.mckinney@twosigma.com>

Closes #209 from wesm/PARQUET-816 and squashes the following commits:

21fdcbe [Wes McKinney] Move FileVersion to an inner class in FileMetaData
64e7f95 [Wes McKinney] Remove unnecessary std::move causing clang warning
bacb815 [Wes McKinney] Fix compilation error in benchmarks
f4c259e [Wes McKinney] cpplint
1e8c160 [Wes McKinney] clang-format
d2aa9a8 [Wes McKinney] Do not continue reading data pages in SerializedPageReader reading the indicated number of rows in a row group
2638490 [Wes McKinney] Bring in IMPALA-694 workaround for PARQUET-816
bd3e949 [Wes McKinney] Optimistically decode truncated data pages. Add example data file

112 of 112 new or added lines in 8 files covered. (100.0%)

10362 of 10749 relevant lines covered (96.4%)

64599.29 hits per line