1
76%
master: 76%

Ran 21 Jun 2019 10:32AM UTC

Files 1065

Run time 1min

Badge

Embed ▾

Committed 21 Jun 2019 10:32AM UTC coverage: 76.389% (+4.5%) from 71.899%

Job # 1784.1

Build Type

push

jenkins

Committed by kunal642

Commit Message

[CARBONDATA-3447]Index server performance improvement

Problem:
When number of splits are high, index server performance is slow as
compared to old flow(Driver caching). This is because data is transferred
over network is more and causing performance bottleneck.

Solution:
1. If data transferred is less we can sent through network, but when it
grows we can write to file and only send file name and in Main driver
it will read the file and construct input split.
2. Use snappy to compress the data, so data transferred through network/written
to file size will be less, so IO time wont impact performance
3. In main driver pruning is done in multiple thread, added same for
index executor as now index executor will do the pruning
4. In case of block cache no need to send blockletdetailinfo object as
size is more and same can be constructed in executor from file footer

This closes #3281

Co-authored-by: kunal642 <kunalkapoor642@gmail.com>

Coverage Stats

63647 of 83320 relevant lines covered (76.39%)

1.0 hits per line

apache / carbondata / 1784 / 1
76%
master: 76%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 1784.1

apache / carbondata / 1784 / 1 76% master: 76%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 1784.1

apache / carbondata / 1784 / 1
76%
master: 76%

README BADGES
x