1784
76%

Ran 21 Jun 2019 10:32AM UTC

Jobs 1

Files 1065

Run time 1min

Badge

Embed ▾

pending completion

Build # 1784

Build Type

push

jenkins

Committed by kunal642

Commit Message

[CARBONDATA-3447]Index server performance improvement

Problem:
When number of splits are high, index server performance is slow as
compared to old flow(Driver caching). This is because data is transferred
over network is more and causing performance bottleneck.

Solution:
1. If data transferred is less we can sent through network, but when it
grows we can write to file and only send file name and in Main driver
it will read the file and construct input split.
2. Use snappy to compress the data, so data transferred through network/written
to file size will be less, so IO time wont impact performance
3. In main driver pruning is done in multiple thread, added same for
index executor as now index executor will do the pruning
4. In case of block cache no need to send blockletdetailinfo object as
size is more and same can be constructed in executor from file footer

This closes #3281

Co-authored-by: kunal642 <kunalkapoor642@gmail.com>

Run Details

63647 of 83320 relevant lines covered (76.39%)

1.0 hits per line

Jobs

ID	Job ID	Ran	Files	Coverage
1	1784.1	21 Jun 2019 10:32AM UTC	0	76.39

Source Files on build 1784

Detailed source file information is not available for this build.

apache / carbondata / 1784
76%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 1784

apache / carbondata / 1784 76%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 1784

apache / carbondata / 1784
76%

README BADGES
x