1772
76%

Ran 17 Jun 2019 05:43PM UTC

Jobs 1

Files 1061

Run time 1min

Badge

Embed ▾

pending completion

Build # 1772

Build Type

push

jenkins

Committed by xubo245

Commit Message

[CARBONDATA-3415] Merge index is not working for partition table. Merge index for partition table is taking significantly longer time than normal table.

Problem:
(1) Merge index is not working for partition table.
(2) Merge index for partition table is significantly more than the normal carbon table.

Root cause:
(1) Merge index event listener is moved to preStatusUpdateEvent in #3221 . But preStatusUpdateEvent is not triggered in case of partition table. Test case to validate merge index on partition table is also wrong. Not caught in the test builders.
(2) Currently, merge index job will trigger tasks like one segment one task. But for a partition table, there are partitions in a segments and merge index is for partitions. So per segment it has to iterate and merge the index files inside partitions, because of this the time is little more when the number of partitions are high. Number of Tasks = Number of Segments.

Solution:
(1) Correct the test case and trigger merge index listener for partition table.
(2) Parallelize the tasks launched to the partitions. Number of tasks = Number of partitions in a segment

This closes #3262

Run Details

64081 of 82880 relevant lines covered (77.32%)

1.01 hits per line

Jobs

ID	Job ID	Ran	Files	Coverage
1	1772.1	17 Jun 2019 05:43PM UTC	0	77.32

Source Files on build 1772

Detailed source file information is not available for this build.

apache / carbondata / 1772
76%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 1772

apache / carbondata / 1772 76%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 1772

apache / carbondata / 1772
76%

README BADGES
x