apache / carbondata / 1290
76%

DEFAULT BRANCH: master
Ran
Jobs 1
Files 1014
Run time 49s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
1290

push

jenkins

manishgupta88 
[CARBONDATA-3088][Compaction] support prefetch for compaction

Current compaction performance is low. By adding logs to observe the compaction procedure, we found that in
`CarbonFactDataHandlerColumnar.addDataToStore(CarbonRow)`, it will wait about 30ms before submitting a new TablePage producer. Since the method
`addDataToStore` is called in single thread, it will result the waiting every 32000 records since it will collect 32000 records to form a TablePage.

To reduce the waiting time, we can prepare the 32000 records ahead. This an be achived using prefetch.

We will prepare two buffers, one will provide the records to the downstream (`addDataToStore`) and the other one will prepare the records
asynchronously. The first is called working buffer and the second is called backup buffer. Once working buffer is exhausted, the two buffers
will exchange their roles: the backup buffer will be the new working buffer and the old working buffer will be the new backup buffer and it
will be filled asynchronously.

Two parameters are involved for this feature:

1. carbon.detail.batch.size: This is an existed parameter and the default value is 100. This parameter controls the batch size of records that
return to the client. For normal query, it is OK to keep it as 100. But for compaction, since all the records will be operated, we suggest you
to set it to a larger value such as 32000. (32000 is the max rows for a table page that the down stream wants).

2. carbon.compaction.prefetch.enable: This is a new parameter and the default value is `false` (We may change it to `true` later). This
parameter controls whether we will prefetch the records for compation.

By using this prefetch feature, we can enhance the performance for compaction. More test results can be found in the PR description.

This closes #2906

55736 of 77040 relevant lines covered (72.35%)

0.98 hits per line

Jobs
ID Job ID Ran Files Coverage
1 1290.1 0
72.35
Source Files on build 1290
Detailed source file information is not available for this build.