1
76%
master: 76%

Ran 26 Aug 2018 07:51PM UTC

Files 991

Run time 48s

Badge

Embed ▾

Committed 26 Aug 2018 07:51PM UTC coverage: 75.545% (+0.004%) from 75.541%

Job # 913.1

Build Type

push

jenkins

Committed by ravipesala

Commit Message

[CARBONDATA-2874] Support SDK writer as thread safe api

Problem: Currently CarbonWriter.write() not a thread safe. if multiple threads calls .write() for one writer.
Data count inconsistency is observed.

root casue: As all the threads are writing to same batch of blocking queue. need to synchronize this. Else one thread data overwrite the other thread data.

Solution:
a) DataLoadExecutor is using only one iterator, take number of threads as input and internally create that many iterator to loop over the data. This will reduce the blocking time of queue as each iterator has its own queue.
b) InputProcessor step is taking only default 2 cores (2 thread) for data load in SDK flow, can use the same number as number of threads created by user.
c) writer step is using only 2 cores (2 thread). can use the same number as number of threads created by user.

This closes #2653

Run Details

53716 of 71105 relevant lines covered (75.54%)

0.98 hits per line

apache / carbondata / 913 / 1
76%
master: 76%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 913.1

apache / carbondata / 913 / 1 76% master: 76%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 913.1

apache / carbondata / 913 / 1
76%
master: 76%

README BADGES
x