apache / carbondata / 913

76%

push

jenkins

[CARBONDATA-2874] Support SDK writer as thread safe api

Problem: Currently CarbonWriter.write() not a thread safe. if multiple threads calls .write() for one writer.
Data count inconsistency is observed.

root casue: As all the threads are writing to same batch of blocking queue. need to synchronize this. Else one thread data overwrite the other thread data.

Solution:
a) DataLoadExecutor is using only one iterator, take number of threads as input and internally create that many iterator to loop over the data. This will reduce the blocking time of queue as each iterator has its own queue.
b) InputProcessor step is taking only default 2 cores (2 thread) for data load in SDK flow, can use the same number as number of threads created by user.
c) writer step is using only 2 cores (2 thread). can use the same number as number of threads created by user.

This closes #2653

69 of 69 new or added lines in 7 files covered. (100.0%)

53716 of 71105 relevant lines covered (75.54%)

0.98 hits per line