apache / carbondata / 590

76%

push

jenkins

[CARBONDATA-2669] Local Dictionary Store Size optimisation and other function issues

Problems
Local dictionary store size issue.
When all column data is empty and columns are not present in sort columns local dictionary size was more than no dictionary dictionary store size.
Page level dictionary merging Issue
While merging the page used dictionary values in a blocklet it was missing some of the dictionary values, this is because, AND operation was done on bitset
Local Dictionary null values
Null value was not added in LV because of this new dictionary values was getting generated for null values
Local dictionary generator thread specific
Solution:
Added rle for unsorted dictionary values to reduce the size.
Now OR operation is performed while merging the dictionary values
Added LV for null values
Local dictionary generator task specific

This closes #2427

98 of 98 new or added lines in 11 files covered. (100.0%)

50893 of 71849 relevant lines covered (70.83%)

0.92 hits per line