apache / carbondata / 1595

76%

push

jenkins

[CARBONDATA-3321] Improved Single/Concurrent query Performance

What changes were proposed in this pull request?
Problem
Single/Concurrent query is slow when number of segment is more because of below root cause:

Memory footprint is more because of this gc is more and reducing query performance
Converting to Unsafe data map row to safe data map during pruning
Multi threaded pruning in case of non filter query is not supported
Retrieval from unsafe data map row is slower
Solution

Reduce memory footprint during query
Number of object created during query execution was high because of this GC was more and impacting query performance
Reduced memory footprint of temporary objects.
a) Added lazy decoding of data map row
b) Remove convetToSafe, this was used for converting UnsafeDataMapRow to DataMapRowImpl for
faster retrieval. Changed the Unsafe datamap row format for faster retrieval
C) Reduced Unnecessary string object creation

Added Multi thread pruning in case of non filter query
When number of segments/blocks are more pruning is slow for non filter query. As multi threaded
pruning in case of filter query is already supported added same for non filter query

Changed UnsafeDmStore storage format for faster retrieval.
Earlier only sequential access was allowed on UnsafeDataMapRow because of this converting
unsafedatamaprow to blocklet was slow and impacting query performance.
Changed format in UnsafeDataMapRow so random access can be done for faster retrieval

How was this patch tested?
Tested in 17 Node cluster with 5K/10K segments

This closes #3154

453 of 453 new or added lines in 26 files covered. (100.0%)

62880 of 79441 relevant lines covered (79.15%)

1.05 hits per line