apache / carbondata / 833

76%

push

jenkins

[Documentation] [Unsafe Configuration] Added carbon.unsafe.driver.working.memory.in.mb parameter to differentiate between driver and executor unsafe memory

Added carbon.unsafe.driver.working.memory.in.mb parameter to differentiate between driver and executor unsafe memory

Usually in production scenarios driver memory will be less than the executor memory. Now we are using unsafe for caching block/blocklet dataMap in driver. Current unsafe memory configured for executor is getting used for driver also which is not a good idea.
Therefore it is required to separate out driver and executor unsafe memory.
You can observe the same in spark configuration also that spark has given different parameters for configuring driver and executor memory overhead to control the unsafe memory usage.
spark.yarn.driver.memoryOverhead and spark.yarn.executor.memoryOverhead

This closes #2595

14 of 14 new or added lines in 1 file covered. (100.0%)

57051 of 70480 relevant lines covered (80.95%)

1.08 hits per line