apache / carbondata / 783

76%

push

jenkins

[HOTFIX] Removed file existence check to improve dataMap loading performance

Problem
DataMap loading performance degraded after adding file existence check.

Analysis
When carbonIndex file is read and carbondata file path to its metadata Info map is prepared, file physical existence is getting checked every time which in case of HDFS file system is a namenode call. This degrades the dataMap loading performance. This is done to avoid failures for

Handle IUD scenario where after delete operation carbondata file is deleted but the entry still exists in index file.
Fix
Modified code to check for physical file existence only in case when any IUD operation has happened on the table

This closes #2560

6 of 6 new or added lines in 1 file covered. (100.0%)

57118 of 70293 relevant lines covered (81.26%)

1.08 hits per line