1
76%
master: 76%

Ran 01 Nov 2018 10:07AM UTC

Files 1026

Run time 9min

Badge

Embed ▾

Committed 01 Nov 2018 10:07AM UTC coverage: 80.116% (+0.05%) from 80.066%

Job # 1230.1

Build Type

push

jenkins

Committed by Jacky Li

Commit Message

[CARBONDATA-3052] Improve drop table performance by reducing the namenode RPC calls during physical deletion of files

Problem
Current drop table command takes more than 1 minute to delete 3000 files during drop table operation from HDFS

Analysis
Even though we are using HDFS file system we are explicitly we are recursively iterating through the table folders and deleting each file. For each file deletion and file listing one rpc call is made to namenode. To delete 3000 files 3000 rpc calls are made to namenode for file deletion and few more rpc calls for file listing in each folder.

Solution
HDFS provides an API for deleting all folders and files recursively for a given path in a single RPC call. Use that API and improve the drop table operation performance.

Result: After these code changes drop table operation time to delete 3000 files from HDFS has reduced from 1 minute to ~2 sec.

This closes #2868

Coverage Stats

61811 of 77152 relevant lines covered (80.12%)

1.06 hits per line

apache / carbondata / 1230 / 1
76%
master: 76%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 1230.1

apache / carbondata / 1230 / 1 76% master: 76%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 1230.1

apache / carbondata / 1230 / 1
76%
master: 76%

README BADGES
x