1230
76%

Ran 01 Nov 2018 10:07AM UTC

Jobs 1

Files 1026

Run time 10min

Badge

Embed ▾

pending completion

Build # 1230

Build Type

push

jenkins

Committed by Jacky Li

Commit Message

[CARBONDATA-3052] Improve drop table performance by reducing the namenode RPC calls during physical deletion of files

Problem
Current drop table command takes more than 1 minute to delete 3000 files during drop table operation from HDFS

Analysis
Even though we are using HDFS file system we are explicitly we are recursively iterating through the table folders and deleting each file. For each file deletion and file listing one rpc call is made to namenode. To delete 3000 files 3000 rpc calls are made to namenode for file deletion and few more rpc calls for file listing in each folder.

Solution
HDFS provides an API for deleting all folders and files recursively for a given path in a single RPC call. Use that API and improve the drop table operation performance.

Result: After these code changes drop table operation time to delete 3000 files from HDFS has reduced from 1 minute to ~2 sec.

This closes #2868

Run Details

61811 of 77152 relevant lines covered (80.12%)

1.06 hits per line

Jobs

ID	Job ID	Ran	Files	Coverage
1	1230.1	01 Nov 2018 10:07AM UTC	0	80.12

Source Files on build 1230

Detailed source file information is not available for this build.

apache / carbondata / 1230
76%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 1230

apache / carbondata / 1230 76%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 1230

apache / carbondata / 1230
76%

README BADGES
x