1313
76%

Ran 27 Nov 2018 08:37AM UTC

Jobs 1

Files 1017

Run time 44s

Badge

Embed ▾

pending completion

Build # 1313

Build Type

push

jenkins

Committed by ravipesala

Commit Message

[CARBONDATA-3122]CarbonReader memory leak

Issue Detail
CarbonReader has List of initialized RecordReader for each Split and each split holds page data till the reference of RecordReader is present in the List . Same is applicable for GC once user comes out from his/her calling method ( not cleaned even in close() ) but till then from each split , last page will be in memory which is not correct. For ex. if 1K carbon files then last page ( ~32K * 100 ,size if 100 String columns in memory ) of each file will be in memory till last split so total ~3GB memory will be occupied ( 1K * 32K * 100 .
Check heap dump of 3 split after reader.close() is called ,It is be seen that currentreader+all list reader are still holding memory

Solution

Once reader is finished assign currentReader to null in RecordReader List.
OR
Assign future object as null in org.apache.carbondata.core.scan.processor.DataBlockIterator#close()
Solution 2 is adopted so that it will give benefit to other than CarbonReader Flow.

This closes #2944

Run Details

61607 of 77468 relevant lines covered (79.53%)

1.05 hits per line

Jobs

ID	Job ID	Ran	Files	Coverage
1	1313.1	27 Nov 2018 08:37AM UTC	0	79.53

Source Files on build 1313

Detailed source file information is not available for this build.

apache / carbondata / 1313
76%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 1313

apache / carbondata / 1313 76%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 1313

apache / carbondata / 1313
76%

README BADGES
x