1440
76%

Ran 08 Jan 2019 09:22AM UTC

Jobs 1

Files 1024

Run time 54s

Badge

Embed ▾

pending completion

Build # 1440

Build Type

push

jenkins

Committed by ravipesala

Commit Message

[CARBONDATA-3219] Support range partition the input data for local_sort/global sort data loading

For global_sort/local_sort table, load data command add RANGE_COLUMN option

load data inpath '<path>' into table <table name>
options('RANGE_COLUMN'='<a column>')
when we know the total size of input data, we can calculate the number of the partitions.
load data inpath '<path>' into table <table name>
options('RANGE_COLUMN'='<a column>', 'global_sort_partitions'='10')
when we don't know the total size of the input data, we can give the size of each partition.
load data inpath '<path>' into table <table name>
options('RANGE_COLUMN'='<a column>', 'scale_factor'='10')
it will calcute the number of the partitions as follows.

splitSize =  Math.max(blocklet_size, (block_size - blocklet_size)) * scale_factor
numPartitions = Math.ceil(total size / splitSize)
Limitation:

not support insert into, support only load data command,
not support multiple range columns, support only one range column
exists data skew

This closes #2971

Run Details

62214 of 78637 relevant lines covered (79.12%)

1.05 hits per line

Jobs

ID	Job ID	Ran	Files	Coverage
1	1440.1	08 Jan 2019 09:22AM UTC	0	79.12

Source Files on build 1440

Detailed source file information is not available for this build.

apache / carbondata / 1440
76%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 1440

apache / carbondata / 1440 76%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 1440

apache / carbondata / 1440
76%

README BADGES
x