bigdatagenomics / adam / 3068 / 1

Ran 17 Oct 2017 08:02PM UTC

Files 140

Run time 14s

Badge

Embed ▾

Committed 17 Oct 2017 08:02PM UTC coverage: 82.74% (+0.01%) from 82.729%

Job # 3068.1

Build Type

push

jenkins

Committed by Michael L Heuer

Commit Message

[ADAM-1680] Eliminate non-determinism in the ShuffleRegionJoin.

Resolves #1680. In the existing shuffle region join code, the partition
start/stop boundaries are determined by sorting the data and looking at the
coordinates of the first and last record on each partition. This is done via a
sampling process, which is fundamentally non-deterministic. Once these partition
bounds are picked, we replicate records from the right side of the join into the
partitions they overlap. There was a bug in this step that led to records being
dropped from the first partition of each contig (except for the first contig).
This was due to a bug in how the IntervalArray was being used to search for
repartitioning bounds.

This PR replaces the interval array with a map between contig names and the
partitions that contain their data. The first and last partition of each contig
are extended to the start/end of each contig.

Coverage Stats

6256 of 7561 relevant lines covered (82.74%)

0.83 hits per line

bigdatagenomics / adam / 3068 / 1

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 3068.1

bigdatagenomics / adam / 3068 / 1

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 3068.1

README BADGES
x