dask / dask / 3560
53%

DEFAULT BRANCH: master
Ran
Jobs 2
Files 76
Run time 3min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
3560

push

travis-ci

mrocklin 
quantiles for repartitioning (#1261)

* Create a new method, `Series._repartition_quantiles`, to compute quantiles for repartitioning.

This calculates fewer quantiles than before, which will reduce the time required
to merge the quantiles together.  This is based on some hand-wavy heuristics, and
we introduce some random percentiles to try to avoid certain types of pathologies.
Give it a try!

* Use number of partitions proportional to chunk length.

This requires us to calculate the len of series before we create percentiles.

Also, slightly reduce the number of random samples.

* Handle case when number of percentiles would be more than number of elements

* Add docstring.

* Assume partitions are of comparable lengths.  Don't block to take a sum.

* Convert percentile `merge_sorted` to a reduction.  Add `upsample` keyword.

* Fix most failing tests.

The random percentiles is causing a test that compares names to fail.

* Use deterministic random seeds.

* Combine steps 1-4 into a single step.

These were very linear and inter-dependent anyway, so we might as well make
the graph simpler and reduce the number of tasks.

* docstring for `create_merge_tree`

* Docstring for `_prepare_percentile_merge`, which could use a better name.

* typo

* Don't close over df.name in lambda, because this closes over df, which is bad.

* Move code for determining new partition divisions using quantiles to a new file.

* Handle categoricals explicitly and convert values to native Python for more efficient serialization.

Serialization with `cloudpickle` should now be 10-20x faster than before, but it is
still the dominant factor.  `cloudpickle` remains about 50x slower than `cPickle`.

* update doctest.

* Add more docs, including a high level discussion in the module docstring.

* Don't create empty partitions if possible.  Also, merge equal values together.

* If undersampled, interpolate extra divisions if given a number dtype.

* Test interpolating divisions for undersampled data.

9157 of 9764 relevant lines covered (93.78%)

1.87 hits per line

Jobs
ID Job ID Ran Files Coverage
1 3560.1 (PYTHON=2.7 NUMPY=1.10.4 PANDAS=0.17.0) 0
93.76
 Travis Job 3560.1
2 3560.2 (PYTHON=2.7 NUMPY=1.11.0 PANDAS=0.18.0) 0
93.12
 Travis Job 3560.2
Source Files on build 3560
Detailed source file information is not available for this build.