dask / dask / 3560 / 1
53%
master: 53%

DEFAULT BRANCH: master
Ran
Files 76
Run time 6s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 93.763% (+0.02%) from 93.745%
PYTHON=2.7 NUMPY=1.10.4 PANDAS=0.17.0

push

travis-ci

mrocklin 
quantiles for repartitioning (#1261)

* Create a new method, `Series._repartition_quantiles`, to compute quantiles for repartitioning.

This calculates fewer quantiles than before, which will reduce the time required
to merge the quantiles together.  This is based on some hand-wavy heuristics, and
we introduce some random percentiles to try to avoid certain types of pathologies.
Give it a try!

* Use number of partitions proportional to chunk length.

This requires us to calculate the len of series before we create percentiles.

Also, slightly reduce the number of random samples.

* Handle case when number of percentiles would be more than number of elements

* Add docstring.

* Assume partitions are of comparable lengths.  Don't block to take a sum.

* Convert percentile `merge_sorted` to a reduction.  Add `upsample` keyword.

* Fix most failing tests.

The random percentiles is causing a test that compares names to fail.

* Use deterministic random seeds.

* Combine steps 1-4 into a single step.

These were very linear and inter-dependent anyway, so we might as well make
the graph simpler and reduce the number of tasks.

* docstring for `create_merge_tree`

* Docstring for `_prepare_percentile_merge`, which could use a better name.

* typo

* Don't close over df.name in lambda, because this closes over df, which is bad.

* Move code for determining new partition divisions using quantiles to a new file.

* Handle categoricals explicitly and convert values to native Python for more efficient serialization.

Serialization with `cloudpickle` should now be 10-20x faster than before, but it is
still the dominant factor.  `cloudpickle` remains about 50x slower than `cPickle`.

* update doctest.

* Add more docs, including a high level discussion in the module docstring.

* Don't create empty partitions if possible.  Also, merge equal values together.

* If undersampled, interpolate extra divisions if given a number dtype.

* Test interpolating divisions for undersampled data.

9155 of 9764 relevant lines covered (93.76%)

0.94 hits per line

Source Files on job 3560.1 (PYTHON=2.7 NUMPY=1.10.4 PANDAS=0.17.0)