mrocklin / dask / 2218

92%

master: 94%

push

travis-ci

Avoid sorting large stacks in order

When performning task ordering we sort tasks based on the
number of dependents/dependencies they have.  This is critical to
low-memory processing.

However, sometimes individual tasks have millions of dependencies,
for which an n*log(n) sort adds significant overhead.  In these cases
we give up on sorting, and just hope that the tasks are well ordered
naturally (such as is often the case in Python 3.6+ due to sorted
dicts and the natural ordering that exists when constructing common
graphs)

See https://github.com/pangeo-data/pangeo/issues/150#issuecomment-373066066
for a real-world case

9 of 9 new or added lines in 1 file covered. (100.0%)

14635 of 15903 relevant lines covered (92.03%)

0.92 hits per line