pantsbuild / pants / 6587
0%
main: 93%

LAST BUILD BRANCH: add/port-frozen-ordered-set
DEFAULT BRANCH: main
Ran
Jobs 10
Files 381
Run time 13min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
6587

push

travis-ci

stuhood 
Introduce job prioritization to ExecutionGraph

See googledoc for details: https://docs.google.com/a/twitter.com/document/d/1BGVAR_AdGXsf08MT0nBLh8s-lYenx7NK7Kk7keD_gwg/edit?usp=sharing

PROBLEM:
The jobs are submitted to the worker pool in a sort of insertion order. If the submission order was based on the “size” or “importance” of the job, it could result in less total execution time.

SOLUTION:
Introduce a priority queue where ready-to-be-submitted jobs are waiting for the moment when a worker becomes available, being sorted by “importance”. When a job finishes, its unblocked dependees is not submitted to the worker pool, but rather put in the priority queue. When a worker becomes available (i.e. when any job is finished), the most “important” job is taken from the priority queue. There is an additional latency (before, ready jobs are submitted to the worker pool immediately; after, they are all first put in the priority queue, and only then the most prioritized ones are chosen and submitted).

Additionally, make tracking of ready-to-be-submitted jobs smarter with a counter in status table which is decremented with every finished dependency.

Testing Done:
$ ./pants test tests/python/pants_test:all
SUCCESS

CI green: https://travis-ci.org/megaserg/pants/builds/78164959

Bugs closed: 2109

Reviewed at https://rbcommons.com/s/twitter/r/2601/

17054 of 22745 relevant lines covered (74.98%)

1.41 hits per line

Jobs
ID Job ID Ran Files Coverage
1 6587.1 (CI_FLAGS="-cjlpn 'Various pants self checks'") 0
Travis Job 6587.1
2 6587.2 (CI_FLAGS="-fkmsrcn -u 0/2 'Unit tests for pants and pants-plugins - shard 1'") 0
69.99
 Travis Job 6587.2
3 6587.3 (CI_FLAGS="-fkmsrcn -u 1/2 'Unit tests for pants and pants-plugins - shard 2'") 0
71.12
 Travis Job 6587.3
4 6587.4 (CI_FLAGS="-fkmsrcjlp 'Python contrib tests'") 0
Travis Job 6587.4
5 6587.5 (CI_FLAGS="-fkmsrjlpn -i 0/6 'Python integration tests for pants - shard 1'") 0
Travis Job 6587.5
6 6587.6 (CI_FLAGS="-fkmsrjlpn -i 1/6 'Python integration tests for pants - shard 2'") 0
Travis Job 6587.6
7 6587.7 (CI_FLAGS="-fkmsrjlpn -i 2/6 'Python integration tests for pants - shard 3'") 0
Travis Job 6587.7
8 6587.8 (CI_FLAGS="-fkmsrjlpn -i 3/6 'Python integration tests for pants - shard 4'") 0
Travis Job 6587.8
9 6587.9 (CI_FLAGS="-fkmsrjlpn -i 4/6 'Python integration tests for pants - shard 5'") 0
Travis Job 6587.9
10 6587.10 (CI_FLAGS="-fkmsrjlpn -i 5/6 'Python integration tests for pants - shard 6'") 0
Travis Job 6587.10
Source Files on build 6587
Detailed source file information is not available for this build.