pantsbuild / pants / 4792
0%

DEFAULT BRANCH: master
Ran
Jobs 10
Files 324
Run time 15min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
4792

push

travis-ci

stuhood 
Zinc patches to improve roundtrip time

_(applies atop https://rbcommons.com/s/twitter/r/2177/)_

This is an import of two changes we made to improve the roundtrip time from pants to zinc during an incremental compile with the isolated strategy (and parallelism, although that's orthogonal.)

The worst case that we've found for incremental compile is many unchanged targets downstream of a changed target. In that case, zinc is invoked for each downstream target, but usually just needs to do enough inspection of inputs to decide that it doesn't need to compile anything.

With a graph of 900+ upstream targets, detecting upstream changes was initially filesystem bound.

We did a few things to fix this. First, we determined that we weren't actually [providing all relevant analysis to zinc](https://rbcommons.com/s/twitter/r/2155/), so it was recomputing in some cases. After fixing that, the next two hot things in the profile were:

1. In order to hit the in-memory analysis cache, zinc uses a SHA1 hash of upstream analysis files in order to decide that it could use them. This was admirably thorough, but almost certainly overkill. Instead, we use the analysis filename and timestamp here. It would be nice to upstream a patch to include a UUID or something in the header of each analysis file to make this bullet proof though.
2. To decide where classes have been defined, sbt uses a `definesClass` function that (for a given directory/jar) returns a lookup table from String=>Boolean which indicates whether the relevant directory/jar defines the class. Oddly enough, it was not using the upstream analysis to decide whether directories contained particular classes, and was instead stat'ing files and listing jars in all cases. So, instead, when we have valid upstream analysis this patch uses it to execute class lookups.

----

Together, these two changes took the roundtrip time for a noop zinc compile with 900+ upstream analysis files from 5+ seconds to <0.5 seconds.

Testing Done:
deployed internally for java and scala. could definitely use tests though.

Reviewed at https://rbcommons.com/s/twitter/r/2178/

13301 of 19253 relevant lines covered (69.09%)

0.69 hits per line

Jobs
ID Job ID Ran Files Coverage
1 4792.1 (CI_FLAGS="-cjlpnet 'Various pants self checks'") 0
Travis Job 4792.1
2 4792.2 (CI_FLAGS="-fkmsrcjlpn 'Test examples and testprojects'") 0
Travis Job 4792.2
3 4792.3 (CI_FLAGS="-fkmsrcnet 'Unit tests for pants and pants-plugins'") 324
69.09
 Travis Job 4792.3
4 4792.4 (CI_FLAGS="-fkmsrcjlpet 'Python contrib tests'") 0
Travis Job 4792.4
5 4792.5 (CI_FLAGS="-fkmsrjlpnet -i 6:0 'Python integration tests for pants - shard 1'") 0
Travis Job 4792.5
6 4792.6 (CI_FLAGS="-fkmsrjlpnet -i 6:1 'Python integration tests for pants - shard 2'") 0
Travis Job 4792.6
7 4792.7 (CI_FLAGS="-fkmsrjlpnet -i 6:2 'Python integration tests for pants - shard 3'") 0
Travis Job 4792.7
8 4792.8 (CI_FLAGS="-fkmsrjlpnet -i 6:3 'Python integration tests for pants - shard 4'") 0
Travis Job 4792.8
9 4792.9 (CI_FLAGS="-fkmsrjlpnet -i 6:4 'Python integration tests for pants - shard 5'") 0
Travis Job 4792.9
10 4792.10 (CI_FLAGS="-fkmsrjlpnet -i 6:5 'Python integration tests for pants - shard 6'") 0
Travis Job 4792.10
Source Files on build 4792
Coverage ∆ File Lines Relevant Covered Missed Hits/Line