pantsbuild / pants / 6016
0%

DEFAULT BRANCH: master
Ran
Jobs 10
Files 348
Run time 14min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
6016

push

travis-ci

stuhood 
Add "omnivorous" ZincCompile to consume both java and scala sources

PROBLEM:
Currently, java and scala targets are chunked separately, and (with isolated strategy) a separate execution graph is created for every chunk by JavaZincCompile and ScalaZincCompile. These graphs are then executed sequentially, meaning there are unnecessary artificial barriers in the execution flow, increasing the total compilation time.

Also, when Zinc is run in Nailgun, the static final caches like analysisCache are preserved in Nailgun between Zinc invocations. Since every GroupMember is a NailgunTask, and NailgunTask uses NailgunExecutor parameterized by the task's class name (actually by the executor_workdir, which includes the task's class name), different GroupMembers use different NailgunExecutors, ergo different Nailgun instances. In particular, JavaZincCompile and ScalaZincCompile use two different Nailgun instances, which means storing two separate copies of Zinc's internal state. While not necessarily affecting performance, this increases memory pressure.

SOLUTION:
Define a new GroupMember which subclasses ZincCompile and consumes both java and scala sources. In GroupTask, it will claim all java and scala targets (the whole build graph), which will form a single chunk. When isolated strategy is used, the build graph is translated to the execution graph, which is executed in a single instance of the worker pool. Also, naturally, only one Nailgun instance is used to run Zinc for both languages, therefore only one copy of Zinc caches is stored.

Additionally, --enabled argument is added for jmake-based JavaCompile GroupMember, "True" by default.

RESULT:
The sequence of GroupMembers now is: [AptCompile, JavaCompile, ZincCompile].
1. All AnnotationProcessor Java targets are claimed by jmake-based AptCompile.
2a. When --no-compile-java-enabled is used, JavaCompile becomes a no-op. All remaining Java and Scala targets are claimed by ZincCompile and compiled in a singl... (continued)

30 of 30 new or added lines in 8 files covered. (100.0%)

15278 of 20827 relevant lines covered (73.36%)

1.38 hits per line

New Missed Lines in Diff

Lines Coverage ∆ File
1
53.95
 0.52% src/python/pants/backend/jvm/tasks/jvm_compile/java/java_compile.py
2
57.83
 0.69% src/python/pants/backend/jvm/register.py
2
43.59
 0.08% src/python/pants/backend/jvm/tasks/jvm_compile/jvm_compile.py
2
20.0
 0.0% src/python/pants/backend/jvm/tasks/jvm_compile/jvm_compile_isolated_strategy.py
2
30.99
 src/python/pants/backend/jvm/tasks/jvm_compile/scala/zinc_compile.py

Uncovered Existing Lines

Lines Coverage ∆ File
1
57.83
 0.69% src/python/pants/backend/jvm/register.py
Jobs
ID Job ID Ran Files Coverage
1 6016.1 (CI_FLAGS="-cjlpn 'Various pants self checks'") 0
Travis Job 6016.1
2 6016.2 (CI_FLAGS="-fkmsrcn -u 0/2 'Unit tests for pants and pants-plugins - shard 1'") 348
69.43
 Travis Job 6016.2
3 6016.3 (CI_FLAGS="-fkmsrcn -u 1/2 'Unit tests for pants and pants-plugins - shard 2'") 348
68.65
 Travis Job 6016.3
4 6016.4 (CI_FLAGS="-fkmsrcjlp 'Python contrib tests'") 0
Travis Job 6016.4
5 6016.5 (CI_FLAGS="-fkmsrjlpn -i 0/6 'Python integration tests for pants - shard 1'") 0
Travis Job 6016.5
6 6016.6 (CI_FLAGS="-fkmsrjlpn -i 1/6 'Python integration tests for pants - shard 2'") 0
Travis Job 6016.6
7 6016.7 (CI_FLAGS="-fkmsrjlpn -i 2/6 'Python integration tests for pants - shard 3'") 0
Travis Job 6016.7
8 6016.8 (CI_FLAGS="-fkmsrjlpn -i 3/6 'Python integration tests for pants - shard 4'") 0
Travis Job 6016.8
9 6016.9 (CI_FLAGS="-fkmsrjlpn -i 4/6 'Python integration tests for pants - shard 5'") 0
Travis Job 6016.9
10 6016.10 (CI_FLAGS="-fkmsrjlpn -i 5/6 'Python integration tests for pants - shard 6'") 0
Travis Job 6016.10
Source Files on build 6016
Coverage ∆ File Lines Relevant Covered Missed Hits/Line