pantsbuild / pants / 8788
0%
main: 93%

LAST BUILD BRANCH: fix/immutable-inputs-revalidation
DEFAULT BRANCH: main
Ran
Jobs 7
Files 0
Run time –
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
8788

push

travis-ci

stuhood 
[engine] Eager execution and fully declarative dependencies

This is a refactoring of the experimental engine to:
1. mix planning with execution
2. declare all dependencies of tasks ahead of time

----

- Planners are now `("output product type", "input product selection clause", "task function")` triples. The selection clause is an ANDed series of `Selector` instances which declare the dependencies of the function. See the bottom of `planners.py` for the updated examples.
- Don't create an execution graph before beginning to execute it: instead, grow a ProductGraph _while_ executing it.
    - While being more difficult to experiment with without causing sideeffects, this model allows for significant power: for one thing, it allows dependency lists to be created/determined by task code. This opens the door to tasks that do dependency inference by parsing input files (up to and including implementing BUILD file parsing inside the engine!).
- A ProductGraph is made up of nodes which each uniquely identify a computation.
    - The primary reason for the complexity of the `Node.step` API is that all of the dependencies of some nodes cannot be computed until some of their dependencies become available. In particular a `SelectDependencies` Node begins by requesting the product it needs to determine the dependency list; once that is available, it requests those dependencies as well.
- Rename `config_selectors` to `variants`, give them names, and make them transitive through the graph (also possible to specify in an address on the CLI)
    - One of the goals of this change is to allow for using the same system to solve three problems:
        1. thrift cross-compilation (as John already demonstrated)
        2. scala/javascript cross compilation (easy: left to the imagination)
        3. frozen ivy resolutions (by configuring a variant on binary targets that travels all the way down to their 3rdparty dependencies)

----

The visualizer tool continues to work, but now requires that the graph be executed before it can be displayed. Attached is the graph for:

    ./pants run src/python/pants/engine/exp/examples:viz -- \
      tests/python/pants_test/engine/exp/examples/scheduler_inputs \
      compile \
      src/java/codegen/selector:selected

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/105050996

Bugs closed: 2782, 2825

Reviewed at https://rbcommons.com/s/twitter/r/3339/
Jobs
ID Job ID Ran Files Coverage
1 8788.1 (CI_FLAGS="-cjlpn 'Various pants self checks'") 0
Travis Job 8788.1
4 8788.4 (CI_FLAGS="-fkmsrcjlp 'Python contrib tests'") 0
Travis Job 8788.4
5 8788.5 (CI_FLAGS="-fkmsrjlpn -i 0/6 'Python integration tests for pants - shard 1'") 0
Travis Job 8788.5
6 8788.6 (CI_FLAGS="-fkmsrjlpn -i 1/6 'Python integration tests for pants - shard 2'") 0
Travis Job 8788.6
7 8788.7 (CI_FLAGS="-fkmsrjlpn -i 2/6 'Python integration tests for pants - shard 3'") 0
Travis Job 8788.7
8 8788.8 (CI_FLAGS="-fkmsrjlpn -i 3/6 'Python integration tests for pants - shard 4'") 0
Travis Job 8788.8
10 8788.10 (CI_FLAGS="-fkmsrjlpn -i 5/6 'Python integration tests for pants - shard 6'") 0
Travis Job 8788.10
Source Files on build 8788
Detailed source file information is not available for this build.