pantsbuild / pants / 9144
0%
main: 93%

LAST BUILD BRANCH: graph/entry-storage-indirection
DEFAULT BRANCH: main
Ran
Jobs 7
Files 0
Run time –
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
9144

push

travis-ci

stuhood 
[engine] First round of work for 'native' filesystem support

Extract an `engine.exp.fs` module to handle all filesystem operations. Currently it contains the (existing) support for recursive directory walks, and some new support for requesting the file contents of literal globs (`PathGlobs`). Support for recursive globs (ie, `src/**/*.java`) will come in the next half of the review.

This is the first half of [#2946](https://github.com/pantsbuild/pants/issues/2946), but it shuffles a bunch of modules around, so I wanted to get it out sooner rather than later.

- Extract `nodes`, `selectors`, `sources`, and `fs` modules in order to shrink `scheduler`, and to allow `fs` to depend on `selectors` and expose its own tasks.
   - Many things in the `fs` package will eventually be "native", in that they will be implemented without the end-user task API. But for now it's easiest to experiment with them as tasks.
- Add PathGlobs, which consumes the (apparently static!) methods of `wrapped_globs.(Globs,RGlobs,ZGlobs)` to normalize inputs into easily parsed 'filespecs'.
- Add `Sources.path_globs`, which can be projected from a Sources object in order to request actual Paths or FileContents.
    - This is used in this review to power all filesystem access for BUILD file parsing and scala dep inference.
- Remove outdated methods of `Sources` and `Target` which executed graph and filesystem walks.
- Add `Locatable` interface, to mark a `Serializable` as needing to receive the 'spec_path' it was parsed from.
    - Structs sometimes do not have a name (and so may not have an Address), but all Structs were parsed from some namespace. The purpose of the interface is to avoid storing the location a Struct was parsed from in cases where it isn't needed later.
- Add an explicit `ProjectionNode` to allow `SelectProjection` to work in cases where a Subject is not already of the necessary input type, but might be after selection.
    - This is a case very similar to `Dependencies`, where we first select a 'deps_product', and then project its dependencies. There is obviously still room to improve the selection syntax.
- Add another visualizer entrypoint to support executing the equivalent of "ls" using the engine (and visualizing the result).
    - Since `PathGlobs` only supports literals at the moment, this basically just serves to tell you whether a file exists. In the next review, this will additionally support expanding globs... see comments in `PathGlobs.create`.

And: demo time. Attached is a rendering of the following command, which specifies a buildroot of `.` (ie, the pants working directory), and uses only engine tasks to confirm that `examples/README.md` exists. Pheew.

    ./pants run src/python/pants/engine/exp/examples:viz-fs -- . walk examples/README.md

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/111341847

Bugs closed: 2946, 2951

Reviewed at https://rbcommons.com/s/twitter/r/3488/
Jobs
ID Job ID Ran Files Coverage
1 9144.1 (CI_FLAGS="-cjlpn 'Various pants self checks'") 0
Travis Job 9144.1
4 9144.4 (CI_FLAGS="-fkmsrcjlp 'Python contrib tests'") 0
Travis Job 9144.4
5 9144.5 (CI_FLAGS="-fkmsrjlpn -i 0/6 'Python integration tests for pants - shard 1'") 0
Travis Job 9144.5
6 9144.6 (CI_FLAGS="-fkmsrjlpn -i 1/6 'Python integration tests for pants - shard 2'") 0
Travis Job 9144.6
8 9144.8 (CI_FLAGS="-fkmsrjlpn -i 3/6 'Python integration tests for pants - shard 4'") 0
Travis Job 9144.8
9 9144.9 (CI_FLAGS="-fkmsrjlpn -i 4/6 'Python integration tests for pants - shard 5'") 0
Travis Job 9144.9
10 9144.10 (CI_FLAGS="-fkmsrjlpn -i 5/6 'Python integration tests for pants - shard 6'") 0
Travis Job 9144.10
Source Files on build 9144
Detailed source file information is not available for this build.