pantsbuild / pants / 9463
0%
main: 93%

LAST BUILD BRANCH: 2.33.x
DEFAULT BRANCH: main
Ran
Jobs 7
Files 0
Run time –
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
9463

push

travis-ci

peiyuwang 
[engine] cache StepResult under StepRequest

This review adds:

* `Cache` that caches `StepResult` for `StepRequest` by storing the blobs in the main storage like the rest `subjects` and `states`, in addition, the request key to result key mappings are stored in a separate lookup table. Therefore two lookups are needed for cache hits.
* Lmdb support for creating child database, for secondary index use case.
* Add `add_mapping`, `get_mapping` interface to `Storage`, Cache can choose to have a different location of the subjects storage, or can be the same, but implementation is shared, this uniformation allows for sharing future GC logic for example.
* Hooks in various execution environment to check cache and save to cache: `LocalSerialEngine`, `LocalMultiprocessEngine`, `_stateful_pool_loop`.
* Use `OrderedDict` for `Struct.kwargs` to ensure consistent hashing.

Notes:

* Now since the main blobs `states` and `subjects` are already in keys, both `StepRequest` and `StepResult` are light, so we can afford to do this inside engine before `step` calls, essentially doing cache checks earliest possible.

TODO:

* There is some weirdness from tuples, that even for the *same* tuple for example, ```((u'thrift', u'apache_java'),)``` may produce different hash. I tried if flatten the tuple pickle will be produce consistent result. Still need some work.

Testing Done:
https://travis-ci.org/peiyuwang/pants/builds/114562555
https://travis-ci.org/peiyuwang/pants/builds/116460440

Reviewed at https://rbcommons.com/s/twitter/r/3494/
Jobs
ID Job ID Ran Files Coverage
1 9463.1 (CI_FLAGS="-cjlpn 'Various pants self checks'") 0
Travis Job 9463.1
4 9463.4 (CI_FLAGS="-fkmsrcjlpa 'Python contrib tests'") 0
Travis Job 9463.4
5 9463.5 (CI_FLAGS="-fkmsrjlpn -i 0/6 'Python integration tests for pants - shard 1'") 0
Travis Job 9463.5
6 9463.6 (CI_FLAGS="-fkmsrjlpn -i 1/6 'Python integration tests for pants - shard 2'") 0
Travis Job 9463.6
7 9463.7 (CI_FLAGS="-fkmsrjlpn -i 2/6 'Python integration tests for pants - shard 3'") 0
Travis Job 9463.7
8 9463.8 (CI_FLAGS="-fkmsrjlpn -i 3/6 'Python integration tests for pants - shard 4'") 0
Travis Job 9463.8
9 9463.9 (CI_FLAGS="-fkmsrjlpn -i 4/6 'Python integration tests for pants - shard 5'") 0
Travis Job 9463.9
Source Files on build 9463
Detailed source file information is not available for this build.