chanzuckerberg / miniwdl / 27527009684

96%

master: 97%

Build Type

push

github

Committed by web-flow

Commit Message

[WDL 1.2] Call cache v2 ensuring coherence with source-relative paths (#884)

The WDL 1.2 source-relative inputs feature creates new challenges for call cache coherence, because outputs depend on files/directories that are neither .wdl sources nor explicitly listed in the call inputs. Those dependency paths also can't be inferred statically in general, as they can be computed at runtime based on anything.

The solution here is to trace the additional paths used (at runtime) and include a list of those paths in the cache file (.additionalPaths) to be checked on potential cache hit. Interesting subtleties: (1) source-relative paths can be optional and absent, and in that case we need to remember their absence, so that creating one invalidates the cache; (2) a workflow's call cache entry must include all of its calls' additionalPaths as well as those it uses directly.

This requires a new cache format version, so we also make some other minor improvements, most notably rolling the format version into the cache key digest to ensure older miniwdl versions won't use them.

Coverage Stats

168 of 169 new or added lines in 6 files covered. (99.41%)

5 existing lines in 2 files now uncovered.

8654 of 9033 relevant lines covered (95.8%)

0.96 hits per line