ml6team / fondant / 6380739897

91%

push

github

Fix caching writes (#469)

PR that fixes issues related to caching subsequent components. 

Since cached manifests contain the same run id as the pipeline where
they originated from, they can sometimes override existing data if the
component needs to run again (e.g. cached first component loaded into
second component which has different arguments and thus needs to
execute). This PR makes sure that:

* The run-id of the loaded **input manifest** matches the one from the
**current pipeline run**, this ensures that all new subsets from the
component being executed are written under the corresponding path of
that pipeline run
* The run-id of the **output manifest** is that of the **cached
manifest** (if it applies). This ensures that subsequent components are
able to tell whether the previous component execution was cached or not
[link to
code](https://github.com/ml6team/fondant/blob/39436e7fa/src/fondant/executor.py#L294)

6 of 6 new or added lines in 3 files covered. (100.0%)

1167 of 1260 relevant lines covered (92.62%)

3.7 hits per line