ml6team / fondant / 6081412015

91%

Build Type

push

github

Committed by web-flow

Commit Message

Update caching strategy (#407)

PR that updates the way we check for cached executions. Previous
proposal was to use glob patterns to check for a manifest under a
certain directory, but this caused it to be too slow since there glob
patterns behave differently in remote filesystems where they check for
all nested directories. More details about the issue
[here](https://github.com/ml6team/fondant/issues/368#issuecomment-1700824541).

This PR changes this by writing a reference to a manifest file in a
folder separate from the subsets.

```
base_path/
├── pipelines/
    └── <pipeline_name>/
        ├── <pipeline_id>/
        │   ├── component_1/
        │   │   ├── manifest.json
        │   │   └── subsets/
        │   │       ├── <subset_1>/
        │   │       │   └── part.0.parquet
        │   │       └── <subset_2>/
        │   │           └── part.0.parquet
        │   └── component_2/
        │       └── manifest.json
        ├── cache/   
            ├── <hash_key1>
            ├── <hash_key2>
            ├── <hash_key3>
```

In consequence, we also reverted back to using basic fsspec
functionality which makes it easier to resolve files without having to
explicitly to which protocol they belong to. Closing [this
ticket](https://github.com/ml6team/fondant/pull/402) in favor of this
one.

Tested with all different combinations (remote/local runner/basepath)

Coverage Stats

19 of 29 new or added lines in 2 files covered. (65.52%)

3 existing lines in 1 file now uncovered.

1119 of 1208 relevant lines covered (92.63%)

3.7 hits per line