ml6team / fondant / 6010448952

91%

push

github

Implement caching (#387)

PR that implements caching: 

* The component cache is estimated at compile time based on the images
tag, input parameters, component spec and component resources (GPU,
nodepool). Caching is enabled by default at the componentOp level and
disabled if the image has the `latest` tag

* At runtime, the cache key is checked against the base path to check
for matching executions based on a the cache key (currently appended to
the manifest name).

* If multiple matching executions are found, we return the latest one.
This makes it easier to develop. For example, if we're working with the
`dev` tag and we disable caching to force the execution of a new version
of the component (scenario where the spec and parameters are the same).
Then when we enable caching again, we would only load the manifest from
the latest execution.
 
* A component is cached only if it's enabled at the Op level, a matching
manfiest is found and previous component's run is cached. To check
whether the previous component is cached or not. We check if the run-id
of the previous component (input manifest) has the same run-id as the
current component. If they match, it means that the previous component
had to execute (manifest not loaded from previous pipeline run). Note
that this implies that we don't change the loaded manifest's run-id to
that of the current pipeline run. This might not be ideal for lineage
tracking but I see no other way of doing this check.

54 of 65 new or added lines in 5 files covered. (83.08%)

1120 of 1206 relevant lines covered (92.87%)

3.71 hits per line