ml6team / fondant / 7714175453
91%

DEFAULT BRANCH: main
Ran
Jobs 4
Files 20
Run time 3s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 91.799% (-0.04%) from 91.841%
7714175453

push

github

web-flow 
Start from dataset schema for lightweight python component `consumes` (#789)

Fixes #785 

Opening this as a draft PR since it's not yet clear to me what the
desired behavior is.

I'll be using the "inner" / "outer" terminology which we already use in
our `OperationSpec` class to explain. "inner" schema's are the schema's
that the Python component consumes / produces. "outer" schema's are the
schema's that the `DataIO` layer consumes / produces.

For docker components, the logic works as follows:
1. The `consumes` section in the component spec is the "inner" schema
2. We leverage the `consumes` argument of the `apply` method to
calculate the "outer" schema from the "inner" schema.

For lightweight python components, we do not have a component spec to
start from. So what I currently implemented is this:
1. We start from the dataset schema and reverse alter it with the
`consumes` argument to calculate the "inner" schema.
2. We leverage the `consumes` argument of the `apply` method to
calculate the "outer" schema from the "inner" schema.

This works, but has one big downside. Since we start from the dataset
schema, the calculated "inner" / "outer" consumes contain all the fields
in the dataset. In other words, the lack of a component spec removes the
ability to select which columns from the dataset to load. Since this is
an important part of our optimization, I think we need to find a way
around this.

My best idea at this time is to expand the `lightweight_component`
decorator to add support for this. But curious to hear if anyone has
other ideas.

---------

Co-authored-by: Philippe Moussalli <philippe.moussalli95@gmail.com>
Co-authored-by: Georges Lorré <35808396+GeorgesLorre@users.noreply.github.com>

1959 of 2134 relevant lines covered (91.8%)

3.67 hits per line

Jobs
ID Job ID Ran Files Coverage
1 test-3.11 - 7714175453.1 0
91.71
2 test-3.8 - 7714175453.2 0
91.79
3 test-3.9 - 7714175453.3 0
91.7
4 test-3.10 - 7714175453.4 0
91.71
Source Files on build 7714175453
Detailed source file information is not available for this build.