3
91%
main: 91%

Ran 24 Feb 2024 10:52AM UTC

Files 20

Run time 0s

Badge

Embed ▾

Committed 24 Feb 2024 10:45AM UTC coverage: 91.425% (-0.004%) from 91.429%

Job # 8030025822.3

Build Type

push

github

Committed by

web-flow

Commit Message

Don't write metadata file (#875)

We reintroduced writing the metadata file in #864 to preserve the
divisions of the data when writing and reading again. We turned this
behavior off in the past, but without proper documentation of the
reason.

I'm now running into issues with Dask workers dying when writing large
datasets though, presumably because of the metadata file, as documented
in these Dask issues:
- https://github.com/dask/dask/issues/6600
- https://github.com/dask/dask/issues/3873
- https://github.com/dask/dask/issues/8901

Also, while I ran into issues with the preservation of divisions before,
I can't reproduce this locally with a small example. Let's turn writing
metadata off again and validate if we are still having issues with this.

Run Details

2015 of 2204 relevant lines covered (91.42%)

0.91 hits per line

ml6team / fondant / 8030025822 / 3
91%
main: 91%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job test-3.9 - 8030025822.3

ml6team / fondant / 8030025822 / 3 91% main: 91%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job test-3.9 - 8030025822.3

ml6team / fondant / 8030025822 / 3
91%
main: 91%

README BADGES
x