ml6team / fondant / 5810677455

91%

push

github

Deactivate dask string conversion (#349)

Since version 2023-7-1, Dask DataFrames automatically convert object
data types to string[pyarrow]
([changelog](https://docs.dask.org/en/stable/changelog.html#v2023-7-1))

This leads to `Dask` trying to read image data as strings, which results
in the following error:
> unicodedecodeerror: 'utf-8' codec can't decode byte 0xff in position
0: invalid start byte

---

I tried several things to force Dask to keep this data as bytes:
- Write metadata file with schema in `to_parquet`
- Cast to bytes explicitly after reading using
`read_parquet.astype(...)`

But none of these worked. I believe this is because `pandas` represents
binary data using the `object` dtype, which is then automatically
converted to string by `Dask`.

---

So I believe our only option is to deactivate this behavior completely:

```python
dask.config.set({"dataframe.convert-string": False})
```

I don't think this has any downside for us, as textual data will still
be converted to strings since we explicitly set the schema based on the
component specification.

2 of 2 new or added lines in 1 file covered. (100.0%)

993 of 1100 relevant lines covered (90.27%)

3.6 hits per line