7397728517
91%

Ran 03 Jan 2024 01:16PM UTC

Jobs 0

Files 0

Run time –

Badge

Embed ▾

pending completion

Build # 7397728517

Build Type

push

github

Committed by

web-flow

Commit Message

Propagate dataset schema eagerly (#748)

This PR propagates the dataset schema eagerly when an operation is
applied. This is useful for iterative development in a notebook, since
the schema of the current dataset can be inspected.

Result:

```
In [2]: from fondant.pipeline import Pipeline
In [3]: import pyarrow as pa
In [4]: pipeline = Pipeline(name="my-pipe", base_path="data")
In [5]: text_data = pipeline.read(
   ...:     "load_from_hf_hub",
   ...:     arguments={
   ...:         "dataset_name": "wikitext@~parquet",
   ...:         "n_rows_to_load": 1000,
   ...:     },
   ...:     produces={
   ...:         "text": pa.string(),
   ...:     },
   ...: )
In [6]: text_data.fields
Out[6]: {'text': Field({'name': 'text', 'type': Type(DataType(string)), 'location': '/my-pipe-20231227152908/load_from_hugging_face_hub'})}
In [7]: chunk_data = text_data.apply(
   ...:     "chunk_text",
   ...:     arguments={
   ...:         "chunk_size": 256,
   ...:         "chunk_overlap": 32,
   ...:     },
   ...: )
In [8]: chunk_data.fields
Out[8]: 
{'text': Field({'name': 'text', 'type': Type(DataType(string)), 'location': '/my-pipe-20231227152920/chunk_text'}),
 'original_document_id': Field({'name': 'original_document_id', 'type': Type(DataType(string)), 'location': '/my-pipe-20231227152920/chunk_text'})}
```

Run Details

Source Files on build 7397728517

Detailed source file information is not available for this build.

ml6team / fondant / 7397728517
91%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Source Files on build 7397728517

ml6team / fondant / 7397728517 91%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Source Files on build 7397728517

ml6team / fondant / 7397728517
91%

README BADGES
x