4
92%
main: 92%

Ran 23 Nov 2023 11:41AM UTC

Files 308

Run time 9s

Badge

Embed ▾

Committed 23 Nov 2023 11:26AM UTC coverage: 20.461% (-0.001%) from 20.462%

Job # 6969071273.4

Build Type

push

github

Committed by

web-flow

Commit Message

feat: PreProcessor split by token (tiktoken & Hugging Face) (#5276)

* #4983 implemented split by token for tiktoken tokenizer

* #4983 added unit test for tiktoken splitting

* #4983 implemented and added a test for splitting documents with HuggingFace tokenizer

* #4983 added support for passing HF model names (instead of objects) and added an example to the HF token splitting test

* mocked HTTP model loading in unit tests, fixed pylint error

* fix lossy tokenizers splitting, use LazyImport, ignore UnicodeEncodeError for tiktoken

* reno

* rename reno file

---------

Co-authored-by: Stefano Fiorucci <44616784+anakin87@users.noreply.github.com>
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>

Run Details

5428 of 26529 relevant lines covered (20.46%)

0.2 hits per line

deepset-ai / haystack / 6969071273 / 4
92%
main: 92%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job agents - 6969071273.4

deepset-ai / haystack / 6969071273 / 4 92% main: 92%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job agents - 6969071273.4

deepset-ai / haystack / 6969071273 / 4
92%
main: 92%

README BADGES
x