6969071273
90%

Ran 23 Nov 2023 11:30AM UTC

Jobs 9

Files 308

Run time 43s

Badge

Embed ▾

Committed 23 Nov 2023 11:26AM UTC coverage: 40.164% (+0.08%) from 40.086%

Build # 6969071273

Build Type

push

github

Committed by

web-flow

Commit Message

feat: PreProcessor split by token (tiktoken & Hugging Face) (#5276)

* #4983 implemented split by token for tiktoken tokenizer

* #4983 added unit test for tiktoken splitting

* #4983 implemented and added a test for splitting documents with HuggingFace tokenizer

* #4983 added support for passing HF model names (instead of objects) and added an example to the HF token splitting test

* mocked HTTP model loading in unit tests, fixed pylint error

* fix lossy tokenizers splitting, use LazyImport, ignore UnicodeEncodeError for tiktoken

* reno

* rename reno file

---------

Co-authored-by: Stefano Fiorucci <44616784+anakin87@users.noreply.github.com>
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>

Run Details

10655 of 26529 relevant lines covered (40.16%)

2.14 hits per line

Uncovered Existing Lines

Lines	Coverage	∆	File
4	81.82	0.87%	nodes/preprocessor/base.py
58	83.91	0.61%	nodes/preprocessor/preprocessor.py

Jobs

ID	Job ID	Ran	Files	Coverage
1	utils - 6969071273.1	23 Nov 2023 11:31AM UTC	308	18.48	GitHub Action Run
2	modeling - 6969071273.2	23 Nov 2023 11:31AM UTC	308	18.35	GitHub Action Run
3	pipelines - 6969071273.3	23 Nov 2023 11:31AM UTC	308	21.75	GitHub Action Run
4	agents - 6969071273.4	23 Nov 2023 11:30AM UTC	308	20.46	GitHub Action Run
5	cli - 6969071273.5	23 Nov 2023 11:30AM UTC	308	18.05	GitHub Action Run
6	document_stores - 6969071273.6	23 Nov 2023 11:31AM UTC	308	23.1	GitHub Action Run
7	others - 6969071273.7	23 Nov 2023 11:31AM UTC	308	20.23	GitHub Action Run
8	prompt - 6969071273.8	23 Nov 2023 11:31AM UTC	308	22.81	GitHub Action Run
9	nodes - 6969071273.9	23 Nov 2023 11:32AM UTC	308	28.45	GitHub Action Run

deepset-ai / haystack / 6969071273
90%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Uncovered Existing Lines

Jobs

Source Files on build 6969071273

deepset-ai / haystack / 6969071273 90%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Uncovered Existing Lines

Jobs

Source Files on build 6969071273

deepset-ai / haystack / 6969071273
90%

README BADGES
x