6969071273
92%

Ran 23 Nov 2023 11:30AM UTC

Jobs 9

Files 308

Run time 43s

Badge

Embed ▾

Committed 23 Nov 2023 11:26AM UTC coverage: 40.164% (+0.08%) from 40.086%

Build # 6969071273

Build Type

push

github

Committed by

web-flow

Commit Message

feat: PreProcessor split by token (tiktoken & Hugging Face) (#5276)

* #4983 implemented split by token for tiktoken tokenizer

* #4983 added unit test for tiktoken splitting

* #4983 implemented and added a test for splitting documents with HuggingFace tokenizer

* #4983 added support for passing HF model names (instead of objects) and added an example to the HF token splitting test

* mocked HTTP model loading in unit tests, fixed pylint error

* fix lossy tokenizers splitting, use LazyImport, ignore UnicodeEncodeError for tiktoken

* reno

* rename reno file

---------

Co-authored-by: Stefano Fiorucci <44616784+anakin87@users.noreply.github.com>
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>

Run Details

10655 of 26529 relevant lines covered (40.16%)

2.14 hits per line

Jobs

ID	Job ID	Ran	Coverage
1	utils - 6969071273.1	23 Nov 2023 11:31AM UTC	18.48	GitHub Action Run
2	modeling - 6969071273.2	23 Nov 2023 11:31AM UTC	18.35	GitHub Action Run
3	pipelines - 6969071273.3	23 Nov 2023 11:31AM UTC	21.75	GitHub Action Run
4	agents - 6969071273.4	23 Nov 2023 11:30AM UTC	20.46	GitHub Action Run
5	cli - 6969071273.5	23 Nov 2023 11:30AM UTC	18.05	GitHub Action Run
6	document_stores - 6969071273.6	23 Nov 2023 11:31AM UTC	23.1	GitHub Action Run
7	others - 6969071273.7	23 Nov 2023 11:31AM UTC	20.23	GitHub Action Run
8	prompt - 6969071273.8	23 Nov 2023 11:31AM UTC	22.81	GitHub Action Run
9	nodes - 6969071273.9	23 Nov 2023 11:32AM UTC	28.45	GitHub Action Run

Source Files on build 6969071273

Detailed source file information is not available for this build.

deepset-ai / haystack / 6969071273
92%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 6969071273

deepset-ai / haystack / 6969071273 92%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 6969071273

deepset-ai / haystack / 6969071273
92%

README BADGES
x