5875603780
91%

Ran 16 Aug 2023 06:53AM UTC

Jobs 4

Files 15

Run time 2s

Badge

Embed ▾

pending completion

Build # 5875603780

Build Type

push

github

Committed by

web-flow

Commit Message

[LLM pipeline] Update text normalization component (#335)

Add minor improvements to the text normalization component. Mainly based
on the work of [Penedo et al ](https://arxiv.org/pdf/2306.01116.pdf)

Quality can be improved by removing specific patterns in single lines:
> We analyse documents line-by-line, and
discard or edit the lines based on the following rules:
• If it is mainly composed of uppercase characters (discard);
• If it is only composed of numerical characters (discard);
• If it is a counter (e.g. 3 likes) (discard);
• If it only contains one word (discard);

Coverage Stats

1003 of 1068 relevant lines covered (93.91%)

3.75 hits per line

Jobs

ID	Job ID	Ran	Coverage
1	test-3.11 - 5875603780.1	16 Aug 2023 06:54AM UTC	93.73
2	test-3.9 - 5875603780.2	16 Aug 2023 06:54AM UTC	93.72
3	test-3.8 - 5875603780.3	16 Aug 2023 06:54AM UTC	93.91
4	test-3.10 - 5875603780.4	16 Aug 2023 06:54AM UTC	93.73

Source Files on build 5875603780

Detailed source file information is not available for this build.

ml6team / fondant / 5875603780
91%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 5875603780

ml6team / fondant / 5875603780 91%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 5875603780

ml6team / fondant / 5875603780
91%

README BADGES
x