6010471655
91%

Ran 29 Aug 2023 09:33AM UTC

Jobs 4

Files 15

Run time 2s

Badge

Embed ▾

Committed 29 Aug 2023 09:30AM UTC coverage: 92.869%. Remained the same

Build # 6010471655

Build Type

push

github

Committed by

web-flow

Commit Message

Add commoncrawl pipeline that starts from warc paths (#392)

This PR adds a commoncrawl pipeline which contains 2 components:
- A component that downloads `warc.paths.gz` files and outputs all the
paths as separate partitions
- A component that downloads a single warc file and extracts the images
and its licenses

---------

Co-authored-by: Sharon Grundmann <sharon.grundmann@ml6.eu>

Coverage Stats

1120 of 1206 relevant lines covered (92.87%)

3.71 hits per line

Jobs

ID	Job ID	Ran	Coverage
1	test-3.10 - 6010471655.1	29 Aug 2023 09:35AM UTC	92.7
2	test-3.9 - 6010471655.2	29 Aug 2023 09:35AM UTC	92.7
3	test-3.8 - 6010471655.3	29 Aug 2023 09:35AM UTC	92.86
4	test-3.11 - 6010471655.4	29 Aug 2023 09:35AM UTC	92.7

Source Files on build 6010471655

Detailed source file information is not available for this build.

ml6team / fondant / 6010471655
91%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 6010471655

ml6team / fondant / 6010471655 91%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 6010471655

ml6team / fondant / 6010471655
91%

README BADGES
x