OpenRefine / OpenRefine / 10200488541

50%

push

github

Upload files in chunks (#6510) (#6531)

* Upload files in chunks (#6510)

Closes #4303

Adds function for uploading a file in chunks as described on
https://www.mediawiki.org/wiki/API:Upload#Example_3:_Upload_file_in_chunks.
The chunks are created as temporary files and are removed once uploaded.

* Use chunked uploading for files larger than a certain size (#6530)

The threshold is 100,000,000 bytes for now to be below the upload limit for
Wikimedia Commons, see
https://commons.wikimedia.org/wiki/Commons:Maximum_file_size. This makes
sure that files are chunked if they can't otherwise be uploaded. The value is
hard coded for now, but will be read from the manifest in a later patch.

* Make sure that the file page exists before uploading SDC (#6724)

Checks if the file page has been created after a file has been
uploaded. If not it will wait for a bit and retry. This will
continue until the page exists or the user interrupts, with the
delay between checks increasing up to a maximum. At the moment the
wait time starts at one second and increases by one second per wait.
The max wait time is one minute.

Adds logging for:
* When a file will be uploaded in chunks
* When a chunk is uploaded
* When all chunks have been uploaded
* When the commit request has been made
* Waiting for file page to exist

Co-authored-by: Antonin Delpeuch <antonin@delpeuch.eu>
Co-authored-by: Alicia Fagerving <alicia@fagerving.se>

6765 of 12289 branches covered (55.05%)

Branch coverage included in aggregate %.

17197 of 25906 relevant lines covered (66.38%)

2.99 hits per line