release-engineering / exodus-rsync / 15169849513

100%

main: 100%

Pull #545

github

Batch jobs assigned to gw client's UploadWorkers [RHELDST-30656]

Previously, jobs was a channel that workers pulled from to upload
to AWS S3, and items were passed into jobs as they were filtered.
Since multiple workers were all attempting to upload from the same jobs
queue, it was possible for duplicate uploads to occur.

This commit changes the gw client's EnsureUploaded method so that jobs is
an array to which processed items are appended. When workers are created,
they are each assigned a batch of those jobs, preventing any two workers
from handling the same job(s).

Where previously the procssing of items was done in parallel to worker
uploading, it now happens sequentially, before workers begin uploading.
So, this change may slow the client down in cases where there are a
large amount of items/files to upload. However, reducing duplicate
uploads may speed the upload process up and reduce AWS costs
associated with requests made against S3.

15 of 15 new or added lines in 1 file covered. (100.0%)

1958 of 1958 relevant lines covered (100.0%)

1.14 hits per line