MITLibraries / s3-bagit-validator / 14172089121

94%

main: 90%

push

github

Parallelize checksum generation

Why these changes are being introduced:

An AIP was encountered that had 8k+ files in the data folder.
This required a VERY long time to generate checksums sequentially
eventually timing out the lambda.

Originally this work was performed in parallel, but it was
simplified during the initial first pass.

How this addresses that need:
* Massively parallelize AWS operations to retrieve and/or generate
a checksum for a file.  S3 is designed for parallelization and the
local resources requirement is small for all the parallel threads.

Side effects of this change:
* Increased system resources: when using 256 threads looking at
about 500mb memory usage.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/IR-193

38 of 42 new or added lines in 4 files covered. (90.48%)

318 of 338 relevant lines covered (94.08%)

0.94 hits per line