MITLibraries / archival-packaging-tool / 15026957009

96%

main: 96%

push

github

Replace smart-open with direct boto3

Why these changes are being introduced:

Some testing in Dev environment revealed that downloading
and uploading of files from S3, via the utils function stream_file_transfer()
was not as performant as it could be. For very large files in a Bag, this
resulted in long Lambda runtimes that could exceed the 15 minute timeout.

How this addresses that need:

Instead of using smart-open to handle S3 or local files, the utility
function stream_file_transfer() now detects the type of transfer,
e.g. S3-to-Local, Local-to-S3, Local-to-Local, or S3-to-S3, and uses
more tailored code and libraries for each.

Read/write times are notably faster now, clocking in around
450 Mbps (megabits per second).  Any remaining bottlenecks
are likely related more to the Lambda's network throughput as
governed by its memory setting (they are correlated) or the EFS
mount's throughput.  Any improvements to those will be handled
outside of code in infrastructure most likely.

Side effects of this change:
* Removes `smart-open` as a dependency
* Runtimes are shorter due to increased S3 read/write throughput

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/IN-1294

28 of 28 new or added lines in 1 file covered. (100.0%)

268 of 280 relevant lines covered (95.71%)

0.96 hits per line