MITLibraries / dspace-submission-composer / 16680249799

push

github

Add CLI command to sync data between two locations (#188)

* Add CLI command to sync data between two locations

Why these changes are being introduced:
DSC supports submissions to both test and prod instances of MITL's
DSpace repositories. Deposits to test are performed in the Dev and
Stage environments, while deposits to prod are performed in the
Prod environment. The required data must be present in, or uploaded to,
the appropriate S3 buckets for each environment. This method
provides an easy way to sync the data between two S3 buckets
using the AWS CLI 'aws s3 sync' command.

How this addresses that need:
* Add 'sync' CLI command
* Add optional env var 'S3_BUCKET_SYNC_SOURCE'
* Update Dockerfile to include install of AWS CLI (and its dependencies)
* Add 'sync' unit test with moto_server
* Raise click.Usage error when required configs aren't set for sync

Side effects of this change:
* In the case of DSC workflows, the command will be used to sync data
from an S3 bucket to (an S3 bucket) in another AWS account. While the
implemented 'sync' DSC CLI command is quite simple, this relies on
configuring bucket policies and IAM Roles and policies to
enable cross-account data transfer.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/IN-1352

34 of 39 new or added lines in 2 files covered. (87.18%)

960 of 991 relevant lines covered (96.87%)

0.97 hits per line