MITLibraries / s3-bagit-validator / 14198657654

94%

main: 90%

Build Type

Pull #22

github

Committed by ghukill

Commit Message

Improve Athena query for AIP inventory data

Why these changes are being introduced:

The updated AIP inventory Athena query has a potentially
non-deterministic quality where S3 inventory rows may not
have originated from the most recent S3 inventory pull
(which is partitioned by date).  Though the data may
have been accurate, it felt a little sloppy.

How this addresses that need:

Instead of a window function, we focus on retrieving only
the most recent data pull via the most recent 'dt' partition.
Additionally, if we know a) it is the most recent data pull
and b) it is 'is_latest' then we don't need to sort by
last_modified_date anymore.

Confirmed that query time remains about the same, but data
scanned is reduced, suggesting that use of partitions is more
effective here.

Side effects of this change:
* None

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/IR-193

Pull Request Pull Request #22: IR 193 - parallelize checksum generation

Run Details

38 of 42 new or added lines in 4 files covered. (90.48%)

4 existing lines in 1 file now uncovered.

318 of 338 relevant lines covered (94.08%)

0.94 hits per line