1
94%
main: 90%

Ran 01 Apr 2025 02:43PM UTC

Files 7

Run time 0s

Badge

Embed ▾

Committed 01 Apr 2025 02:39PM UTC coverage: 94.083% (-0.4%) from 94.498%

Job # 14198657654.1

Build Type

Pull #22

github

Committed by

ghukill

Commit Message

Improve Athena query for AIP inventory data

Why these changes are being introduced:

The updated AIP inventory Athena query has a potentially
non-deterministic quality where S3 inventory rows may not
have originated from the most recent S3 inventory pull
(which is partitioned by date).  Though the data may
have been accurate, it felt a little sloppy.

How this addresses that need:

Instead of a window function, we focus on retrieving only
the most recent data pull via the most recent 'dt' partition.
Additionally, if we know a) it is the most recent data pull
and b) it is 'is_latest' then we don't need to sort by
last_modified_date anymore.

Confirmed that query time remains about the same, but data
scanned is reduced, suggesting that use of partitions is more
effective here.

Side effects of this change:
* None

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/IR-193

Pull Request Pull Request #22: IR 193 - parallelize checksum generation

Run Details

318 of 338 relevant lines covered (94.08%)

0.94 hits per line

MITLibraries / s3-bagit-validator / 14198657654 / 1
94%
main: 90%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 14198657654.1

MITLibraries / s3-bagit-validator / 14198657654 / 1 94% main: 90%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Source Files on job 14198657654.1

MITLibraries / s3-bagit-validator / 14198657654 / 1
94%
main: 90%

README BADGES
x