MITLibraries / transmogrifier
99%
main: 99%

LAST BUILD BRANCH: v2.6
DEFAULT BRANCH: main
Repo Added
Files 19
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH TIMX-406-add-provenance-data
branch: TIMX-406-add-provenance-data
CHANGE BRANCH
x

coverage: 98.765% (+0.008%) from 98.757%
12817776883

push

github

ghukill 
Add TIMDEX provenance object to transformed records

Why these changes are being introduced:

Transitioning to a parquet dataset architecture for TIMDEX ETL provides
additional data related to each transformed record as part of that
record's row in the dataset.  But this data is only helpful if you tether
the record you encounter in Opensearch with a row in the dataset.

Certainly related, but not dependent on the parquet dataset change,
was the desire for more information about a record in TIMDEX, e.g. when
was it transformed and indexed.

We might consider this information "provenance" about the TIMDEX record
as encountered in Opensearch and/or the TIMDEX API.

How this addresses that need:

A new "timdex_provenance" field is added to the TIMDEX data model that
includes information about the origins of the TIMDEX record.  As it
pertains to the parquet dataset, this provenance data includes fields like
"run_id" and "run_record_offset" which help pinpoint the row in the
parquet dataset for this record.  With this linkage, it becomes possible to
very quickly retrieve the original source record for a transformed record.

In addition to support random access reads of the dataset, this provenance
data provides some metadata about the TIMDEX record that is immediately
informative like "run_date".

Side effects of this change:
* None, really.  TIM will need to be updated to include this new field
in the Opensearch mapping, but until then, it's just extra data in the
transformed record.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-406

12 of 12 new or added lines in 2 files covered. (100.0%)

1760 of 1782 relevant lines covered (98.77%)

0.99 hits per line

Relevant lines Covered
1782 RELEVANT LINES 1760 COVERED LINES
0.99 HITS PER LINE
Source Files on TIMX-406-add-provenance-data

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
12817776883 TIMX-406-add-provenance-data Add TIMDEX provenance object to transformed records Why these changes are being introduced: Transitioning to a parquet dataset architecture for TIMDEX ETL provides additional data related to each transformed record as part of that record's row i... push ghukill github
98.77
See All Builds (691)