MITLibraries / timdex-dataset-api
97%
main: 93%

LAST BUILD BRANCH: v3.4
DEFAULT BRANCH: main
Repo Added
Token 6Ra2O2Hw9sRKiMVfLUH0SUOYrZqZct8CJ regen
Build 304 Last
Files 7
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH TIMX-465-run-record-offset-column
branch: TIMX-465-run-record-offset-column
CHANGE BRANCH
x
Sync Branches

coverage: 97.423% (+0.03%) from 97.396%
12794648113

push

github

ghukill 
Add run_record_offset column to dataset

Why these changes are being introduced:

Bulk reading and writing from the TIMDEX dataset is a primary responsibility,
but occassional random access (e.g. locating a single record row) will be
helpful (e.g. looking at the original source record for a problematic record).

Each TIMDEX JSON record in Opensearch will contain a "provenance" object that will
include things like run_date, run_id, and now run_record_offset.  This offset
allows for quicker (time) and more efficient (data read) retrieval of a single
record given information in the TIMDEX provenance object.

How this addresses that need:

Parquet files have metadata embedded that describe what values can be found
in subsets of the file, but this is only helpful when the min/max values
in that metadata can inform query engines if a desired record may be
present.  Unfortunately, the timdex_record_id is a) not lexicographically
sortable (at least not easily), and b) are not ordered during write.

By adding this offset, effectively an incrementing counter as records are
yielded for writing, we have a value that is pre-sorted and provides nice
ranges in the parquet file metadata.  Query engines can utilize this to
dramatically improve random access reads.  By including this offset integer
in the TIMDEX record "provenance" section we close the loop and provide
enough information in the Opensearch record to efficiently retrieve it
from the parquet dataset.

Side effects of this change:
* Dataset will now include a new column 'run_record_offset'

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-465

5 of 5 new or added lines in 3 files covered. (100.0%)

189 of 194 relevant lines covered (97.42%)

0.97 hits per line

Relevant lines Covered
194 RELEVANT LINES 189 COVERED LINES
0.97 HITS PER LINE
Source Files on TIMX-465-run-record-offset-column

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
12794648113 TIMX-465-run-record-offset-column Add run_record_offset column to dataset Why these changes are being introduced: Bulk reading and writing from the TIMDEX dataset is a primary responsibility, but occassional random access (e.g. locating a single record row) will be helpful (e.g.... push ghukill github
97.42
See All Builds (304)

Badge your Repo: timdex-dataset-api

We detected this repo isn’t badged! Grab the embed code to the right, add it to your repo to show off your code coverage, and when the badge is live hit the refresh button to remove this message.
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

Refresh