MITLibraries / timdex-dataset-api
94%
main: 93%

LAST BUILD BRANCH: v3.4
DEFAULT BRANCH: main
Repo Added
Token 6Ra2O2Hw9sRKiMVfLUH0SUOYrZqZct8CJ regen
Build 304 Last
Files 7
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH TIMX-494-new-timdexsource-class
branch: TIMX-494-new-timdexsource-class
CHANGE BRANCH
x
Sync Branches

coverage: 94.136% (-3.4%) from 97.5%
15121591888

Pull #141

github

ghukill 
Add TIMDEXRunManager and TIMDEXSource classes

Why these changes are being introduced:

Until now, this library aimed to support the known use cases
of efficiently writing the results of an ETL run to the dataset
and efficiently reading all records from an ETL run for indexing
into Opensearch.  However, it was known that eventually this library
should support more nuanced access of the dataset, including a
representation of the current state of a source in TIMDEX.

How this addresses that need:

This commit introduces two new classes:
1. TIDMEXRunManager
2. TIMDEXSource

TIMDEXRunManager is a utility class that efficiently retrieves
metadata about all ETL runs in the dataset. Equally important, it is
capable of then returning specific parquet files from the dataset
that are associated with current records in TIMDEX for the source.

Building on this, TIMDEXSource aims to provide the more nuanced behavior
of yielding only the current versions of records for a source.  This class
extends TIMDEXDataset, providing all the base class functionality,
with the layered opinionation of only yielding "current" versions of
a record.

Together, these classes provide efficient and normalized patterns
for yielding all current records from the dataset for a source,
pulling them from the timeseries nature of the dataset.

Side effects of this change:
* None at this time

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-494
Pull Request #141: TIMX 494 - add TIMDEXRunManager and TIMDEXSource classes

116 of 130 new or added lines in 5 files covered. (89.23%)

305 of 324 relevant lines covered (94.14%)

0.94 hits per line

Relevant lines Covered
324 RELEVANT LINES 305 COVERED LINES
0.94 HITS PER LINE
Source Files on TIMX-494-new-timdexsource-class

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
15121591888 TIMX-494-new-timdexsource-class Add TIMDEXRunManager and TIMDEXSource classes Why these changes are being introduced: Until now, this library aimed to support the known use cases of efficiently writing the results of an ETL run to the dataset and efficiently reading all record... Pull #141 ghukill github
94.14
15121174566 TIMX-494-new-timdexsource-class Add TIMDEXRunManager and TIMDEXSource classes Why these changes are being introduced: Until now, this library aimed to support the known use cases of efficiently writing the results of an ETL run to the dataset and efficiently reading all record... push ghukill github
94.14
See All Builds (304)

Badge your Repo: timdex-dataset-api

We detected this repo isn’t badged! Grab the embed code to the right, add it to your repo to show off your code coverage, and when the badge is live hit the refresh button to remove this message.
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

Refresh