MITLibraries / timdex-dataset-api
95%
main: 94%

LAST BUILD BRANCH: USE-142-dataset-embedding-imports
DEFAULT BRANCH: main
Repo Added
Token 6Ra2O2Hw9sRKiMVfLUH0SUOYrZqZct8CJ regen
Build 318 Last
Files 8
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH TIMX-533-rework-dataset-load
branch: TIMX-533-rework-dataset-load
CHANGE BRANCH
x
Sync Branches

coverage: 94.828% (+0.3%) from 94.558%
16840527470

Pull #160

github

ghukill 
Load pyarrow dataset on TIMDEXDataset init

Why these changes are being introduced:

As the TIMDEXDatasetMetadata becomes more integrated, there is
less need to be explicit about how we load the pyarrow dataset.

Formerly, the method .load() needed to be called manually and
supported options like 'current_records' or 'include_parquet_files'.
This also reflected a time when 'TIMDEXDataset.load()' suggested that
"loading" was the pyarrow dataset only.  With the introduction of
metadata, it is also better to be specific we are loading a pyarrow
dataset which is only one of many assets associated with a
TIMDEXDataset instance.

How this addresses that need:

Renames .load() to .load_pyarrow_dataset() to be explicit about
what is happening.

We no longer store the pyarrow dataset filesystem or paths on self,
as they are only used briefly during this dataset load.  We can get
them anytime via .dataset.

Really most important, we limit the root 'location' that we init
a TIMDEXDataset instance to be a string only, the root of the dataset.
Now that we don't allow a list of strings at that level, we can trust
the nature of self.location to be a string, and the root of the TIMDEX
dataset.

Side effects of this change:
* TIMDEXDataset and TIMDEXDatasetMetadata can only be initialized
with a string, which is the root of the TIMDEX dataset.  From there,
both know where their assets can be found.
* You cannot "pre-filter" the pyarrow dataset when loading, which had
confusing overlap with the read methods; the read methods themselves
may change somewhat dramatically now that we have metadata to use.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-533
Pull Request #160: TIMX 533 - Load pyarrow dataset on TIMDEXDataset init

38 of 38 new or added lines in 1 file covered. (100.0%)

1 existing line in 1 file now uncovered.

385 of 406 relevant lines covered (94.83%)

0.95 hits per line

Relevant lines Covered
406 RELEVANT LINES 385 COVERED LINES
0.95 HITS PER LINE
Source Files on TIMX-533-rework-dataset-load

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
16840527470 TIMX-533-rework-dataset-load Load pyarrow dataset on TIMDEXDataset init Why these changes are being introduced: As the TIMDEXDatasetMetadata becomes more integrated, there is less need to be explicit about how we load the pyarrow dataset. Formerly, the method .load() neede... Pull #160 ghukill github
94.83
16815676743 TIMX-533-rework-dataset-load Load pyarrow dataset on TIMDEXDataset init Why these changes are being introduced: As the TIMDEXDatasetMetadata becomes more integrated, there is less need to be explicit about how we load the pyarrow dataset. Formerly, the method .load() neede... Pull #160 ghukill github
94.83
16815561643 TIMX-533-rework-dataset-load Load pyarrow dataset on TIMDEXDataset init Why these changes are being introduced: As the TIMDEXDatasetMetadata becomes more integrated, there is less need to be explicit about how we load the pyarrow dataset. Formerly, the method .load() neede... Pull #160 ghukill github
94.83
See All Builds (318)

Badge your Repo: timdex-dataset-api

We detected this repo isn’t badged! Grab the embed code to the right, add it to your repo to show off your code coverage, and when the badge is live hit the refresh button to remove this message.
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

Refresh