MITLibraries / timdex-dataset-api
93%
main: 93%

LAST BUILD BRANCH: v3.4
DEFAULT BRANCH: main
Repo Added
Token 6Ra2O2Hw9sRKiMVfLUH0SUOYrZqZct8CJ regen
Build 304 Last
Files 7
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH TIMX-543-keyset-pagination-for-reading
branch: TIMX-543-keyset-pagination-for-reading
CHANGE BRANCH
x
Sync Branches

coverage: 93.011% (+0.02%) from 92.989%
17333521896

Pull #169

github

ghukill 
Use keyset pagination for meta and data read methods

Why these changes are being introduced:

For all read methods, the former approach was to perform a metadata query
and store the entire results in memory, then loop through chunks of that
metadata and build SQL queries to perform data retrieval.  Even for
metadata queries that may bring back 3-4 million results, this worked,
but there is an upper limit.

Ideally, we would perform all of our queries -- metadata and data -- in
chunks to ease memory pressure.  And in some cases, this can increase
performance.

How this addresses that need:

This reworks the base read_batches_iter() method to perform smaller,
chunked metadata queries.  To paginate the results, instead of using
the slow LIMIT / OFFSET approach, we use keyset pagination, which means
we can look for values greater than a tuple of values that are ordered.
This is often the preferred way to perform paginated querying when you
have nicely ordered columns.

In support of this, we also begin hashing the filename and run_id
columns for ordering, providing almost an order magnitude speedup.
The performance penalty for creating the hash is offset by the speedup
of ordering integers versus very long strings.

The net effect is no changes to the input/ouput signatures of the read
methods, but improved memory usage and performance.

Side effects of this change:
* None

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-543
Pull Request #169: TIMX 543 - keyset pagination for read methods

51 of 55 new or added lines in 3 files covered. (92.73%)

519 of 558 relevant lines covered (93.01%)

0.93 hits per line

Relevant lines Covered
558 RELEVANT LINES 519 COVERED LINES
0.93 HITS PER LINE
Source Files on TIMX-543-keyset-pagination-for-reading

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
17333521896 TIMX-543-keyset-pagination-for-reading Use keyset pagination for meta and data read methods Why these changes are being introduced: For all read methods, the former approach was to perform a metadata query and store the entire results in memory, then loop through chunks of that metad... Pull #169 ghukill github
93.01
17324732994 TIMX-543-keyset-pagination-for-reading Use keyset pagination for meta and data read methods Why these changes are being introduced: For all read methods, the former approach was to perform a metadata query and store the entire results in memory, then loop through chunks of that metad... push ghukill github
93.25
See All Builds (304)

Badge your Repo: timdex-dataset-api

We detected this repo isn’t badged! Grab the embed code to the right, add it to your repo to show off your code coverage, and when the badge is live hit the refresh button to remove this message.
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

Refresh