15350236835
96%
main: 94%

Ran 30 May 2025 03:34PM UTC

Jobs 1

Files 6

Run time 1min

Badge

Embed ▾

Committed 30 May 2025 03:27PM UTC coverage: 95.541%. Remained the same

Build # 15350236835

Build Type

push

github

Committed by

ghukill

Commit Message

Add migrations folder and run_timestamp migration

Why these changes are being introduced:

We have a need to perform a onetime bulk edit of the parquet dataset,
which is very similar to a SQL database migration: addition of a column
and backfilling of data.

While we may not require the strictness of SQL schema migrations, which
often can be "replayed" in order to upgrade or reverse order to downgrade,
we nonetheless would benefit from some structure and code-as-documentation
for bulk operations we perform on the actual parquet dataset.

How this addresses that need:
* Creates a new root level migrations/ folder
* Includes a README with a proposed simple structure for migrations (mostly
naming conventions)
* Includes our first migration to backfill all pre-existing parquet files
with a new 'run_timestamp' column
  * The run_timestamp will be pulled from the creation date of the parquet
    file in S3, which is close enough for our purposes

Side effects of this change:
* TDA becomes the location for storing code related to "migrations" for
the parquet dataset
* All pre-existing parquet files will now have a run_timestamp column

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-496

Run Details

300 of 314 relevant lines covered (95.54%)

0.96 hits per line

Jobs

ID	Job ID	Ran	Files	Coverage
1	15350236835.1	30 May 2025 03:34PM UTC	6	95.54	GitHub Action Run

MITLibraries / timdex-dataset-api / 15350236835
96%
main: 94%

README BADGES
x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 15350236835

MITLibraries / timdex-dataset-api / 15350236835 96% main: 94%

README BADGES x

Markdown

Textile

RDoc

HTML

Rst

Jobs

Source Files on build 15350236835

MITLibraries / timdex-dataset-api / 15350236835
96%
main: 94%

README BADGES
x