MITLibraries / timdex-dataset-api

97%

main: 93%

LAST BUILD ON BRANCH TIMX-512-row-group-sizes

branch: TIMX-512-row-group-sizes

CHANGE BRANCH

x

Reset

Sync Branches

• TIMX-512-row-group-sizes
• TIMX-414-scaffold-library-project
• TIMX-415-load-dataset
• TIMX-415-write-to-dataset
• TIMX-417-read-from-dataset
• TIMX-424-reorder-partition-columns
• TIMX-425-update-load-dataset-and-apply-filtering
• TIMX-427-improve-logging
• TIMX-432-rework-dataset-partitions
• TIMX-453-read-transformed-records-from-dataset
• TIMX-456-bump-version-number
• TIMX-456-filter-with-or-conditions
• TIMX-465-run-record-offset-column
• TIMX-468-read-configs
• TIMX-494-new-timdexsource-class
• TIMX-494-pip-audit-and-logging-updates
• TIMX-494-run-metadata
• TIMX-494-source-current-runs-and-records
• TIMX-496-add-same-day-run-timestamp
• TIMX-496-establish-migrations-and-backfill-migration
• TIMX-497-filtering-current-records
• TIMX-504-dataset-fragments-vs-batches
• TIMX-506-dataset-metadata-class-client
• TIMX-507-current-records-utilize-metadata-layer
• TIMX-508-run-timestamp-data-migration
• TIMX-509-explicit-run-timestamp
• TIMX-515-hotfix-install-duckdb-httpfs-extension
• TIMX-515-static-duckdb-file-prep
• TIMX-526-projected-views
• TIMX-527-write-append-deltas
• TIMX-528-merge-append-deltas
• TIMX-529-sql-based-read-methods
• TIMX-530-create-static-metadata-db-file
• TIMX-530-prep-work-and-s3-client
• TIMX-533-rework-dataset-load
• TIMX-537-bump-to-major-version-3
• TIMX-540-ecs-duckdb-s3-connection
• TIMX-541-extension-installation-lambda-context
• TIMX-543-cr-optimize-v2
• TIMX-543-keyset-pagination-for-reading
• bump-version-0-6-0
• dependabot/pip/boto3-1.35.72
• dependabot/pip/boto3-1.35.74
• dependabot/pip/boto3-1.35.76
• dependabot/pip/boto3-1.35.77
• dependabot/pip/boto3-1.35.78
• dependabot/pip/boto3-1.35.79
• dependabot/pip/boto3-1.35.80
• dependabot/pip/boto3-1.35.81
• dependabot/pip/boto3-1.35.82
• dependabot/pip/boto3-1.35.83
• dependabot/pip/boto3-1.35.84
• dependabot/pip/boto3-1.35.85
• dependabot/pip/boto3-1.35.86
• dependabot/pip/boto3-1.35.87
• dependabot/pip/boto3-1.35.88
• dependabot/pip/boto3-1.35.90
• dependabot/pip/boto3-1.35.91
• dependabot/pip/boto3-1.35.92
• dependabot/pip/boto3-1.35.93
• dependabot/pip/boto3-1.35.94
• dependabot/pip/boto3-1.35.96
• dependabot/pip/boto3-1.35.97
• dependabot/pip/boto3-1.35.98
• dependabot/pip/boto3-1.35.99
• dependabot/pip/boto3-1.36.0
• dependabot/pip/boto3-1.36.1
• dependabot/pip/boto3-1.36.10
• dependabot/pip/boto3-1.36.11
• dependabot/pip/boto3-1.36.12
• dependabot/pip/boto3-1.36.13
• dependabot/pip/boto3-1.36.14
• dependabot/pip/boto3-1.36.15
• dependabot/pip/boto3-1.36.16
• dependabot/pip/boto3-1.36.17
• dependabot/pip/boto3-1.36.18
• dependabot/pip/boto3-1.36.19
• dependabot/pip/boto3-1.36.2
• dependabot/pip/boto3-1.36.20
• dependabot/pip/boto3-1.36.21
• dependabot/pip/boto3-1.36.22
• dependabot/pip/boto3-1.36.24
• dependabot/pip/boto3-1.36.25
• dependabot/pip/boto3-1.36.26
• dependabot/pip/boto3-1.36.4
• dependabot/pip/boto3-1.36.5
• dependabot/pip/boto3-1.36.6
• dependabot/pip/boto3-1.36.7
• dependabot/pip/boto3-1.36.8
• dependabot/pip/boto3-1.36.9
• dependabot/pip/boto3-1.37.0
• dependabot/pip/boto3-1.37.1
• dependabot/pip/boto3-1.37.2
• dependabot/pip/boto3-1.37.3
• dependabot/pip/boto3-stubs-1.35.76
• dependabot/pip/boto3-stubs-1.35.77
• dependabot/pip/boto3-stubs-1.35.78
• dependabot/pip/boto3-stubs-1.35.79
• dependabot/pip/boto3-stubs-1.35.80
• dependabot/pip/boto3-stubs-1.35.81
• dependabot/pip/boto3-stubs-1.35.82
• dependabot/pip/boto3-stubs-1.35.83
• dependabot/pip/boto3-stubs-1.35.84
• dependabot/pip/boto3-stubs-1.35.85
• dependabot/pip/boto3-stubs-1.35.86
• dependabot/pip/boto3-stubs-1.35.87
• dependabot/pip/boto3-stubs-1.35.88
• dependabot/pip/boto3-stubs-1.35.90
• dependabot/pip/boto3-stubs-1.35.91
• dependabot/pip/boto3-stubs-1.35.92
• dependabot/pip/boto3-stubs-1.35.93
• dependabot/pip/boto3-stubs-1.35.94
• dependabot/pip/boto3-stubs-1.35.96
• dependabot/pip/boto3-stubs-1.35.97
• dependabot/pip/boto3-stubs-1.35.98
• dependabot/pip/boto3-stubs-1.35.99
• dependabot/pip/boto3-stubs-1.36.0
• dependabot/pip/boto3-stubs-1.36.1
• dependabot/pip/boto3-stubs-1.36.10
• dependabot/pip/boto3-stubs-1.36.11
• dependabot/pip/boto3-stubs-1.36.12
• dependabot/pip/boto3-stubs-1.36.13
• dependabot/pip/boto3-stubs-1.36.14
• dependabot/pip/boto3-stubs-1.36.15
• dependabot/pip/boto3-stubs-1.36.16
• dependabot/pip/boto3-stubs-1.36.17
• dependabot/pip/boto3-stubs-1.36.18
• dependabot/pip/boto3-stubs-1.36.19
• dependabot/pip/boto3-stubs-1.36.2
• dependabot/pip/boto3-stubs-1.36.21
• dependabot/pip/boto3-stubs-1.36.22
• dependabot/pip/boto3-stubs-1.36.24
• dependabot/pip/boto3-stubs-1.36.25
• dependabot/pip/boto3-stubs-1.36.26
• dependabot/pip/boto3-stubs-1.36.4
• dependabot/pip/boto3-stubs-1.36.5
• dependabot/pip/boto3-stubs-1.36.6
• dependabot/pip/boto3-stubs-1.36.7
• dependabot/pip/boto3-stubs-1.36.8
• dependabot/pip/boto3-stubs-1.36.9
• dependabot/pip/boto3-stubs-1.37.0
• dependabot/pip/boto3-stubs-1.37.1
• dependabot/pip/boto3-stubs-1.37.2
• dependabot/pip/boto3-stubs-1.37.3
• dependabot/pip/cryptography-44.0.1
• dependabot/pip/ipython-8.30.0
• dependabot/pip/ipython-8.31.0
• dependabot/pip/ipython-8.32.0
• dependabot/pip/moto-5.0.23
• dependabot/pip/moto-5.0.24
• dependabot/pip/moto-5.0.25
• dependabot/pip/moto-5.0.26
• dependabot/pip/moto-5.0.27
• dependabot/pip/moto-5.0.28
• dependabot/pip/moto-5.1.0
• dependabot/pip/mypy-1.14.0
• dependabot/pip/mypy-1.14.1
• dependabot/pip/mypy-1.15.0
• dependabot/pip/pandas-stubs-2.2.3.241126
• dependabot/pip/pyarrow-stubs-17.13
• dependabot/pip/pyarrow-stubs-17.14
• dependabot/pip/pytest-8.3.4
• dependabot/pip/ruff-0.8.1
• dependabot/pip/ruff-0.8.2
• dependabot/pip/ruff-0.8.3
• dependabot/pip/ruff-0.8.4
• epic-TIMX-515
• epic-timx-515
• hotfix-pin-duckdb-v1-3-2
• main
• temp-IN-1438-marimo-ecs-permissions
• v0.2.0
• v0.3.0
• v0.6.0
• v1-release
• v1.0
• v1.1
• v2.0
• v2.1
• v2.2
• v2.3
• v3.0
• v3.2
• v3.3
• v3.4

Build # 16059426820

Build Type

push

github

Committed by ghukill

Commit Message

Add 003 row group size data migration

Why these changes are being introduced:

It was discovered that some parquet files, mostly those before
2025-06-27, have potentially a single, very large row group instead
of maximum rows per group of 1k.

This has been addressed in recent work, resulting in no files after
2025-06-28 having a single row group if total rows exceed 1k.

How this addresses that need:

Migration script that re-packs row groups (rewrites) when a single
row group is found for a number of rows that should be split up.

Side effects of this change:
* None

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-512

Run Details

312 of 323 relevant lines covered (96.59%)

0.97 hits per line