MITLibraries / timdex-dataset-api

93%

main: 93%

LAST BUILD ON BRANCH USE-306-handle-missing-metadata-or-embeddings

branch: USE-306-handle-missing-metadata-or-embeddings

CHANGE BRANCH

x

Reset

Sync Branches

• USE-306-handle-missing-metadata-or-embeddings
• TIMX-414-scaffold-library-project
• TIMX-415-load-dataset
• TIMX-415-write-to-dataset
• TIMX-417-read-from-dataset
• TIMX-424-reorder-partition-columns
• TIMX-425-update-load-dataset-and-apply-filtering
• TIMX-427-improve-logging
• TIMX-432-rework-dataset-partitions
• TIMX-453-read-transformed-records-from-dataset
• TIMX-456-bump-version-number
• TIMX-456-filter-with-or-conditions
• TIMX-465-run-record-offset-column
• TIMX-468-read-configs
• TIMX-494-new-timdexsource-class
• TIMX-494-pip-audit-and-logging-updates
• TIMX-494-run-metadata
• TIMX-494-source-current-runs-and-records
• TIMX-496-add-same-day-run-timestamp
• TIMX-496-establish-migrations-and-backfill-migration
• TIMX-497-filtering-current-records
• TIMX-504-dataset-fragments-vs-batches
• TIMX-506-dataset-metadata-class-client
• TIMX-507-current-records-utilize-metadata-layer
• TIMX-508-run-timestamp-data-migration
• TIMX-509-explicit-run-timestamp
• TIMX-512-row-group-sizes
• TIMX-515-hotfix-install-duckdb-httpfs-extension
• TIMX-515-static-duckdb-file-prep
• TIMX-526-projected-views
• TIMX-527-write-append-deltas
• TIMX-528-merge-append-deltas
• TIMX-529-sql-based-read-methods
• TIMX-530-create-static-metadata-db-file
• TIMX-530-prep-work-and-s3-client
• TIMX-533-rework-dataset-load
• TIMX-537-bump-to-major-version-3
• TIMX-540-ecs-duckdb-s3-connection
• TIMX-541-extension-installation-lambda-context
• TIMX-543-cr-optimize-v2
• TIMX-543-keyset-pagination-for-reading
• TIMX-559-update-duckdb-dependency
• USE-142-dataset-embedding-imports
• USE-142-dataset-embedding-types
• USE-142-embeddings-source-and-write
• USE-143-always-metadata-join
• USE-143-embeddings-read
• USE-143-record-metadata-join
• USE-306-refactor-class-relationships
• USE-58-lazy-load-current-records
• bump-version-0-6-0
• dependabot/pip/boto3-1.35.72
• dependabot/pip/boto3-1.35.74
• dependabot/pip/boto3-1.35.76
• dependabot/pip/boto3-1.35.77
• dependabot/pip/boto3-1.35.78
• dependabot/pip/boto3-1.35.79
• dependabot/pip/boto3-1.35.80
• dependabot/pip/boto3-1.35.81
• dependabot/pip/boto3-1.35.82
• dependabot/pip/boto3-1.35.83
• dependabot/pip/boto3-1.35.84
• dependabot/pip/boto3-1.35.85
• dependabot/pip/boto3-1.35.86
• dependabot/pip/boto3-1.35.87
• dependabot/pip/boto3-1.35.88
• dependabot/pip/boto3-1.35.90
• dependabot/pip/boto3-1.35.91
• dependabot/pip/boto3-1.35.92
• dependabot/pip/boto3-1.35.93
• dependabot/pip/boto3-1.35.94
• dependabot/pip/boto3-1.35.96
• dependabot/pip/boto3-1.35.97
• dependabot/pip/boto3-1.35.98
• dependabot/pip/boto3-1.35.99
• dependabot/pip/boto3-1.36.0
• dependabot/pip/boto3-1.36.1
• dependabot/pip/boto3-1.36.10
• dependabot/pip/boto3-1.36.11
• dependabot/pip/boto3-1.36.12
• dependabot/pip/boto3-1.36.13
• dependabot/pip/boto3-1.36.14
• dependabot/pip/boto3-1.36.15
• dependabot/pip/boto3-1.36.16
• dependabot/pip/boto3-1.36.17
• dependabot/pip/boto3-1.36.18
• dependabot/pip/boto3-1.36.19
• dependabot/pip/boto3-1.36.2
• dependabot/pip/boto3-1.36.20
• dependabot/pip/boto3-1.36.21
• dependabot/pip/boto3-1.36.22
• dependabot/pip/boto3-1.36.24
• dependabot/pip/boto3-1.36.25
• dependabot/pip/boto3-1.36.26
• dependabot/pip/boto3-1.36.4
• dependabot/pip/boto3-1.36.5
• dependabot/pip/boto3-1.36.6
• dependabot/pip/boto3-1.36.7
• dependabot/pip/boto3-1.36.8
• dependabot/pip/boto3-1.36.9
• dependabot/pip/boto3-1.37.0
• dependabot/pip/boto3-1.37.1
• dependabot/pip/boto3-1.37.2
• dependabot/pip/boto3-1.37.3
• dependabot/pip/boto3-stubs-1.35.76
• dependabot/pip/boto3-stubs-1.35.77
• dependabot/pip/boto3-stubs-1.35.78
• dependabot/pip/boto3-stubs-1.35.79
• dependabot/pip/boto3-stubs-1.35.80
• dependabot/pip/boto3-stubs-1.35.81
• dependabot/pip/boto3-stubs-1.35.82
• dependabot/pip/boto3-stubs-1.35.83
• dependabot/pip/boto3-stubs-1.35.84
• dependabot/pip/boto3-stubs-1.35.85
• dependabot/pip/boto3-stubs-1.35.86
• dependabot/pip/boto3-stubs-1.35.87
• dependabot/pip/boto3-stubs-1.35.88
• dependabot/pip/boto3-stubs-1.35.90
• dependabot/pip/boto3-stubs-1.35.91
• dependabot/pip/boto3-stubs-1.35.92
• dependabot/pip/boto3-stubs-1.35.93
• dependabot/pip/boto3-stubs-1.35.94
• dependabot/pip/boto3-stubs-1.35.96
• dependabot/pip/boto3-stubs-1.35.97
• dependabot/pip/boto3-stubs-1.35.98
• dependabot/pip/boto3-stubs-1.35.99
• dependabot/pip/boto3-stubs-1.36.0
• dependabot/pip/boto3-stubs-1.36.1
• dependabot/pip/boto3-stubs-1.36.10
• dependabot/pip/boto3-stubs-1.36.11
• dependabot/pip/boto3-stubs-1.36.12
• dependabot/pip/boto3-stubs-1.36.13
• dependabot/pip/boto3-stubs-1.36.14
• dependabot/pip/boto3-stubs-1.36.15
• dependabot/pip/boto3-stubs-1.36.16
• dependabot/pip/boto3-stubs-1.36.17
• dependabot/pip/boto3-stubs-1.36.18
• dependabot/pip/boto3-stubs-1.36.19
• dependabot/pip/boto3-stubs-1.36.2
• dependabot/pip/boto3-stubs-1.36.21
• dependabot/pip/boto3-stubs-1.36.22
• dependabot/pip/boto3-stubs-1.36.24
• dependabot/pip/boto3-stubs-1.36.25
• dependabot/pip/boto3-stubs-1.36.26
• dependabot/pip/boto3-stubs-1.36.4
• dependabot/pip/boto3-stubs-1.36.5
• dependabot/pip/boto3-stubs-1.36.6
• dependabot/pip/boto3-stubs-1.36.7
• dependabot/pip/boto3-stubs-1.36.8
• dependabot/pip/boto3-stubs-1.36.9
• dependabot/pip/boto3-stubs-1.37.0
• dependabot/pip/boto3-stubs-1.37.1
• dependabot/pip/boto3-stubs-1.37.2
• dependabot/pip/boto3-stubs-1.37.3
• dependabot/pip/cryptography-44.0.1
• dependabot/pip/ipython-8.30.0
• dependabot/pip/ipython-8.31.0
• dependabot/pip/ipython-8.32.0
• dependabot/pip/moto-5.0.23
• dependabot/pip/moto-5.0.24
• dependabot/pip/moto-5.0.25
• dependabot/pip/moto-5.0.26
• dependabot/pip/moto-5.0.27
• dependabot/pip/moto-5.0.28
• dependabot/pip/moto-5.1.0
• dependabot/pip/mypy-1.14.0
• dependabot/pip/mypy-1.14.1
• dependabot/pip/mypy-1.15.0
• dependabot/pip/pandas-stubs-2.2.3.241126
• dependabot/pip/pyarrow-stubs-17.13
• dependabot/pip/pyarrow-stubs-17.14
• dependabot/pip/pytest-8.3.4
• dependabot/pip/ruff-0.8.1
• dependabot/pip/ruff-0.8.2
• dependabot/pip/ruff-0.8.3
• dependabot/pip/ruff-0.8.4
• epic-TIMX-515
• epic-timx-515
• hotfix-pin-duckdb-v1-3-2
• hotfix-pin-sqlalchemy-2.0.44
• main
• temp-IN-1438-marimo-ecs-permissions
• v0.2.0
• v0.3.0
• v0.6.0
• v1-release
• v1.0
• v1.1
• v2.0
• v2.1
• v2.2
• v2.3
• v3.0
• v3.2
• v3.3
• v3.4

Build # 20435740402

Build Type

push

github

Committed by ghukill

Commit Message

Incremental table read error handling

Why these changes are being introduced:

There are a copule of primary ways in which a read method can fail based on the table requested:
1. The table name is invalid and will never work.
2. The table name is valid, but does not yet exist in the DuckDB context.

For the first, we want to raise an error immediately.  For the second, there is a bit more nuance depending
on the table requested.

How this addresses that need:

For TIMDEXDataset.read_* methods, we operate from the assumption that `records` and `current_records` should always be available and so we raise an error indicating a metadata rebuild is required.

But for TIMDEXEmbeddings, and potentially other data sources as added, we may legitimately not have data yet.
As such, we'll want to log warnings and suggest a refresh, but just return an empty set.

Side effects of this change:
* Applications like TIM, which now attempt embeddings reading for its `reindex-source` CLI command, will
no longer encounter an error if embeddings don't yet exist.
* In the rare edge cases of a brand new dataset, we have better error raising and logging.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/USE-306

Run Details

16 of 16 new or added lines in 3 files covered. (100.0%)

702 of 752 relevant lines covered (93.35%)

0.93 hits per line