MITLibraries / transmogrifier-ab-diff

88%

main: 86%

LAST BUILD ON BRANCH WIP-TIMX-383

branch: WIP-TIMX-383

CHANGE BRANCH

x

Reset

Sync Branches

• WIP-TIMX-383
• IN-1240-pip-audit
• TIMX-338-init-job
• TIMX-339-scaffold-project
• TIMX-340-build-ab-images
• TIMX-341-run-ab-transforms
• TIMX-344-update-job-and-run-structures
• TIMX-345-cli-init-job
• TIMX-348-flask-app-scaffold
• TIMX-349-collate-ab-transforms
• TIMX-350-351-358-346-combined
• TIMX-350-calc-ab-diffs
• TIMX-353-download-extract-files
• TIMX-354-no-job-name-docker-image
• TIMX-356-limit-parallel-containers
• TIMX-367-json-viewer-renderings
• TIMX-368-improve-log-display
• TIMX-369-core-status-checks
• TIMX-371-dedupe-records
• TIMX-372-zero-diff-metrics-bug
• TIMX-373-improve-diff-calculating
• TIMX-379-input-files-options-and-helpers
• TIMX-380-365-add-util-parser-and-use-source-slug
• TIMX-382-write-container-logs-as-run-completes
• TIMX-383-pipeline-tweaks-large-runs
• TIMX-385-records-preview
• TIMX-387-tmog-output-filename-bug
• TIMX-388-TIMX-389-missing-records-and-final-records
• TIMX-401-expect-v2-datasets
• dependabot/pip/boto3-1.35.54
• dependabot/pip/boto3-1.35.55
• dependabot/pip/boto3-1.35.56
• dependabot/pip/boto3-1.35.57
• dependabot/pip/boto3-1.35.58
• dependabot/pip/boto3-1.35.59
• dependabot/pip/boto3-1.35.60
• dependabot/pip/boto3-1.35.61
• dependabot/pip/boto3-1.35.63
• dependabot/pip/boto3-1.35.65
• dependabot/pip/boto3-1.35.66
• dependabot/pip/boto3-1.35.68
• dependabot/pip/boto3-stubs-1.35.56
• dependabot/pip/boto3-stubs-1.35.57
• dependabot/pip/boto3-stubs-1.35.58
• dependabot/pip/boto3-stubs-1.35.59
• dependabot/pip/boto3-stubs-1.35.60
• dependabot/pip/boto3-stubs-1.35.61
• dependabot/pip/boto3-stubs-1.35.63
• dependabot/pip/boto3-stubs-1.35.65
• dependabot/pip/boto3-stubs-1.35.66
• dependabot/pip/boto3-stubs-1.35.68
• dependabot/pip/duckdb-1.1.3
• dependabot/pip/ipython-8.27.0
• dependabot/pip/ipython-8.28.0
• dependabot/pip/ipython-8.29.0
• dependabot/pip/mypy-1.13.0
• dependabot/pip/pre-commit-4.0.0
• dependabot/pip/pre-commit-4.0.1
• dependabot/pip/pyarrow-stubs-17.10
• dependabot/pip/pyarrow-stubs-17.11
• dependabot/pip/pyarrow-stubs-17.8
• dependabot/pip/pytest-8.3.3
• dependabot/pip/ruff-0.6.3
• dependabot/pip/ruff-0.6.4
• dependabot/pip/ruff-0.6.5
• dependabot/pip/ruff-0.6.6
• dependabot/pip/ruff-0.6.7
• dependabot/pip/ruff-0.6.8
• dependabot/pip/ruff-0.6.9
• dependabot/pip/ruff-0.7.1
• dependabot/pip/ruff-0.7.2
• dependabot/pip/ruff-0.7.3
• dependabot/pip/ruff-0.7.4
• dependabot/pip/ruff-0.8.0
• dependabot/pip/setuptools-74.1.0
• dependabot/pip/setuptools-74.1.1
• dependabot/pip/setuptools-74.1.2
• dependabot/pip/setuptools-75.1.0
• dependabot/pip/setuptools-75.2.0
• dependabot/pip/setuptools-75.4.0
• dependabot/pip/setuptools-75.5.0
• dependabot/pip/setuptools-75.6.0
• dependabot/pip/werkzeug-3.0.6
• main
• v1.0

Build # 11672053282

Build Type

push

github

Committed by ghukill

Commit Message

Create final records dataset from diffs and metrics

Why these changes are being introduced:

With the full pipeline mostly formed it was observed that we could
simplify the final artifacts produced by a run by combining the 'diffs'
and 'metrics' datasets into a final 'records' dataset.

This dovetails with the new optional env var 'PRESERVE_ARTIFACTS' where
every other artifact except this one would be removed after successful creation.

How this addresses that need:
* Adds new core function 'create_final_records' that writes a final
dataset containing all records and diff information needed for statistical
and individual record analysis.

Side effects of this change:
* Fewer final disparate datasets for a run

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-370

Run Details

23 of 48 new or added lines in 4 files covered. (47.92%)

741 of 846 relevant lines covered (87.59%)

0.88 hits per line