MITLibraries / transmogrifier-ab-diff

99%

main: 86%

LAST BUILD ON BRANCH TIMX-341-run-ab-transforms

branch: TIMX-341-run-ab-transforms

CHANGE BRANCH

x

Reset

Sync Branches

• TIMX-341-run-ab-transforms
• IN-1240-pip-audit
• TIMX-338-init-job
• TIMX-339-scaffold-project
• TIMX-340-build-ab-images
• TIMX-344-update-job-and-run-structures
• TIMX-345-cli-init-job
• TIMX-348-flask-app-scaffold
• TIMX-349-collate-ab-transforms
• TIMX-350-351-358-346-combined
• TIMX-350-calc-ab-diffs
• TIMX-353-download-extract-files
• TIMX-354-no-job-name-docker-image
• TIMX-356-limit-parallel-containers
• TIMX-367-json-viewer-renderings
• TIMX-368-improve-log-display
• TIMX-369-core-status-checks
• TIMX-371-dedupe-records
• TIMX-372-zero-diff-metrics-bug
• TIMX-373-improve-diff-calculating
• TIMX-379-input-files-options-and-helpers
• TIMX-380-365-add-util-parser-and-use-source-slug
• TIMX-382-write-container-logs-as-run-completes
• TIMX-383-pipeline-tweaks-large-runs
• TIMX-385-records-preview
• TIMX-387-tmog-output-filename-bug
• TIMX-388-TIMX-389-missing-records-and-final-records
• TIMX-401-expect-v2-datasets
• WIP-TIMX-383
• dependabot/pip/boto3-1.35.54
• dependabot/pip/boto3-1.35.55
• dependabot/pip/boto3-1.35.56
• dependabot/pip/boto3-1.35.57
• dependabot/pip/boto3-1.35.58
• dependabot/pip/boto3-1.35.59
• dependabot/pip/boto3-1.35.60
• dependabot/pip/boto3-1.35.61
• dependabot/pip/boto3-1.35.63
• dependabot/pip/boto3-1.35.65
• dependabot/pip/boto3-1.35.66
• dependabot/pip/boto3-1.35.68
• dependabot/pip/boto3-stubs-1.35.56
• dependabot/pip/boto3-stubs-1.35.57
• dependabot/pip/boto3-stubs-1.35.58
• dependabot/pip/boto3-stubs-1.35.59
• dependabot/pip/boto3-stubs-1.35.60
• dependabot/pip/boto3-stubs-1.35.61
• dependabot/pip/boto3-stubs-1.35.63
• dependabot/pip/boto3-stubs-1.35.65
• dependabot/pip/boto3-stubs-1.35.66
• dependabot/pip/boto3-stubs-1.35.68
• dependabot/pip/duckdb-1.1.3
• dependabot/pip/ipython-8.27.0
• dependabot/pip/ipython-8.28.0
• dependabot/pip/ipython-8.29.0
• dependabot/pip/mypy-1.13.0
• dependabot/pip/pre-commit-4.0.0
• dependabot/pip/pre-commit-4.0.1
• dependabot/pip/pyarrow-stubs-17.10
• dependabot/pip/pyarrow-stubs-17.11
• dependabot/pip/pyarrow-stubs-17.8
• dependabot/pip/pytest-8.3.3
• dependabot/pip/ruff-0.6.3
• dependabot/pip/ruff-0.6.4
• dependabot/pip/ruff-0.6.5
• dependabot/pip/ruff-0.6.6
• dependabot/pip/ruff-0.6.7
• dependabot/pip/ruff-0.6.8
• dependabot/pip/ruff-0.6.9
• dependabot/pip/ruff-0.7.1
• dependabot/pip/ruff-0.7.2
• dependabot/pip/ruff-0.7.3
• dependabot/pip/ruff-0.7.4
• dependabot/pip/ruff-0.8.0
• dependabot/pip/setuptools-74.1.0
• dependabot/pip/setuptools-74.1.1
• dependabot/pip/setuptools-74.1.2
• dependabot/pip/setuptools-75.1.0
• dependabot/pip/setuptools-75.2.0
• dependabot/pip/setuptools-75.4.0
• dependabot/pip/setuptools-75.5.0
• dependabot/pip/setuptools-75.6.0
• dependabot/pip/werkzeug-3.0.6
• main
• v1.0

Build # 10927264232

Build Type

Pull #6

github

Committed by jonavellecuerdo

Commit Message

Add run_ab_transforms command

Why these changes are being introduced:
* The A/B diff workflow requires a function
that can run Docker containers from  A/B images of
transmogrifier, transform a given set of input files,
and write transformed output files to the appropriate
location within a 'run' directory.

How this addresses that need:
* Create function that runs Docker containers
* Create function that waits for Docker containers with timeout
* Create function that aggregates logs from containers
* Create function for retreiving list of filepaths to transformed files
* Create function that parses transform details from extract filename
* Create util function that creates directories
* Add exception raised for Docker container runs that exceed timeout

Side effects of this change:
* None

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-341

Pull Request Pull Request #6: Timx 341 run ab transforms

Run Details

99 of 101 new or added lines in 5 files covered. (98.02%)

266 of 270 relevant lines covered (98.52%)

0.99 hits per line