MITLibraries / transmogrifier-ab-diff

96%

main: 86%

LAST BUILD ON BRANCH TIMX-372-zero-diff-metrics-bug

branch: TIMX-372-zero-diff-metrics-bug

CHANGE BRANCH

x

Reset

Sync Branches

• TIMX-372-zero-diff-metrics-bug
• IN-1240-pip-audit
• TIMX-338-init-job
• TIMX-339-scaffold-project
• TIMX-340-build-ab-images
• TIMX-341-run-ab-transforms
• TIMX-344-update-job-and-run-structures
• TIMX-345-cli-init-job
• TIMX-348-flask-app-scaffold
• TIMX-349-collate-ab-transforms
• TIMX-350-351-358-346-combined
• TIMX-350-calc-ab-diffs
• TIMX-353-download-extract-files
• TIMX-354-no-job-name-docker-image
• TIMX-356-limit-parallel-containers
• TIMX-367-json-viewer-renderings
• TIMX-368-improve-log-display
• TIMX-369-core-status-checks
• TIMX-371-dedupe-records
• TIMX-373-improve-diff-calculating
• TIMX-379-input-files-options-and-helpers
• TIMX-380-365-add-util-parser-and-use-source-slug
• TIMX-382-write-container-logs-as-run-completes
• TIMX-383-pipeline-tweaks-large-runs
• TIMX-385-records-preview
• TIMX-387-tmog-output-filename-bug
• TIMX-388-TIMX-389-missing-records-and-final-records
• TIMX-401-expect-v2-datasets
• WIP-TIMX-383
• dependabot/pip/boto3-1.35.54
• dependabot/pip/boto3-1.35.55
• dependabot/pip/boto3-1.35.56
• dependabot/pip/boto3-1.35.57
• dependabot/pip/boto3-1.35.58
• dependabot/pip/boto3-1.35.59
• dependabot/pip/boto3-1.35.60
• dependabot/pip/boto3-1.35.61
• dependabot/pip/boto3-1.35.63
• dependabot/pip/boto3-1.35.65
• dependabot/pip/boto3-1.35.66
• dependabot/pip/boto3-1.35.68
• dependabot/pip/boto3-stubs-1.35.56
• dependabot/pip/boto3-stubs-1.35.57
• dependabot/pip/boto3-stubs-1.35.58
• dependabot/pip/boto3-stubs-1.35.59
• dependabot/pip/boto3-stubs-1.35.60
• dependabot/pip/boto3-stubs-1.35.61
• dependabot/pip/boto3-stubs-1.35.63
• dependabot/pip/boto3-stubs-1.35.65
• dependabot/pip/boto3-stubs-1.35.66
• dependabot/pip/boto3-stubs-1.35.68
• dependabot/pip/duckdb-1.1.3
• dependabot/pip/ipython-8.27.0
• dependabot/pip/ipython-8.28.0
• dependabot/pip/ipython-8.29.0
• dependabot/pip/mypy-1.13.0
• dependabot/pip/pre-commit-4.0.0
• dependabot/pip/pre-commit-4.0.1
• dependabot/pip/pyarrow-stubs-17.10
• dependabot/pip/pyarrow-stubs-17.11
• dependabot/pip/pyarrow-stubs-17.8
• dependabot/pip/pytest-8.3.3
• dependabot/pip/ruff-0.6.3
• dependabot/pip/ruff-0.6.4
• dependabot/pip/ruff-0.6.5
• dependabot/pip/ruff-0.6.6
• dependabot/pip/ruff-0.6.7
• dependabot/pip/ruff-0.6.8
• dependabot/pip/ruff-0.6.9
• dependabot/pip/ruff-0.7.1
• dependabot/pip/ruff-0.7.2
• dependabot/pip/ruff-0.7.3
• dependabot/pip/ruff-0.7.4
• dependabot/pip/ruff-0.8.0
• dependabot/pip/setuptools-74.1.0
• dependabot/pip/setuptools-74.1.1
• dependabot/pip/setuptools-74.1.2
• dependabot/pip/setuptools-75.1.0
• dependabot/pip/setuptools-75.2.0
• dependabot/pip/setuptools-75.4.0
• dependabot/pip/setuptools-75.5.0
• dependabot/pip/setuptools-75.6.0
• dependabot/pip/werkzeug-3.0.6
• main
• v1.0

Build # 11408587656

Build Type

push

github

Committed by ghukill

Commit Message

Parse metrics even with zero diffs

Why these changes are being introduced:

A bug surfaced where attempting to calculate metrics from
the diffs dataset would throw an error if zero records
had any diffs.  This was the result of dynamically preparing
SQL queries for DuckDB that looked for any field with a value
of 1. With no diffs, there were no fields, and the 'OR' clause
was malformed.

In the process of addressing this bug, it was determined
this somewhat brittle SQL query generation could be avoided
by carrying over the 'has_diff' column from the diffs dataset
and use that to identify rows in the sparse matrix that had
some kind of diff.

How this addresses that need:
* add 'has_diff' column to sparse matrix, carried over from
diffs dataset
* utilize 'has_diff' column to identify sparse matrix
rows with diffs
* remove dynamic field based 'OR' queries that were problematic
when no fields to check for

Side effects of this change:
* None

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-372

Run Details

5 of 6 new or added lines in 2 files covered. (83.33%)

570 of 592 relevant lines covered (96.28%)

0.96 hits per line