MITLibraries / transmogrifier-ab-diff / 11820448005

87%

main: 86%

Pull #65

github

Use parquet globbing vs registering datasets

Why these changes are being introduced:

For very large runs, the previous code attempted to register
a pyarrow dataset python object as a DuckDB table.  This works
for smaller tables, but caused an out of memory (OOM) error
as it called to_table() which brought the entire dataset into
memory.

How this addresses that need:
* Utilize parquet globbing, like we do elsewhere, to ensure
that a full input dataset is not brought into memory.

Side effects of this change:
* No OOM errors during final records dataset creation for
large runs.

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/TIMX-389

9 of 11 new or added lines in 3 files covered. (81.82%)

828 of 955 relevant lines covered (86.7%)

0.87 hits per line