pulibrary / tigerdata-app / c8ab3ca0-aa05-470c-8cd5-1fe6fc8ca625

91%

Build Type

push

circleci

Committed by web-flow

Commit Message

File Inventory speed optimization (#2416)

Refactored the file inventory to "walk the tree" rather than fetching
the tree for the project all at once. This requires creating new
iterators for each folder within the collection and fetching them
individually but it seems to perform much better than the original
approach.

The downside of the new approach is that we need to calculate each
file's path as we go but because the culprit in the original approach
was the [time that it takes Mediaflux to fetch the
path](https://github.com/pulibrary/tigerdata-app/issues/1274#issuecomment-2710860502)
this is a worthwhile compromise.

In QA a test project with a copy of the "Project G" data (`mediaflux_id
= 1083`) with 189,000 files takes `690 seconds` with the new method
compared with `2300 seconds` with the original code.

`690 seconds` (`~11 minutes`) is still a long time, but much better than
the original `2300 seconds` (`~37 minutes`) and not too bad for a job
that runs in the background.

Closes #1344

Run Details

51 of 52 new or added lines in 3 files covered. (98.08%)

4 existing lines in 1 file now uncovered.

3139 of 3451 relevant lines covered (90.96%)

591.86 hits per line