loot / libloot / 25071265585

push

github

Store asset paths instead of hashes

This avoids the issue of hash collisions, within any one archive, a plugin's archives, or between different plugins' archives. That means that sorting has accurate data on what the asset counts loaded by plugins are, and what the asset overlap between plugins, so it can make better decisions when adding overlap edges.

It does require that the archives include the folder and file names, which is apparently not strictly required, but it seems that having the names is practically required[1][2], and I verified that the parsing code can handle archives from all of the supported games (apart from Morrowind, which doesn't have its BSAs read by LOOT). That testing covered 321 archives containing 771246 folder records and 3404007 file records, including some mod BSAs/BA2s.

If an archive doesn't have the flags set for containing folder and file names, then libloot will log an error and effectively ignore its contents, so the new behaviour means that some archives that were (potentially inaccurately) taken into account before may now be ignored. Falling back to using the hashes for archives that don't have names would be an option, but that complicates comparison against archives that do have names, and I have no evidence that the fallback would be useful in practice.

This doubles the size of each map key and set entry (from 8 to 16 bytes), introduces another level of indirection when comparing keys or values, and also means that the name strings need to be stored, further increasing memory usage.

I tested the performance impact when sorting Skyrim SE and Starfield load orders, using LOOT v0.29.1 and comparing against using it with libloot v0.24.4:

- The Skyrim SE load order had 1630 plugins and 18 BSAs totalling 3.14 GB (not including Skyrim.esm's or its BSAs, which LOOT doesn't fully load).
  - Loading plugins (which includes parsing BSAs) was ~ 2% (5 ms) faster
  - Sorting was ~ 1% (21 ms) faster
  - Memory usage aft... (continued)

128 of 147 new or added lines in 6 files covered. (87.07%)

12347 of 13103 relevant lines covered (94.23%)

143.08 hits per line