FAIRmat-NFDI / pynxtools / 25662515105

82%

Build Type

push

github

Committed by web-flow

Commit Message

Memory optimization NOMAD nexus parser (#750)

* incomplete skeleton, untested, unfinished, next steps, implement statistics, for chunked need to compute by hand, take care for complex values

* Welford's algorithm skeleton

* skeleton for refactoring field_statistics data structure

* fix bug in _get_value

* first test version

* typing

* bugfixing

* done with a draft of the feature, conflict of interest remains, stable welford needs incremental updates which wont end up getting vectorized, the only issue that this comes at all up is because we wish to take the mean value (and show population standard deviation), we should consider to keep it with min and max and just pick the first value, it will avoid using Welford and alike and with ought to speed up the parsing significantly, the mean value of a coordinate axis value triplet 1, 0, 0 with 0.333 and stdev is anyway of limited value, I understood @rettigl such that uses summary stats for navigation, might need discussion for a better compromise, mind that also np.mean and np.std even on cotiguous array are not without imprecisions, in edge cases eventually even weaker than Welford, I am supportive though that for something that is just a value to show in the GUI in NOMAD we should not invoke an eventually very costly algorithm, I kept both the incremental (non vectorized) as well as the numpy batch implementation to support the discussion, my preference is to remove mean and stdev altogether, if people are interested in this, they should compute it in pynxtools-plugin, when they have that dataset anyway at some point in main memory, we should not abuse the parser for these computations, as currently with contiguous storage layout they create memory consumption spikes.

* fix assignment bug

* alternative approach, drop std and go with naive summation by default (#761)

* alternative approach, drop std and go with naive summation by default

* keep std deactivated

---------

Co-authored-by: mkueh... (continued)

Coverage Stats

162 of 191 new or added lines in 4 files covered. (84.82%)

2 existing lines in 1 file now uncovered.

6108 of 7489 relevant lines covered (81.56%)

0.82 hits per line