FAIRmat-NFDI
/
pynxtools
/
25801914969
/
1
82%
master: 82%
master: 82%
|
Ran
|
Files
76
|
Run time
4s
|
Badge
Embed ▾
README BADGES
|
coverage: 81.852% (+0.3%) from 81.56%
push
github
web-flowMemory optimization PYNXTOOLS validate (#752) * wip * fixed tests * update default_en.txt as changes occurred on the nomad-FAIR side 3672d5cd3 * add vscode config to debug the validation, key edit in hdf-specific version of is_valid_enum which defers the loading of data to occurr only when required, most h5py.Datasets do not qualify as an enum. This change could also in future version of the template-base validation be added. That is not expected to be so performance improving as in the case of template based validation prior to NeXus file writing all data are already in main memory. For HDF5 though they need to loaded first from disk, so far the clean_str_attr(dataset[()]) did that ALWAYS and for all datasets irrespective their size, just to discard most of these data with the next two lines when the code figures the dataset is not an instance of an enum concept * remove * update definitions to align with those used in #768 that fix nametype resolving issues for em * bump ruff for precommit to align with #769 * cherry picking of changes from #768 * update launch config for validation of large tests, finished running initial round of tests with float32 there validation is now blazing fast for all three cases (contiguous, chunked uncompressed, chunked compressed), the substantial speedup is not a signature of failure or bypassing certain function calls but expected, in the contiguous float32 test case the concept entry1/measurement/event1/image1/real is float32 yes its 10GB uncompressed payload. But keep in mind that h5py and HDF5 store datasets of elementary datatypes always as type-homogeneous arrays, that means a test if the field is valid reduces to a simple test of the dtype arribute against accepted ones, no need to load the data at all, unless we have NX_POSINT as datatype. Similar expected speedup signified alsso compared to v0.13.2 is valid enum, in that version the data were always fully loaded first ju... (continued)
6251 of 7637 relevant lines covered (81.85%)
0.82 hits per line
