FAIRmat-NFDI / pynxtools-apm / 23415808064

push

github

Apm prepare nomad v142 (#93)

* Code for generic batch processing of collections of legacy atom probe data in various formats used globally into NeXus/HDF5 files

* Utilities for dealing with datasets from multiple machines and every possible country in the world

* Utility tools to inspect content of single files, zip, tar, rar, and 7z compressed files

* linting

* Hashing queue operational, now running, required to get unique object names which especially for ranging definitions file is relevant

* batch queus used for hashing content

* clean-up

* Workflow utilities

* Added draft of extract code and extraction pipeline

* Further work on the extraction given new examples

* Identify workflows for new datasets

* linting

* examples are clean now, remaining work on removing duplicates, thereafter we can decompress

* duplicates removed

* New formats

* Updates on batch processing preparation code

* Store the state of the scripts that were used for unpacking the original files from the scientific community. A total of 340 GB in 1240 files were prepared for the batch-processing, these include typical open-source formats like pos, epos, rrng, rng, apt, but also CamecaROOT-based formats like rhit, hits, and community formats

* Add batch processing script reduce redundant code and rename variables to explain better what the scripts do

* work on linting

* work on linting

* attempting fixing of file paths

* linting

* Refactoring version resolving and adding to the package

* Clean-up, parser scripts seems to work, next step: i) modify chunking in actual reader, ii) hook in newly developed parsers from ifes library, iii) run-through

* suggestion for different chunking strategy

* source code documentation

* modify reader functions that connect to use the ifes library, using chunking and raising the default compression strength to 9, note that this will make the writing part slower but ought to result in smaller files

* Moving out other default... (continued)

81 of 532 new or added lines in 22 files covered. (15.23%)

357 of 2129 relevant lines covered (16.77%)

0.17 hits per line