hydrologie / xhydro / 13685183489

92%

push

github

Major refactoring of sampled_indicators (#240)

<!-- Please ensure the PR fulfills the following requirements! -->
<!-- If this is your first PR, make sure to add your details to the
AUTHORS.rst! -->
### Pull Request Checklist:
- [x] This PR addresses an already opened issue (for bug fixes /
features)
  - This PR fixes #xyz
- [x] (If applicable) Documentation has been added / updated (for bug
fixes / features).
- [x] (If applicable) Tests have been added.
- [x] CHANGELOG.rst has been updated (with summary of main changes).
- [x] Link to issue (:issue:`number`) and pull request (:pull:`number`)
has been added.

### What kind of change does this PR introduce?

* Turns out that `sampled_indicators` was too memory-inefficient for
bigger applications and could quickly grow to 10s or even 100s of Gb of
RAM. Thus, the function has been split in two to allow for writing the
results of the sampling on disk and offload some of the RAM usage. No
miracle can be done, but this should be slightly better.
* The default number of iterations was reduced from 50,000 to 5000.
* Uniform weights are now only added to extra dimensions specifically
named in `include_dims`, instead of being automatic. The dimensions
`time` and `horizon` are still never sampled.
* `_weighted_sampling` was almost completely rewritten to hopefully make
better use of chunks and multithreading through `dask`.

### Does this PR introduce a breaking change?

- Yes. While this does not affect the numerical results themselves, this
is an almost complete refactoring of `sampled_indicators`, from the
inputs required to the outputs produced.


### Other information:

66 of 67 new or added lines in 1 file covered. (98.51%)

2428 of 2586 relevant lines covered (93.89%)

3.4 hits per line