Ouranosinc / xclim / 10069141429

92%

push

github

Fast MBCn (a la groupies) (#1580)

<!--Please ensure the PR fulfills the following requirements! -->
<!-- If this is your first PR, make sure to add your details to the
AUTHORS.rst! -->
### Pull Request Checklist:
- [x] This PR addresses an already opened issue (for bug fixes /
features)
    - This PR fixes #xyz
- [x] Tests for the changes have been added (for bug fixes / features)
- [x] (If applicable) Documentation has been added / updated (for bug
fixes / features)
- [x] CHANGES.rst has been updated (with summary of main changes)
- [x] Link to issue (:issue:`number`) and pull request (:pull:`number`)
has been added

### What kind of change does this PR introduce?

New `MBCn` TrainAdjust class. The train part finds adjustment factors
for the npdf transform. The adjust part does the rest.

* A single numpy function to perform all rotations of the npdf_transform
makes the process faster
* Grouping is handled using the same logic as in numpy_groupies. I
initially tried to stop using map_blocks by using what I call a the Big
Dataset (BD) solution. It was a dataset that included the group windowed
blocks. This was working well but sometimes caused dask workers to die.
Maybe a better chunking could have solved this problem. But instead of
constructing a BD, we simply loop over blocks, and simply specify time
indices in each block (à la groupies) in the original datasets. The
resulting code is a bit more messy, but it seems to be working well
performance-wise.

The function also changes how windowed group blocks are handled
throughout the computation. Now, a block is preserved its form from
begin to start of the MBCn computation.
* This is in contrast to the current way which was grouping and
ungrouping block between each iteration of the NpdfTransform.
* The standardization is performed on a block
* The univariate bias correction is maintainted as blocks, reordered,
*then* the blocks are ungrouped
* In the sdba noteb... (continued)

200 of 230 new or added lines in 5 files covered. (86.96%)

9049 of 10003 relevant lines covered (90.46%)

5.75 hits per line