Ouranosinc / xscen / 35857258869

90%

Build Type

push

github

Committed by web-flow

Commit Message

Add possibility to "unstack" iterable columns in catalog (#763)

<!-- Please ensure the PR fulfills the following requirements! -->
<!-- If this is your first PR, make sure to add your details to the
AUTHORS.rst! -->
### Pull Request Checklist:
- [x] This PR addresses an already opened issue (for bug fixes /
features)
    - This PR fixes #762 
- [x] (If applicable) Documentation has been added / updated (for bug
fixes / features).
- [x] (If applicable) Tests have been added.
- [x] This PR does not seem to break the templates.
- [x] CHANGELOG.rst has been updated (with summary of main changes).
- [x] Link to issue (:issue:`number`) and pull request (:pull:`number`)
has been added.

### What kind of change does this PR introduce?

* `DataCatalog.unstack(col)` will take the iterables in "col" and create
one entry per element, copying all other fields over.
* `DataCatalog.stack(col)` does the opposite : for all groups where all
columns except "col" are the same, merge entries by making "col" a
tuple.
* `search_data_catalogs` now will `unstack` before filtering and `stack`
(if needed) at the end.

`search_data_catalogs` has one step where it keeps only the coarsest
frequencies for each variable in the result. But by grouping on
"variable" is doesn't see that "(XYZ,)" is similar to "(XYZ, ABC)". That
was my bug in #762 .

In general, I think this can help for some filtering stuff. Once
unstacked, the datasets seems to open correctly, even though the same
paths are opened multiple times. So, re-stacking is not required, one
can keep the non-iterable variable column.

### Does this PR introduce a breaking change?
I don't think so, but the test suite might not cover all
`search_data_catalogs` cases...


### Other information:
The way this is done might not be compatible with the newest intake
esm... We'll see!

This also introduces a new sample catalog and dataset to use in the
tests. We didn't have any multi-variable catalog.

Coverage Stats

45 of 48 new or added lines in 2 files covered. (93.75%)

1 existing line in 1 file now uncovered.

4477 of 4965 relevant lines covered (90.17%)

9.75 hits per line