Ouranosinc / xscen / 26974600752
90%

DEFAULT BRANCH: main
Ran
Jobs 11
Files 17
Run time 1min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 90.079%. Remained the same
26974600752

push

github

web-flow 
Use more memory when opening catalogs to support all cases (#747)

<!-- Please ensure the PR fulfills the following requirements! -->
<!-- If this is your first PR, make sure to add your details to the
AUTHORS.rst! -->
### Pull Request Checklist:
- [ ] This PR addresses an already opened issue (for bug fixes /
features)
    - This PR fixes #xyz
- [ ] (If applicable) Documentation has been added / updated (for bug
fixes / features).
- [ ] (If applicable) Tests have been added.
- [x] This PR does not seem to break the templates.
- [x] CHANGELOG.rst has been updated (with summary of main changes).
- [x] Link to issue (:issue:`number`) and pull request (:pull:`number`)
has been added.

### What kind of change does this PR introduce?
When reading a large CSV, pandas will default to parsing it in chunks
and merge those later. However, when using categorical columns, the
merge is not successful if the different chunks have different dtypes
for the categories.

In our case, this can happen when a given column was empty for the whole
chunk, yielding a value of "NaN" and this a "categorical" dtype of
Float. Other chunks where this column has at least one value will use
the categorical dtype of that value, most likely string.

It seems to be only happening in pandas 3, the catalog giving me this
error was openable with pandas 2.2.3. I think this is specific to the
"c" parser, which is the default.

I think we were simply lucky up to now in that our catalogs were small
enough to use only one chunk, or that the first chunk never had
completely empty columns. In my case, I stumbled upon this with the new
"simulation" catalog on Narval when I added the MRCC5-CMIP6 data to it.

Solutions I thought of:
- Modify `ouracat.py` to "shuffle" the catalog so that the first chunk
always has values in the categorical columns : but I was like « ugh »
and didn't to it
- Remove the use of categorical columns
- Switch to another parser. But "pyarrow" doesn't work because we are
us... (continued)

4440 of 4929 relevant lines covered (90.08%)

9.74 hits per line

Jobs
ID Job ID Ran Files Coverage
1 run-Python3.11-pypi-ubuntu-latest - 26974600752.1 17
87.54
 GitHub Action Run
2 run-Python3.14-pypi-ubuntu-latest - 26974600752.2 17
87.52
 GitHub Action Run
3 run-Python3.12-pypi-macos-latest - 26974600752.3 17
87.48
 GitHub Action Run
4 run-Python3.12-pypi-ubuntu-latest - 26974600752.4 17
87.54
 GitHub Action Run
5 run-Python3.12-conda-ubuntu-latest - 26974600752.5 17
89.86
 GitHub Action Run
6 run-Python3.10-pypi-ubuntu-latest - 26974600752.6 17
87.3
 GitHub Action Run
7 run-Python3.13-pypi-ubuntu-latest - 26974600752.7 17
87.54
 GitHub Action Run
8 run-Python3.10-conda-ubuntu-latest - 26974600752.8 17
89.53
 GitHub Action Run
9 run-Python3.14-conda-ubuntu-latest - 26974600752.9 17
89.86
 GitHub Action Run
10 run-Python3.13-conda-ubuntu-latest - 26974600752.10 17
89.86
 GitHub Action Run
11 run-Python3.11-conda-ubuntu-latest - 26974600752.11 17
89.86
 GitHub Action Run
Source Files on build 26974600752