OSGeo / gdal / 15183548885
72%

DEFAULT BRANCH: master
Ran
Jobs 1
Files 1904
Run time 4min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 70.967% (+0.007%) from 70.96%
15183548885

push

github

web-flow 
GDALOverviews: Limit external file size in GDALRegenerateOverviewsMultiBand (#12392)

## What does this PR do?
In GDALRegenerateOverviewsMultiBand() (ROMB), when too much memory would be needed for generating an overview block, the overviews are generated in a temporary file and then copied to the actual overview dataset. The size of this temporary file is not limited, and it has extra problems with pixel interleaved output, since the bands are copied one at a time for the full overview.
This issue was reported in issue #12303 and was partially addressed in #12307, which raised the allowable memory limit which triggers the problem code path, but did not limit the size of the temporary file or the pixel interleaved problems.

This PR addresses the remaining problems by chunking the output overview when possible, and using the temporary file path for a single chunk at a time.

## What are related issues/pull requests?
#12303, #12307

## Test case
To trigger the code path in question, set GDAL_OVER_CHUNK_MAX_SIZE_FOR_TEMP_FILE to 10485760 and build power of 2 overviews for a 11000 by 11000 8bit, pixel interleaved RGB MRF with a blocksize of 1024

```
gdal_translate -outsize 11000 11000  -co BLOCKSIZE=1024 any_small_rgb.jpg test_file.mrf
gdaladdo -r average test_file.mrf
```

## Changes

The "chunking" code triggers when a temporary file would be used, but an output chunk (tile) smaller than the whole overview does exist. The chunk would still exceed the memory usage limit. It then calls ROMB() for every chunk, with extra parameters that limit the processing extent.
On the next pass through ROMB, the external file will be created, this time limited to the chunk extent. This part was also modified, using a VRT to effect the scaling, with the correct resampling, before writing the chunk data in the external file.
Finally, the chunk data is transferred from the temporary output to the destination overview.

In addtion, an extra flag had to be added to the Ra... (continued)

175 of 190 new or added lines in 3 files covered. (92.11%)

68 existing lines in 30 files now uncovered.

568406 of 800943 relevant lines covered (70.97%)

236826.33 hits per line

New Missed Lines in Diff

Lines Coverage ∆ File
15
87.25
 0.43% gcore/overview.cpp

Uncovered Existing Lines

Lines Coverage ∆ File
1
85.5
 -0.03% alg/gdalwarpkernel.cpp
1
97.48
 -0.06% apps/gdalalg_raster_tile.cpp
1
7.72
 -0.0% build-coverage/swig/java/org/gdal/gdal/gdal_wrap.cpp
1
91.64
 -0.02% frmts/gtiff/gtiffdataset_write.cpp
1
87.9
 -0.14% frmts/gtiff/gtiffrasterband_read.cpp
1
77.99
 0.0% gcore/gdalrasterband.cpp
1
74.1
 -0.09% ogr/ogrsf_frmts/mitab/mitab_mapfile.cpp
1
88.11
 0.13% port/cpl_minixml.cpp
1
79.47
 0.41% port/cpl_multiproc.cpp
1
87.03
 0.24% port/cpl_vsil_curl_streaming.cpp
2
17.5
 -1.67% frmts/gff/gff_dataset.cpp
2
91.34
 -0.06% frmts/gtiff/gtiffdataset_read.cpp
2
79.1
 -0.39% frmts/gtiff/libtiff/tif_predict.c
2
56.64
 0.0% frmts/gtiff/libtiff/tif_write.c
2
39.94
 -0.32% frmts/raw/cpgdataset.cpp
2
83.54
 -0.04% gcore/gdaldataset.cpp
2
87.25
 0.43% gcore/overview.cpp
2
92.52
 -0.45% ogr/ogrsf_frmts/mvt/mvt_tile.cpp
2
89.7
 -0.18% port/cpl_string.cpp
2
40.25
 0.0% port/cpl_virtualmem.cpp
3
81.33
 -0.4% alg/gdalpansharpen.cpp
3
66.47
 0.3% frmts/gtiff/libtiff/tif_read.c
3
91.42
 0.09% gcore/rasterio.cpp
3
91.57
 -0.6% port/cplstringlist.cpp
4
53.97
 -0.24% frmts/gtiff/libtiff/tif_dir.c
4
34.1
 -0.02% frmts/gtiff/libtiff/tif_dirread.c
4
69.06
 -1.51% port/cpl_vsisimple.cpp
4
71.78
 0.29% third_party/LercLib/Lerc2.h
5
54.4
 0.0% frmts/gtiff/libtiff/tif_dirwrite.c
5
75.73
 -0.35% ogr/ogrct.cpp
Jobs
ID Job ID Ran Files Coverage
1 15183548885.1 1904
70.97
 GitHub Action Run
Source Files on build 15183548885