geo-engine / geoengine / 41039847285

88%

main: 88%

Build Type

push

github

Committed by web-flow

Commit Message

feat: query runtime decoupling (#1364)

* feat(backend): run queries on a shared processing runtime instead of the HTTP worker

actix runs every HTTP worker as a single-threaded runtime, so a query's
whole stream used to run on the worker that accepted the request.

Queries now run as tasks on the server's main multi-thread runtime
(`QueryContext::processing_runtime`, set by `PostgresContext`) and hand
finished items back through a bounded channel (`spawn_query`,
`spawn_plot_query`). Dropping the consumer aborts the task, errors are
forwarded as items and a panic ends the stream with an error.

Used by the PNG/GeoTIFF utils (WMS, WCS, OGC API tiles/coverages,
dataset from workflow), WFS, plots and the websocket streams.

* add plan document (temporary)

* feat(backend): run query consumers and tasks on the processing runtime

The previous step moved query streams off the HTTP workers, but their
consumers still ran there: PNG blit and encode, WCS GeoTIFF writing,
WFS GeoJSON merging, websocket Arrow IPC encoding and background tasks.

- `spawn_query_task` runs a whole query (query, fold, encode) as one
  abort-on-drop task on the processing runtime; `spawn_stream` does the
  same for streams with work mapped onto them, `spawn_query` builds on it
- PNG and GeoTIFF encoding run in `spawn_blocking` inside that task
- plots use `spawn_query_task`, `spawn_plot_query` is gone
- the task manager spawns tasks on the processing runtime
- failed query tasks are logged with their panic message
- Rayon threads are named `ge-rayon-{i}`

* test(backend): check that PNG queries run on the processing runtime

MockQueryContext gets an optional processing runtime. The test renders
the NDVI PNG with a separate, named runtime and checks that the query
and every tile poll happen on its threads.

* make Tokio Handle not optional

* fix(backend): give the PNG runtime test its processing runtime again and restore docs

* refactor(backend): address review: non-optional task ru... (continued)

Coverage Stats

403 of 425 new or added lines in 14 files covered. (94.82%)

1 existing line in 1 file now uncovered.

137151 of 156068 relevant lines covered (87.88%)

538222.21 hits per line