Tatsh / incoming-extractor / 28308623766

100%

push

github

project: run conversions concurrently with an async worker pool

Enter asyncio.run() from main() right after setup_logging() and move the
conversion pipeline into async code. convert_file is now async and runs its
synchronous converter in a worker thread; run_conversions enqueues every job and
gathers a pool of worker tasks that consume the queue, modelled on the
patreon-archiver worker layout. Add a -j/--jobs option (default: CPU count) to
size the pool. Paths in the async layer use anyio.Path.

Add anyio as a runtime dependency and pytest-asyncio for the async tests; mirror
both into .wiswa.jsonnet and set asyncio_mode to strict.

Signed-off-by: Andrew Udvare <audvare@gmail.com>

130 of 130 branches covered (100.0%)

Branch coverage included in aggregate %.

72 of 72 new or added lines in 2 files covered. (100.0%)

1213 of 1213 relevant lines covered (100.0%)

5.0 hits per line