rm-hull / news-barge / 41225556324

87%

Build Type

push

github

Committed by rm-hull

Commit Message

refactor(scraper): run URL discovery concurrently across sites

Refactor `_discover_urls` to process multiple sites in parallel using
`asyncio.gather`, bounding concurrency with a new `fetch_semaphore`.

```mermaid
sequenceDiagram
    participant Main as main_async
    participant Disc as _discover_urls
    participant Worker as _discover_site_urls (x N)

    Main->>Disc: sites, session, semaphores
    Note over Disc: Create discovery tasks
    Disc->>Worker: asyncio.gather(...)
    par Parallel Discovery
        Worker->>Worker: Fetch feed / listings (fetch_semaphore)
    and
        Worker->>Worker: Fetch feed / listings (fetch_semaphore)
    end
    Worker-->>Disc: (site, site_logger, urls)
    Disc-->>Main: tasks, site_loggers
```

Coverage Stats

22 of 22 new or added lines in 1 file covered. (100.0%)

74 existing lines in 3 files now uncovered.

958 of 1097 relevant lines covered (87.33%)

0.87 hits per line