rm-hull / news-barge / 36175451397

94%

push

github

feat: add flair classifier for NER (named entity recognition) (#43)

* feat: add NER classifiers and slug renaming scripts

- Add multiple NER extraction scripts using Flair, spaCy, Google GenAI,
  and LiteRT for article entity backfilling.
- Add utility to recompute and rename article slugs based on site
config.

* refactor(scraper): integrate flair named entity recognition into
pipeline

Move Flair-based NER logic and helper functions into core modules to
automatically extract people, locations, and organizations during
article processing.

* ci: cache flair NER model in CI workflows

Add caching for the flair named entity recognition (NER) model in both
the build and scrape GitHub Actions workflows to speed up pipeline
execution times.

* style(scraper): use builtin types and clean up unused imports

- Update `NamedEntities` dataclass to use built-in `list` instead of
`List`.
- Remove unused `yaml` import from output module and sort imports in
pipeline and tests.

* style(scraper): add type hints to filter_duplicate_names

* fix(scraper): extract named entities from markdown body

Update `process_article` to extract named entities from `md_body`
instead of `description` for richer entity recognition across the
entire article content.

* ci(workflow): skip deployment on direct branch pushes

Prevent automatic deployment when code is pushed to branches, reserving
build-and-deploy for scheduled scrapes with changes and manual triggers.

* Dont automatically run on branch push

* refactor: defer expensive metadata generation

Move `article_categories` and `named_entities` calls after the already-exists & dry-run
checks to prevent unnecessary computation.

```mermaid
sequenceDiagram
    participant P as Pipeline
    P->>P: Check dry-run
    alt is dry-run or already exists
        P->>P: Return True
    else actual run
        P->>P: Generate categories
        P->>P: Extract entities
        P->>P: Write file
    end
```

* refactor: deduplicate entity... (continued)

51 of 64 new or added lines in 3 files covered. (79.69%)

499 of 536 relevant lines covered (93.1%)

0.93 hits per line