rero / rero-mef / 34472291150

87%

staging: 87%

push

github

feat(concepts): rebuild the association and stop storing nameless records

* move the rebuild out of alembic into `invenio utils
  rebuild-concept-association`, which commits each record on its own and
  resumes with `--from-pid`. Alembic holds one transaction open for the
  whole run and loses every committed record when the connection dies.
  Revision `dcdc05a29568` now only checks the concept index mappings.
* discard a record the source deleted before we held it, and delete one
  that loses its heading together with the MEF record it empties. A
  record stating a `relation_pid` is kept, and keeps its stored heading:
  `get_latest` reads it to forward the old pid to the new record.
* keep the stored `deleted` stamp on a re-delivered tombstone, so
  `--md5` skips it instead of rewriting it on every harvest.
* stop inventing `TAG: <tag> NOT FOUND` for a record with no heading
  field, so the schema refuses it. Only the missing field is tolerated,
  and a transformation that broke no longer looks like one the source
  stopped naming.
* guard the transformation in `marc_to_json`: one unreadable record is
  counted and written to the error file instead of aborting the file.
* block IdRef, GND and VIAF in the tests; `@pytest.mark.online(...)`
  opens one, `--allow-online` opens all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

40 of 88 new or added lines in 8 files covered. (45.45%)

5627 of 6460 relevant lines covered (87.11%)

0.87 hits per line