rero / rero-mef / 34322584020

87%

push

github

perf(setup): group the commands into batches

Ports the batch pattern of rero-ils: each `batch_*` function of
scripts/setup_batches writes the command lines of one step, and
`invenio rero batch` runs them in one application process instead of
paying the Flask boot per command. `setup --db_init` runs its 13
commands in 9 s.

* extract the colored message helpers into scripts/messages, which
  setup and test both duplicated
* pass the fixture file to `fixtures create-or-update` instead of
  piping `dojson -i` into it: the pipe passed the JSON through
  unchanged and cost one process per entity
* keep out of the batches what cannot join them: `celery purge` is no
  invenio command, `db destroy` issues a DROP DATABASE that fails once
  the process holds a connection, the tolerated failures need a
  `|| true` that a batch cannot give to one line only, and
  `create_json` is deliberately one process per entity
* require rero-invenio-base 1.3.0, which brings `invenio rero batch`
* lift the `pymarc<5.2.0` pin. The comment blamed the marcxml leader,
  which was a 13 character leader in the places test helper where
  MARC21 wants 24. The real break was silent: `Field.indicators`
  became a namedtuple in 5.2, so the list comparisons of
  do_rero_concepts dropped the general, seeReference and
  seeAlsoReference notes of every RERO concept without raising.
  Tested on 5.2.0 to 5.4.0; 5.1.2 no longer works, hence the floor

Co-Authored-by: Peter Weber <peter.weber@rero.ch>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

3 of 3 new or added lines in 1 file covered. (100.0%)

5466 of 6261 relevant lines covered (87.3%)

0.87 hits per line