hathitrust / hathifiles_database / 10742902177

96%

push

github

Dev 1335 date independence (#18)

* DEV-1335 Make hathifiles_database Date Independent
- Add `Log` class for recording file + timestamp in `hf_log` table.
- Add `Hathifiles` class for producing agenda of files to load.
- Add `exe/hathifiles_database_full_update` script for bringing database up to date.
- Update README with `exe/` inventory and notes.
- The existing code was getting too many false changes on ISSNs in monthly delta
  - Loosened restriction on input file format to accommodate more database-like values (allow 0/1 for `access`)
  - Add more tests for round-trip data fidelity -- one should be able to load any hathifile, and the delta with itself should be empty.
- Address Dependabot #10 REXML denial of service vulnerability
- TIDY
  - Remove dead code after __END__ blocks
  - Address issue #11 Remove wait-for and use healthchecks
- Address #8 add prometheus / pushgateway
  - Batch up the calls to milemarker instead of calling for each INSERT
- Monthly update bucket chain must `sort` after `cut` to keep `comm` happy.

83 of 86 new or added lines in 9 files covered. (96.51%)

516 of 538 relevant lines covered (95.91%)

632.96 hits per line