Alan-Jowett / CoPilot-For-Consensus / 21308465134

78%

push

github

parsing: rate-limit legacy source_type warnings + migration script (#1020)

* Initial plan

* Add rate-limited logging and migration script for legacy source_type

- Add rate-limiting to reduce log noise: log WARNING once, then INFO
- Create migration script to backfill source_type for legacy archives
- Update tests to verify rate-limiting behavior
- All 107 tests pass (101 parsing + 6 migration script tests)

Co-authored-by: Alan-Jowett <20480683+Alan-Jowett@users.noreply.github.com>

* Add migration documentation for archive source_type backfill

Co-authored-by: Alan-Jowett <20480683+Alan-Jowett@users.noreply.github.com>

* Fix migration script to use pymongo directly and update docs

- Replace DocumentStore interface with direct pymongo usage (per review)
- Add MongoDB connection handling with environment/secret support
- Update tests to mock pymongo MongoClient instead of DocumentStore
- Fix documentation to clarify script cannot run from parsing container
- All 7 migration tests passing

Co-authored-by: Alan-Jowett <20480683+Alan-Jowett@users.noreply.github.com>

* Address code review nitpicks - clarify test comment

Co-authored-by: Alan-Jowett <20480683+Alan-Jowett@users.noreply.github.com>

* Address code review comments - fix docs path, error handling, and test issues

- Replace hardcoded CI path with generic "repository root" instruction
- Fix error handler to preserve total_count when errors occur after counting
- Update test comment to accurately reflect update error scenario (not connection error)
- Remove unused pytest import
- All 7 migration tests passing

Co-authored-by: Alan-Jowett <20480683+Alan-Jowett@users.noreply.github.com>

* Fix CI: Add license header to migration documentation

- Add SPDX-License-Identifier and Copyright header to archive-source-type-migration.md
- Fixes License Header Check CI failure

Co-authored-by: Alan-Jowett <20480683+Alan-Jowett@users.noreply.github.com>

* security: Prevent credential leakage in error ... (continued)

5 of 5 new or added lines in 1 file covered. (100.0%)

8591 of 11082 relevant lines covered (77.52%)

0.87 hits per line