Alan-Jowett / CoPilot-For-Consensus / 20791556200

76%

push

github

Use ArchiveStore abstraction for storage-agnostic ingestion and parsing (#782)

* Initial plan

* Integrate ArchiveStore into ingestion and parsing services

- Add ArchiveStore initialization to ingestion service with configurable backend
- Update ingestion to store files via ArchiveStore.store_archive()
- Remove file_path from ArchiveIngested events (now storage-agnostic)
- Deprecate local checksums.json cache, use document store for deduplication
- Add ArchiveStore initialization to parsing service
- Update parsing to retrieve files via ArchiveStore.get_archive()
- Support storage-agnostic ArchiveIngested events in parsing
- Add temporary file handling in parsing for mbox parser compatibility

Co-authored-by: Alan-Jowett <20480683+Alan-Jowett@users.noreply.github.com>

* Add ArchiveStore configuration and update event schemas

- Add ARCHIVE_STORE_TYPE and ARCHIVE_STORE_PATH env vars to docker-compose
- Update ArchiveIngested event schema to make file_path optional/deprecated
- Add migration notes for storage-agnostic event consumption
- Document Azure Blob Storage configuration options
- Default to local backend for backward compatibility

Co-authored-by: Alan-Jowett <20480683+Alan-Jowett@users.noreply.github.com>

* Add unit tests for ArchiveStore integration

- Add test for ingestion service initialization with ArchiveStore
- Add test for archive deduplication via document store
- Add test for delete_checksums_for_source with document store
- Add test for parsing service initialization with ArchiveStore
- Add test for parsing archive retrieval from ArchiveStore

Co-authored-by: Alan-Jowett <20480683+Alan-Jowett@users.noreply.github.com>

* Fix duplicate method calls and update ArchiveMetadata docs

- Remove duplicate _save_ingestion_log and _write_archive_record calls
- Add deprecation notices to ArchiveMetadata.file_path documentation
- Align internal model documentation with event schema changes

Co-authored-by: Alan-Jowett <20480683+Alan-Jowett... (continued)

113 of 155 new or added lines in 2 files covered. (72.9%)

6282 of 8183 relevant lines covered (76.77%)

0.86 hits per line