Alan-Jowett / CoPilot-For-Consensus / 20243750613

77%

push

github

Add deterministic hash IDs and update schemas/tests (#260)

* feat(#249): implement deterministic hash-based IDs for audit trail

- Replace UUID-based archive_id with SHA256(file_hash)[:16] for deterministic ingestion
- Add message_key field: SHA256(archive_id|message_id|date|sender|subject)[:16]
- Add chunk_key field: SHA256(message_key|chunk_index)[:16]
- Create message_key_generator utility with generate_message_key() and generate_chunk_key()
- Update ingestion service to use file_hash for archive_id
- Update parsing service to compute and store message_key
- Update chunking adapter to use generate_chunk_key() instead of uuid4()
- Update chunking service to pass message_key and store chunk_key
- Update all schemas to use hash-based IDs instead of UUIDs
- Update event schemas for archive_id hash format
- Update all documentation and examples with new ID format
- Enables idempotent processing and complete audit trail from source file

Signed-off-by: Alan Jowett <alan.jowett@microsoft.com>

* fix: make message_key optional in Chunk and Thread dataclasses, update schema validation test data

Signed-off-by: Alan Jowett <alan.jowett@microsoft.com>

* fix: update chunking service test data with proper hash-format archive_id

Signed-off-by: Alan Jowett <alan.jowett@microsoft.com>

* fix: declare schema validation dependency for chunking

Signed-off-by: Alan Jowett <alan.jowett@microsoft.com>

* fix: install chunking deps before chunking in adapter installer

Signed-off-by: Alan Jowett <alan.jowett@microsoft.com>

* fix: update orchestrator integration fixtures for hash chunk/message ids

Signed-off-by: Alan Jowett <alan.jowett@microsoft.com>

* fix: update embedding integration fixtures for hash chunk/message ids

Signed-off-by: Alan Jowett <alan.jowett@microsoft.com>

* fix: update chunking integration fixtures for hash message_key and extended archive_id

Signed-off-by: Alan Jowett <alan.jowett@microsoft.com>

* fix(#260): address PR review comments - re... (continued)

20 of 37 new or added lines in 6 files covered. (54.05%)

4046 of 4879 relevant lines covered (82.93%)

0.94 hits per line