Alan-Jowett / CoPilot-For-Consensus / 20219704840

78%

push

github

fix: implement queue draining and improve duplicate error handling (#232)

- Fix queue consumption pattern: services now consume from event-type queues (archive.ingested, json.parsed, chunks.prepared, summary.complete) instead of service-specific queues
- Improve error handling in parsing service: DuplicateKeyError and DocumentValidationError are now caught and skipped instead of blocking entire archive processing
- Allow individual messages/threads with permanent errors to be skipped while continuing to process the rest of the archive
- Update schema validation to allow empty 'from.name' and 'subject' fields
- Verify full end-to-end pipeline: all queues drain properly when services are running

This fixes the issue where the archive.ingested queue was not draining because parsing service was consuming from the wrong queue (parsing-service instead of archive.ingested), and where duplicate key errors would cause the entire archive to fail and be requeued infinitely.

Co-authored-by: Alan Jowett <alan.jowett@microsoft.com>

14 of 32 new or added lines in 1 file covered. (43.75%)

1229 of 1460 relevant lines covered (84.18%)

0.84 hits per line