Alan-Jowett / CoPilot-For-Consensus / 21955084380

78%

push

github

fix: reporting UI shows stale results due to in-memory sort buffer overflow (#1164)

* fix: reporting UI shows stale results due to in-memory sort buffer overflow

The reporting service's get_reports() fetched only the first
limit+skip+100 summaries in arbitrary DB order, then sorted
in-memory. With 31K+ summaries, threads newer than ~position 120
in insertion order were never returned to the UI.

Root cause: query_documents() had no sort support, forcing
in-memory sort with a fixed buffer (METADATA_FILTER_BUFFER_SIZE=100).

Fix:
- Add sort_by/sort_order params to DocumentStore.query_documents()
  interface and all implementations (Cosmos, Mongo, InMemory, Validating)
- Denormalize first_message_date/last_message_date from threads into
  summary documents (schema + reporting service process_summary())
- Use DB-level ORDER BY in get_reports() instead of in-memory sort
- Add first_message_date_idx to summaries collection config

Existing 31K summaries were backfilled via a one-time migration script.

Signed-off-by: Alan Jowett <alanjo@microsoft.com>

* fix: address PR review — init thread_docs, handle None in sort, add sort_by mapping test

- Initialize thread_docs as empty list before try block to prevent
  UnboundLocalError if the thread lookup throws
- Log exception with exc_info=True for diagnosability
- Coerce None values to empty string in InMemory sort key to avoid
  TypeError when documents have null sort fields
- Add test_get_reports_passes_sort_by_to_query_documents verifying
  thread_start_date maps to first_message_date, generated_at passes
  through, and no sort_by passes None

Signed-off-by: Alan Jowett <alanjo@microsoft.com>

* fix: update query_documents wrappers in tests for new sort params

Update all test helper functions that wrap query_documents to accept
the new sort_by/sort_order parameters:
- embedding/tests/test_integration.py: _add_in_operator_support
- chunking/tests/conftest.py: create_query_with_in_support, enhanced_query
- ... (continued)

47 of 66 new or added lines in 6 files covered. (71.21%)

8793 of 11297 relevant lines covered (77.83%)

0.87 hits per line