Alan-Jowett / CoPilot-For-Consensus / 21101385375

78%

Build Type

push

github

Committed by web-flow

Commit Message

Add option to cascade delete associated data when source is removed (#913)

* Initial plan

* Implement cascade delete functionality for source deletion

- Add delete_source_cascade() method in IngestionService
  - Deletes archives from document_store and archive_store
  - Deletes threads, messages, chunks, and summaries
  - Logs warnings for embeddings (handled by embedding service)
  - Comprehensive audit logging for all deletions
  - Emits metrics for observability
- Update delete_source() to support cascade parameter
- Update DELETE /api/sources/{name} endpoint to accept cascade query param
- Add comprehensive tests for cascade delete functionality
- Maintain backward compatibility (cascade=false by default)

Co-authored-by: Alan-Jowett <20480683+Alan-Jowett@users.noreply.github.com>

* Add cascade delete documentation to ingestion README

Co-authored-by: Alan-Jowett <20480683+Alan-Jowett@users.noreply.github.com>

* Address code review feedback: refactor cascade delete

- Add helper function _extract_doc_ids() to reduce code duplication
- Improve terminology: "embeddings associated with deleted chunks"
- Use helper function consistently throughout cascade delete method
- Remove redundant str() conversions when IDs are already strings

Co-authored-by: Alan-Jowett <20480683+Alan-Jowett@users.noreply.github.com>

* Fix indentation and message_ids initialization

- Fix incorrect indentation in logger.warning call
- Initialize message_ids before try block to preserve IDs even if deletion fails
- Ensures chunks can still be queried by message_id in partial failure scenarios

Co-authored-by: Alan-Jowett <20480683+Alan-Jowett@users.noreply.github.com>

* Fix None handling and add comprehensive tests

- Add `or []` to all query_documents calls to handle None returns
- Document cascade delete behavior when it fails (source not deleted)
- Add test for partial failure scenario with non-blocking error handling
- Add test to verify archive_store.delete_archive... (continued)

Run Details

104 of 129 new or added lines in 2 files covered. (80.62%)

1 existing line in 1 file now uncovered.

7826 of 10037 relevant lines covered (77.97%)

0.88 hits per line