Alan-Jowett / CoPilot-For-Consensus / 21088803607

78%

push

github

Make ingestion source storage configurable (#896)

* Initial plan

* feat: add configurable source storage backends for ingestion

- Add sources_store_type and sources_file_path config fields
- Support document_store (default) and file backends
- Update scheduler to use configured backend (fixes split-brain issue)
- Merge startup sources into document_store for migration support
- Add comprehensive tests for both backends

Co-authored-by: Alan-Jowett <20480683+Alan-Jowett@users.noreply.github.com>

* docs: add documentation for source storage backends

Update README with configuration options for sources_store_type
and sources_file_path, including migration path

Co-authored-by: Alan-Jowett <20480683+Alan-Jowett@users.noreply.github.com>

* refactor: address code review feedback

- Remove unused settings variable in test
- Use dataclasses.asdict() instead of __dict__ for proper serialization
- Optimize get_stats() to avoid double filtering of sources

Co-authored-by: Alan-Jowett <20480683+Alan-Jowett@users.noreply.github.com>

* fix: address PR review comments

- Fix double filtering in _get_enabled_sources_for_ingestion()
- Remove unused imports (tempfile, pytest, ServiceConfig_Ingestion, ServiceSettings_Ingestion, asdict)
- Fix inconsistent env var name in README documentation

Co-authored-by: Alan-Jowett <20480683+Alan-Jowett@users.noreply.github.com>

* fix: add validation and improve robustness for sources_store_type

- Add explicit validation for sources_store_type in both service.py and main.py
- Log warnings and default to 'document_store' for invalid values
- Improve schema description to document invalid value behavior
- Fix type annotation in main.py (list[dict[str, Any]])
- Add test coverage for invalid sources_store_type value

Co-authored-by: Alan-Jowett <20480683+Alan-Jowett@users.noreply.github.com>

* Changes before error encountered

Co-authored-by: Alan-Jowett <20480683+Alan-Jowett@users.noreply.github.com>

---------

Co-authored-b... (continued)

42 of 48 new or added lines in 2 files covered. (87.5%)

7815 of 10001 relevant lines covered (78.14%)

0.87 hits per line