informatics-isi-edu / deriva-mcp-core / 24225766165

90%

Build Type

push

github

Committed by mikedarcy

Commit Message

Phase 7: generalized RAG sources, dataset enrichment, resources, and prompts

  RAG subsystem (rag/):
  - WebCrawler + WebSource: BFS async crawler for public websites; rate
    limiting, domain scoping, path prefix filter, content-hash dedup
  - LocalSource: index local filesystem Markdown trees
  - Dataset enricher hook: ctx.rag_dataset_indexer() lets plugins register
    a per-row async enricher called on catalog connect; enriched chunks
    carry url + title metadata fields; ChromaDB _ensure_client uses
    double-checked locking (self._client assigned last to prevent race)
  - TTL-guarded enrichment: failed TTL check logs and returns, does not
    fall through to a spurious re-enrichment run
  - rag_import_chunks: bulk import tool for pre-built chunk payloads
  - VectorStore.add(): non-deleting append path for batch ingestion

  Plugin API (plugin/api.py):
  - ctx.rag_web_source(), ctx.rag_local_source(), ctx.rag_github_source(),
    ctx.rag_dataset_indexer() -- declarative RAG registration for plugins
  - RagGitHubSourceDeclaration renamed from RagSourceDeclaration;
    ctx.rag_github_source() renamed from ctx.rag_source()

  Built-in resources (tools/resources.py):
  - deriva://server/status
  - deriva://catalog/{hostname}/{catalog_id}/schema
  - deriva://catalog/{hostname}/{catalog_id}/tables
  - deriva://catalog/{hostname}/{catalog_id}/table/{schema}/{table}

  Built-in prompts (tools/prompts.py): query_guide, entity_guide,
  annotation_guide, catalog_guide MCP prompt resources.

Coverage Stats

554 of 710 new or added lines in 15 files covered. (78.03%)

4 existing lines in 2 files now uncovered.

3071 of 3461 relevant lines covered (88.73%)

0.89 hits per line