dcdpr / jp / 24534241672

66%

Build Type

push

github

Committed by web-flow

Commit Message

chore(tools): Add markdown-first strategy to `web_fetch` (#570)

The `web_fetch` tool previously fetched pages as HTML and converted to
markdown. Many documentation platforms (Mintlify, GitBook, Fumadocs)
publish a clean markdown twin at `{path}.md`, which is cheaper to
process and produces better output than HTML-to-markdown conversion.

The tool now supports three strategies, controlled via `tool.options`:

- `auto` (default): probe `{url}.md` first; fall back to HTML
- `markdown`: require the `.md` variant; error if unavailable
- `html`: skip the probe and fetch HTML directly (previous behavior)

Per-domain overrides are supported via `options.domains`, e.g.:

```toml
options.domains."github.com" = "html"
options.domains."*.mintlify.app" = "markdown"
```

Defaults ship with `github.com`, `docs.rs`, and `crates.io` pinned to
`html` since those sites don't follow the `.md` convention.

The implementation splits the monolithic `fetch.rs` into three
sub-modules: `html` (old pipeline), `markdown` (new pipeline), and
`options` (strategy/domain rule parsing). Section listing and extraction
now work for both HTML and markdown sources, using GitHub-style heading
slugs as anchor IDs in both cases.

---------

Signed-off-by: Jean Mertz <git@jeanmertz.com>

Coverage Stats

466 of 628 new or added lines in 5 files covered. (74.2%)

3 existing lines in 1 file now uncovered.

22934 of 35677 relevant lines covered (64.28%)

206.95 hits per line