lsm / neokai / 26006380712

81%

Build Type

push

github

Committed by web-flow

Commit Message

feat(providers): add custom OpenAI-compatible endpoint support (#1929)

* feat(providers): add custom OpenAI-compatible endpoint support

Adds backend support for user-defined OpenAI-compatible endpoints (LM Studio,
vLLM, LiteLLM, self-hosted deployments, etc.) without requiring an external
proxy.

Each endpoint is persisted as JSON under `settings.customEndpoints` and
registered in the provider registry as `custom:<endpointId>`. Traffic is
routed through an embedded Anthropic Messages ↔ OpenAI Chat Completions
bridge that handles message translation, streaming tool_calls accumulation,
graceful image dropping when vision isn't supported, and per-model
capability gating.

Backend only — UI for endpoint management ships in a follow-up PR.

Acceptance criteria covered:
- Tool use works end-to-end via tool_calls deltas → Anthropic tool_use blocks
- Streaming works for both Anthropic and OpenAI-compatible providers
- Unsupported features (vision, tool use) are silently dropped per model
- Built-in Anthropic provider behaviour is unchanged
- Existing Ollama provider keeps working; new bridge is for generic OAI-compat

* fix(providers): bind custom-endpoint bridge to loopback + honour live thinkingModes

Addresses review feedback on #1929:

P1 — Security: the OpenAI Chat Completions bridge was binding to 0.0.0.0
because `Bun.serve({ port: 0 })` defaults the hostname. On multi-user hosts
another local user could probe ephemeral ports and reach the bridge with the
configured upstream API key. Now explicitly binds to 127.0.0.1.

P1 — Correctness: `addSessionStateOptions` resolved provider thinking mode
exclusively from the static `PROVIDER_THINKING_MODES` map, which doesn't list
`custom:*` providers and therefore defaulted to `'granular'`. That emitted
`thinking: { type: 'enabled', ... }` for endpoints whose capabilities report
`thinkingModes: 'off'`. The bridge silently dropped the field, but the SDK
could error expecting matching reasoning blocks back. Now pr... (continued)

Coverage Stats

8259 of 11724 branches covered (70.45%)

Branch coverage included in aggregate %.

1079 of 1121 new or added lines in 13 files covered. (96.25%)

164 existing lines in 2 files now uncovered.

66505 of 78123 relevant lines covered (85.13%)

300.86 hits per line