agentjido / req_llm / 7e8adff280d80a19edc59717c92307cf9f762260

49%

push

github

feat: Add Google Context Caching support for Gemini models (#193)

Adds explicit context caching API for Gemini models to reduce costs by up to
90% when reusing large amounts of content.

- Add ReqLLM.Providers.Google.CachedContent module for cache CRUD operations
- Support for both Google AI Studio and Vertex AI (when Gemini support is added)
- Add cached_content provider option to reference existing caches
- Comprehensive tests for cache creation, listing, updating, and deletion
- Documentation and examples in Google provider moduledoc
- Updated CHANGELOG

```elixir
{:ok, cache} = ReqLLM.Providers.Google.CachedContent.create(
  provider: :google,
  model: "gemini-2.5-flash",
  api_key: System.get_env("GOOGLE_API_KEY"),
  contents: [%{role: "user", parts: [%{text: large_document}]}],
  ttl: "3600s"
)

{:ok, response} = ReqLLM.generate_text(
  "google:gemini-2.5-flash",
  "Question about the document?",
  provider_options: [cached_content: cache.name]
)

IO.inspect(response.usage.cached_tokens)
```

- Gemini 2.5 Flash: 1,024 minimum tokens
- Gemini 2.5 Pro: 4,096 minimum tokens

6 of 159 new or added lines in 2 files covered. (3.77%)

3597 of 7343 relevant lines covered (48.99%)

56.68 hits per line