agentjido / req_llm / 20cb655c7e63ec10c089c22690d0b034dda10c47-PR-193

49%

main: 49%

Build Type

Pull #193

github

Committed by neilberkman

Commit Message

feat: Add Google Context Caching support for Gemini models

Adds explicit context caching API for Gemini models to reduce costs by up to
90% when reusing large amounts of content.

## Changes

- Add ReqLLM.Providers.Google.CachedContent module for cache CRUD operations
- Support for both Google AI Studio and Vertex AI (when Gemini support is added)
- Add cached_content provider option to reference existing caches
- Comprehensive tests for cache creation, listing, updating, and deletion
- Documentation and examples in Google provider moduledoc
- Updated CHANGELOG

## Usage

```elixir
# Create a cache
{:ok, cache} = ReqLLM.Providers.Google.CachedContent.create(
  provider: :google,
  model: "gemini-2.5-flash",
  api_key: System.get_env("GOOGLE_API_KEY"),
  contents: [%{role: "user", parts: [%{text: large_document}]}],
  ttl: "3600s"
)

# Use the cache in requests
{:ok, response} = ReqLLM.generate_text(
  "google:gemini-2.5-flash",
  "Question about the document?",
  provider_options: [cached_content: cache.name]
)

# Check cached tokens in usage
IO.inspect(response.usage.cached_tokens)
```

## Requirements

- Gemini 2.5 Flash: 1,024 minimum tokens
- Gemini 2.5 Pro: 4,096 minimum tokens

Pull Request Pull Request #193: feat: Add Google Context Caching support for Gemini models

Run Details

11 of 165 new or added lines in 2 files covered. (6.67%)

2 existing lines in 1 file now uncovered.

4003 of 7729 relevant lines covered (51.79%)

345.99 hits per line