Cache

The Cache service returns a stored response for a repeated request instead of calling a model again — cutting both cost and latency. When a request you have served before comes in again, the platform replays the earlier answer and skips the model call entirely.

You will find Cache under Services > Cache.

Turning Cache on

Use the Enabled switch in the top right of the page. While it is off, nothing is cached and nothing is served from cache. Once on, caching happens automatically — you do not need to change your requests.

What gets cached

Caching is automatic but deterministic-only: a request is cached only when its result is reproducible, which means either temperature: 0 or a fixed seed. Requests with temperature above 0 and no seed are left alone — you asked for variety, so you get a fresh answer every time.

You can change this in two ways:

• Per request: send "cache": false in the chat-completions body to bypass the cache for that call (both reading and writing), regardless of settings.
• Per organisation: turn on Also cache non-deterministic responses in the configuration to cache temperature > 0 requests too. This returns an identical answer for identical inputs even where the model would normally vary — only enable it if that is acceptable for your use case.

Exact and semantic matching

Exact matching is always on when the service is enabled. The platform builds a key from the request as you sent it — the model, the messages, the sampling parameters, and any tools — and serves a stored response when an identical request arrives again.

Semantic matching is optional, toggled on the Configuration tab. On an exact miss, the platform embeds the prompt and looks for a near-duplicate earlier request within a similarity threshold you set. This catches paraphrases an exact key would miss — but it can return an answer written for a similar prompt, so raise the threshold if you need matches to be tight.

How a hit is served

The cache keys on the request as you sent it, before any other service runs. A hit therefore short-circuits the whole pipeline: the stored response is returned immediately, skipping retrieval, memory, the PII firewall, and the model call. That is what makes a hit both cheap and fast.

The trade-off is freshness. If you also use RAG or Memory, a cached answer reflects your knowledge base and stored facts as they were when the answer was first produced, until the entry expires. The time-to-live bounds how stale a cached answer can get.

Time-to-live

Set how long entries live on the Configuration tab under Cache lifetime — 1 hour, 24 hours, 7 days, or 30 days. Expired entries are never served and are pruned automatically. Lower the TTL when freshness matters more than hit rate; raise it for stable, repetitive workloads.

Clearing the cache

The Clear cache action empties your organisation's cache immediately, after a confirmation prompt. Use it when you have changed something upstream — a prompt template, a RAG corpus, a model — and want stored answers regenerated rather than waiting for them to expire.

Metering and records

Cache hits are metered and appear in your usage. A hit records a small cache.hit charge and skips the base inference charge — that skipped charge is your saving. Misses run as normal and are billed for the inference they perform. Configuration changes and cache clears are written to the audit log.

Privacy and isolation

Your cache is strictly scoped to your organisation — one tenant's stored responses are never visible to another. Cached responses are stored at rest and may contain personal data from the original answer, so they are isolated per organisation and bounded by the TTL; clearing the cache or letting entries expire removes them.

Good to know

Cache needs the service enabled. While it is off, requests run the full pipeline every time.

Only reproducible requests are cached by default. temperature: 0 or a seed — otherwise the request bypasses the cache unless you opt into caching volatile responses.

A hit skips everything, including guard and the firewall. The stored answer already passed those checks when it was first produced; new policy changes apply to fresh (uncached) requests or after the cache is cleared.

Semantic matching trades precision for hit rate. Exact matching never returns a wrong answer; semantic matching can return one written for a similar prompt — tune the threshold, or leave semantic off if you need exactness.