Skip to main content

The response cache

The response cache answers a repeated request without a call to a provider. It is on for every project on proxium.tech. This page explains how Proxium finds a cached answer, what the answer costs, and where the cache does not apply.

The same question, in other words​

Proxium looks at the last question of a request and at the conversation around it. When the conversation is the same and the question asks the same thing, a cached answer can serve it, even when the words differ. "How long does a refund take?" and "How many days until I get my refund?" can share an answer. "Refund for a card" and "refund for a gift card" do not.

Proxium makes no embedding for this. It compares the text first. When the words differ, Jev, the decision model of TypeSafe, decides. Jev answers for a project that has it included, or that added its own TypeSafe key. For other projects, a question in the same words still gets the cached answer. When your app asks for model: auto, the same Jev call also picks the tier.

Some requests use only the exact match below, for example a request whose last turn is a tool result or holds an image.

Exact match​

The key of a cached answer is a SHA-256 hash of these four things:

  • the project;
  • the first model of the resolved tier;
  • the endpoint;
  • the request body, with the model field that your app sent.

A cached answer serves that request, whichever model of the plan made it. A model or vendor that your project retires keeps its cached answers until they expire. Only a body that is the same, field for field, gets a hit. An answer with no content and no tool call is not stored, so a repeat reaches the model again.

Each cached answer expires after a while, or after the time that the project sets in Settings › Cache. After that time, the same request goes to a provider again.

If the cache fails, Proxium continues as on a miss. A cache error never stops a call.

Fig. 1 · What the cache does with a request
A chat or Messages request arrives
  1. 1Does an earlier question in the same conversation ask the same thing?In the same or other words
    yes
    The stored answer. No vendor call
    no
  2. 2Is the same request in the cache, and younger than the cache time?Same project, model, route and body, except stream, stream_options and user
    yes
    The stored answer. No vendor call
    no
Proxium calls the models, then stores the answer

What is in the key​

The model in the key is the first model of the route that Proxium resolved. If you change the routing of a tier, the new model gets new cache entries.

Proxium removes three fields from the body before it makes the key: stream, stream_options and user. All other fields stay in the key, temperature and tools included.

Because the key has no user field, two end users of one project who send the same body get the same answer. If an answer must depend on the end user, put something of that end user in the messages.

A call that sends x-proxium-memory: recall and gets memories has those memories in its body, so the key includes them. Use project memory explains recall.

One project never reads another project's cache​

The project is the first part of the key. A request of one project cannot match an entry of another project.

Which calls the cache serves​

The cache serves two routes: /v1/chat/completions and /v1/messages. A Messages request goes through the same chat path, so it uses the same cache.

RouteResponse cache
/v1/chat/completionsYes
/v1/messagesYes
/v1/responsesNo
/v1/completions, /v1/embeddings, /v1/moderations, /v1/rerankNo
/v1/images/generations, /v1/audio/speech, /v1/audio/transcriptions, /v1/audio/translationsNo
/video/generations, /video/operations, /video/downloadNo

The routes without a cache send every request to a provider.

Streaming​

stream is not in the key. A streamed request and a buffered request with the same body share one entry.

  • If a streamed request gets a hit, Proxium sends the cached answer as server-sent events. A Messages client gets the Anthropic event sequence.
  • If a streamed request misses, Proxium stores the answer when the stream is complete. Proxium never stores a stream that stopped before the end.

The answer in the cache always has the shape of a complete, non-streamed answer.

Where in the request the lookup happens​

Proxium looks up the cache before it reserves the budget. Two results follow from that order:

  • A hit does not call a provider. By default it is free and does not count against the call limits.
  • When a hit is free, a project at its budget ceiling still gets hits.

What a hit costs​

By default a cached answer is free.

The operator of Proxium can set a price for the hits of a project. The price is Free, Quarter, Half or Full price of the live call. The operator can also set a different price for one x-proxium-source or one virtual key. A sender price wins over a key price, and both win over the project price.

A member of the project cannot see or change this price. On proxium.tech, each project is on Free unless the operator set another price.

When the charge is above zero, a hit is also a request. It counts as one call against the call limits of the project and the key. Proxium refuses it when the project is at its ceiling.

The vendor cache is a second cache​

The response cache above is the cache of Proxium. A vendor has its own prompt cache. It holds the start of a long conversation, so the vendor reads that part at a lower price. An agent that sends a long history on each step gets most of its input from this cache.

Proxium does not control the vendor cache. It reads the cached tokens that the vendor reports, and shows them:

  • Hit rate: the share of the input that the vendor read from its cache.
  • Saved: the cached tokens at the price of the input, minus the price of a cache read, for each model.
  • Calls that lost the cache: the calls that read less than half of their input from the cache. A change of vendor in the middle of a conversation can cause this.

Which cache fits agent traffic​

The response cache answers a request whose body is the same as an earlier body, field for field. An agent adds its history to every step, so two of its bodies are never the same. The cache therefore serves the opposite case: one fixed prompt sent again, a retried call, and a check that runs on a schedule.

7 days to 2026-11-12, over all projects:

MeasureValue
Hits66
Money saved1 millicent, that is 0.00001 USD
Hits that saved 065
Projects with a hit5 of the 13 that made a call
Layer normalized35 hits, 9 601 tokens, 0.00001 USD
Layer exact31 hits, 82 991 tokens, 0 USD
Best project, 30 days302 hits, 28 955 millicents

30 days gives 490 hits, 9 projects and 0.36048 USD. One project whose calls repeat holds almost all of that saving. A hit that saves 0 is not a defect. 30 of the 66 hits are on a model whose price is 0, so a saved call saves nothing.

Two rules of the cache hold an agent answer out of it:

  • An answer with a tool call is not stored, because a tool call carries arguments taken from the exact words of the question.
  • An answer with no content is not stored.

So a bigger response cache does not serve an agent. The levers for agent traffic are the vendor prompt cache below and plan reuse. The cache for the result of a tool call is a separate feature. It keeps an answer by the name and the arguments of a tool. Proxium sees the model call, and it does not see the tool call.

A model with no price​

A model with no price row hides the cost of its calls and the saving of its cache hits. These three models had no row, over 7 days to 2026-11-12:

ModelProviderCallsInput tokens
clef:27bollama2 1367 820 600
Meta-Llama-3_3-70B-Instructovh-ai149541 012
Mistral-Small-3.2-24B-Instruct-2506ovh-ai1614 171

Each of the three now has a price row. The two OVH models carry the OpenRouter price of the same model, which is the rule of the routing presets. clef:27b has no list price, because Ollama Cloud sells it by subscription only, so its row is a price of 0 from a person.

The same audit on 2026-11-19 found two more models that a route reaches: beam on ollama, and the embedding models of ollama-embed. beam is an Ollama Cloud model of the free tier, so a token of it has no marginal price, like clef:27b. The models of ollama-embed run from local weights, so a token of them also has no marginal price. Both now have a row of 0 that a person set.

A model that is free on purpose has a row of 0 that a person set. That row is a price of 0, not a missing price. Two such models are reflection/beam:free and stealth/glyph-cluster:free. Their hits save 0 USD because their calls cost 0 USD. A stub row of 0 with the source refresh is the opposite case. The gateway does not load it, so the model reads as a model with no price.

A fixed prefix for the memory jobs​

The vendor cache reads the start of a request, so it needs that start to be identical between two calls. Each memory job of a project therefore sends one constant instruction as the system message, and the text of the work as the user message. The jobs with a constant prefix are extract, merge, graph, pair, page and consolidate. The classifier does the same.

Three rules keep a job cacheable:

  1. Write the instruction as a constant, and put no variable text in it.
  2. Format the work of the call into a separate string, and send it as the user message.
  3. Send the name of the job, so that each job has its own session key.

A job shares no instruction text with another job. With one key for all jobs, a vendor reads the prefix of one job as a change of prefix for the other. Then no job gets a cache read. So the session key of a memory job is memory:<job>:<tier>. The key of the classifier is classifier or classifier:<model>.

An embedding has no prompt, so it has no prompt cache.

See what the cache saved​

  • Overview shows Answer cache: the money the project did not spend in the window. Beside it, Vendor cache shows the hit rate of the vendor cache, what it saved, and the calls that lost it.
  • Requests names the cached share of each call under the row, beside its tokens.
  • Usage shows a Cache panel: the hits over time, by layer and by model, with what they saved.

The answer itself does not say that it came from the cache. Use the Cache panel on Usage to count hits.

Change the cache of a project​

The cache is on for every project. In Settings › Cache, a project can:

  • turn the cache off;
  • keep an answer for a shorter or a longer time;
  • turn off the match of a question in other words, where the project has it.

The change applies to the next request. No request header turns the cache off for one call.

A request gets a fresh answer from a provider when one of these is true:

  • The cache of the project is off.
  • No cached request asks the same thing in the same conversation.
  • The cached answer is older than the time the project keeps answers.
  • It goes to a route without a cache, such as /v1/responses.

When a project is erased​

The cache cannot list the entries of one project, because each key is a hash. The entries of an erased project expire after a while. The erasure deletes every virtual key of the project first, so nobody can read them in that time.