The response cache

The response cache answers a repeated request without a call to a provider. It is on for every project on proxium.tech. This page explains how proxium finds a cached answer, what the answer costs, and where the cache does not apply.

Exact match only

The key of a cached answer is a SHA-256 hash of the project, the model, the endpoint and the request body. Only a body that is the same, field for field, gets a hit. proxium.tech does not match a similar request to a cached answer, and it makes no embedding for the cache.

Each cached answer expires 3600 seconds after proxium stores it. After that time, the same request goes to a provider again.

If the cache fails, proxium continues as on a miss. A cache error never stops a call.

What is in the key

The model in the key is the first model of the route that proxium resolved. If you change the routing of a tier, the new model gets new cache entries.

proxium removes three fields from the body before it makes the key: stream, stream_options and user. All other fields stay in the key, temperature and tools included.

Because the key has no user field, two end users of one project who send the same body get the same answer. If an answer must depend on the end user, put something of that end user in the messages.

A call that sends x-proxium-memory: recall and gets memories has those memories in its body, so the key includes them. Use project memory explains recall.

One project never reads another project's cache

The project is the first part of the key. A request of one project cannot match an entry of another project.

Which calls the cache serves

The cache serves two routes: /v1/chat/completions and /v1/messages. A Messages request goes through the same chat path, so it uses the same cache.

Route	Response cache
/v1/chat/completions	Yes
/v1/messages	Yes
/v1/responses	No
/v1/embeddings, /v1/moderations, /v1/rerank	No
/v1/images/generations, /v1/audio/speech	No
/video/generations, /video/operations, /video/download	No

The routes without a cache send every request to a provider.

Streaming

stream is not in the key. A streamed request and a buffered request with the same body share one entry.

• If a streamed request gets a hit, proxium sends the cached answer as server-sent events. A Messages client gets the Anthropic event sequence.
• If a streamed request misses, proxium stores the answer when the stream is complete. proxium never stores a stream that stopped before the end.

The answer in the cache always has the shape of a complete, non-streamed answer.

Where in the request the lookup happens

proxium looks up the cache before it reserves the budget. Two results follow from that order:

• A hit does not call a provider. By default it costs $0.00 and does not count against the call limits.
• When a hit is free, a project at its budget ceiling still gets hits.

What a hit costs

By default a cached answer is free. A project can change that, for example when it resells answers at a fixed price per request.

Set the cost in the console: open Settings and go to What a cached answer costs. Select one of Free, Quarter, Half or Full price. The value is a fraction of what the live call would have cost.

Under Per sender or key, you can set a different charge for one x-proxium-source or one virtual key. A sender rule wins over a key rule, and both win over the project setting.

When the charge is above zero, a hit is also a request. It counts as one call against the call limits of the project and the key. proxium refuses it when the project is at its ceiling.

See what the cache saved

• Overview shows Saved by cache: the money the project did not spend in the window.
• Requests shows Cache saved and a Cache panel with the hits over time.

The answer itself does not say that it came from the cache. Use the Cache panel on Requests to count hits.

Turning the cache off

No request header turns the cache off for one call. A project cannot turn the cache off.

A request gets a fresh answer from a provider when one of these is true:

• Its body differs from every cached body in a field other than stream, stream_options and user.
• No cached answer for it is younger than 3600 seconds.
• It goes to a route without a cache, such as /v1/responses.

When a project is erased

The cache cannot list the entries of one project, because each key is a hash. The entries of an erased project expire after 3600 seconds. The erasure deletes every virtual key of the project first, so nobody can read them in that time.