Limits
Proxium has limits on size, time, retries, spend and names. This page says what each limit does and what happens when a call reaches it.
Size
| Limit | At the limit |
|---|---|
| Request body | 413, plain-text body |
| Vendor answer | The call fails. Proxium tries the next model |
| Stored copy of an answer | Proxium cuts the stored copy. Your app gets the full answer |
| Model id of a media call | 400 bad_model |
A media model id can hold only letters, digits, ., _, - and :.
Time
| Limit | At the limit |
|---|---|
x-proxium-timeout-ms | A value above the maximum is lowered to the maximum |
| All calls of one request, with the header | 503 upstream_failed |
| All calls of one request, without the header | No limit |
| One call, not streamed | Timeout. Proxium tries the next model |
| Connect to a vendor | The call fails. Proxium tries the next model |
| Silence in a stream | Proxium ends the stream |
- Proxium ignores an
x-proxium-timeout-msthat is not a positive number. - When the header time ends, the message of the
503starts withclient deadline exceeded. - For a stream, the header time ends at the first piece. After that, only the silence limit applies. A stream has no limit on its total time.
- The silence limit also runs while the vendor prepares its first piece.
The response cache
A short cache is available, so a loop of the same request is cached. Each project can turn its cache off, or keep answers for a shorter or a longer time, in Settings › Cache. See The response cache.
Retries and failover
- A plan holds a few models. When all of them fail, the call gets
503 upstream_failed. - After a server error, a timeout or a "warming" answer, Proxium can call the same model again, after a short wait. Then it tries the next model.
- After a
429, Proxium skips that vendor for the rest of the request. The next call goes to a model at another vendor of the plan. - Proxium honors a vendor's
Retry-Afterup to a limit. - When a vendor keeps failing, Proxium pauses it, then sends a trial call. Good trial answers bring the vendor back.
- When a model keeps failing, Proxium retires it for your project for a while.
- Proxium retires a vendor in the same way when all its models keep failing. It also retires a vendor that says your own key has no credits or is rejected.
- A retired model or vendor comes back by itself.
- A vendor with an allowance is retired when its calls or tokens of the period reach it, until the next reset. When every hop has used its allowance, your app gets
429withretry-after. See Use a vendor's free tier to its end. - A
429never counts toward a retirement. It pauses the vendor for a short time: the vendor'sRetry-After, or 1 s when it sends none. Calls of your project skip the vendor until then. When every vendor of the plan is paused, your app gets429 rate_limitedwithretry-after. - Your project sets the retries, the waits and the retirement rules in its failover policy, on Routing › When a call fails. See Set your failover policy.
- Proxium does not skip a fallback model for its price. A fallback can cost more than your first model.
Spend and rate
| Limit | At the limit |
|---|---|
| App: Calls per hour | 429 source_capped |
| App: Dollars per day | 429 source_capped |
| Key: calls per hour | 429 tenant_capped |
| Key: requests and tokens per minute | 429 tenant_capped |
| Key: dollars per day | 429 tenant_capped |
- A limit that you do not set does not apply.
- Each
429carries aretry-afterheader. - Each limit counts a rolling window: the last hour, the last day or the last minute.
- You set the app limits on Overview › Per-app burn, and the key limits on Keys. See Set budgets and limits.
The loop
A loop is an agent that turns more than once. A project turns the loop on for itself, for one app or for one key prefix. Set it in Settings › Loop, on the Loops screen, or with proxium_set_loop. A scope that nobody set has the loop on. The gateway counts every conversation and refuses nothing while every ceiling of that project is 0. A project that does not want the loop writes one row with enabled = false.
| Setting | Meaning |
|---|---|
| Loop | On or off for this scope. The loop is on when no scope sets it. A scope with no value inherits the scope above it |
| Most iterations | The most calls of one conversation. 0 is no ceiling |
| Most cost | The most money of one conversation. 0 is no ceiling |
| Most stalls | The most repeated tool calls of one conversation. 0 is no ceiling |
| Over a ceiling | refuse answers 429 loop_capped; alert lets the call pass |
| Verifier | other_vendor asks a model of another family than the model of the worker, and any_vendor allows one family. A scope with no value inherits the scope above it |
- The gateway counts every conversation of a scope whose loop is on. The
x-proxium-conversationheader names a conversation, or groups the calls of several; without it the gateway counts by the session anchor of the call. - A caller may turn the loop off for one conversation with the header value
off,0,noorfalse. A caller never turns the loop on. - A switch that a person set for one conversation in Settings › Loop wins over the header of the caller.
- A
429 loop_cappedcarries aretry-afterheader.
The id of a conversation
The gateway puts every call in one conversation, and it counts the calls of that conversation.
-
It reads the header
x-proxium-conversation. The aliasx-axon-conversationworks the same way. -
If the header is absent or blank, it uses the session anchor of the call. The
identityof the classification document builds the anchor. By default the anchor is the first ofx-proxium-conversationandx-proxium-task-id, with the key, the app and the repository (headerx-proxium-repo). Without either header it is the key, the app, the repository and the first system and user messages.The routing uses the same anchor. It keeps the vendor of a conversation and the model that the classifier chose for it. Two sessions that send the same first message get two anchors when they send an id, a repository, or use two keys or two apps.
-
It cuts the id to 200 characters and replaces each control character with a space. The id becomes a field of a Redis key, so it holds no control character.
-
If neither the header nor the anchor gives an id, the conversation is
unattributed. Every call with no id shares that one conversation, and it is a loop only when it repeats.
Where you read the loop
| What | Where |
|---|---|
| The settings of the project, one app or one key prefix | GET and PUT /api/teams/{slug}/loop, proxium_read_loop, proxium_set_loop |
| The conversations that count now, and the runs that finished | GET /api/teams/{slug}/memory/loops, proxium_read_loops |
| The stop condition of a conversation, and its goal state | GET and PUT /api/teams/{slug}/memory/goal, proxium_read_goal_state, proxium_set_stop_condition |
| The changes of a goal that wait for a person, and their approval | GET /api/teams/{slug}/memory/goals/waiting, POST /api/teams/{slug}/memory/goal/approve, proxium_read_waiting_goals, proxium_approve_goal |
| The score of each verifier model, for each day | GET /api/teams/{slug}/memory/verifier, proxium_read_verifier_score |
| The same two lists, in the console | Loops, see Console screens |
The job verifier_replay runs every day. It asks the verifier again on the settled items of each project. Then it writes the score of each model for the day: the items it judged, the judgments that the truth agreed with, and the rate.
A conversation that stops calls for ten minutes becomes a run. The run keeps its calls, its cost, its stalls and its outcome, and the gateway reports it once.
Names and lists
| Limit | At the limit |
|---|---|
| Key name, on Keys | A name that is too long is dropped. The key has no name |
| Endpoint name | 400 bad_name |
| Vendor name | 400 bad_name |
| Models of one vendor | 400 too_many_models |
A vendor name holds only lower-case letters, digits and -, with no - at the start or the end.
How soon a change applies
| Change | Applies |
|---|---|
| Routing and app limits | At once on one server, and soon after on all |
| The model list of each vendor | Proxium reads it again from time to time |