Skip to main content

Limits

Proxium has limits on size, time, retries, spend and names. This page says what each limit does and what happens when a call reaches it.

Size​

LimitAt the limit
Request body413, plain-text body
Vendor answerThe call fails. Proxium tries the next model
Stored copy of an answerProxium cuts the stored copy. Your app gets the full answer
Model id of a media call400 bad_model

A media model id can hold only letters, digits, ., _, - and :.

Time​

LimitAt the limit
x-proxium-timeout-msA value above the maximum is lowered to the maximum
All calls of one request, with the header503 upstream_failed
All calls of one request, without the headerNo limit
One call, not streamedTimeout. Proxium tries the next model
Connect to a vendorThe call fails. Proxium tries the next model
Silence in a streamProxium ends the stream
  • Proxium ignores an x-proxium-timeout-ms that is not a positive number.
  • When the header time ends, the message of the 503 starts with client deadline exceeded.
  • For a stream, the header time ends at the first piece. After that, only the silence limit applies. A stream has no limit on its total time.
  • The silence limit also runs while the vendor prepares its first piece.

The response cache​

A short cache is available, so a loop of the same request is cached. Each project can turn its cache off, or keep answers for a shorter or a longer time, in Settings › Cache. See The response cache.

Retries and failover​

  • A plan holds a few models. When all of them fail, the call gets 503 upstream_failed.
  • After a server error, a timeout or a "warming" answer, Proxium can call the same model again, after a short wait. Then it tries the next model.
  • After a 429, Proxium skips that vendor for the rest of the request. The next call goes to a model at another vendor of the plan.
  • Proxium honors a vendor's Retry-After up to a limit.
  • When a vendor keeps failing, Proxium pauses it, then sends a trial call. Good trial answers bring the vendor back.
  • When a model keeps failing, Proxium retires it for your project for a while.
  • Proxium retires a vendor in the same way when all its models keep failing. It also retires a vendor that says your own key has no credits or is rejected.
  • A retired model or vendor comes back by itself.
  • A vendor with an allowance is retired when its calls or tokens of the period reach it, until the next reset. When every hop has used its allowance, your app gets 429 with retry-after. See Use a vendor's free tier to its end.
  • A 429 never counts toward a retirement. It pauses the vendor for a short time: the vendor's Retry-After, or 1 s when it sends none. Calls of your project skip the vendor until then. When every vendor of the plan is paused, your app gets 429 rate_limited with retry-after.
  • Your project sets the retries, the waits and the retirement rules in its failover policy, on Routing › When a call fails. See Set your failover policy.
  • Proxium does not skip a fallback model for its price. A fallback can cost more than your first model.

Spend and rate​

LimitAt the limit
App: Calls per hour429 source_capped
App: Dollars per day429 source_capped
Key: calls per hour429 tenant_capped
Key: requests and tokens per minute429 tenant_capped
Key: dollars per day429 tenant_capped
  • A limit that you do not set does not apply.
  • Each 429 carries a retry-after header.
  • Each limit counts a rolling window: the last hour, the last day or the last minute.
  • You set the app limits on Overview › Per-app burn, and the key limits on Keys. See Set budgets and limits.

The loop​

A loop is an agent that turns more than once. A project turns the loop on for itself, for one app or for one key prefix. Set it in Settings › Loop, on the Loops screen, or with proxium_set_loop. A scope that nobody set has the loop on. The gateway counts every conversation and refuses nothing while every ceiling of that project is 0. A project that does not want the loop writes one row with enabled = false.

SettingMeaning
LoopOn or off for this scope. The loop is on when no scope sets it. A scope with no value inherits the scope above it
Most iterationsThe most calls of one conversation. 0 is no ceiling
Most costThe most money of one conversation. 0 is no ceiling
Most stallsThe most repeated tool calls of one conversation. 0 is no ceiling
Over a ceilingrefuse answers 429 loop_capped; alert lets the call pass
Verifierother_vendor asks a model of another family than the model of the worker, and any_vendor allows one family. A scope with no value inherits the scope above it
  • The gateway counts every conversation of a scope whose loop is on. The x-proxium-conversation header names a conversation, or groups the calls of several; without it the gateway counts by the session anchor of the call.
  • A caller may turn the loop off for one conversation with the header value off, 0, no or false. A caller never turns the loop on.
  • A switch that a person set for one conversation in Settings › Loop wins over the header of the caller.
  • A 429 loop_capped carries a retry-after header.

The id of a conversation​

The gateway puts every call in one conversation, and it counts the calls of that conversation.

  1. It reads the header x-proxium-conversation. The alias x-axon-conversation works the same way.

  2. If the header is absent or blank, it uses the session anchor of the call. The identity of the classification document builds the anchor. By default the anchor is the first of x-proxium-conversation and x-proxium-task-id, with the key, the app and the repository (header x-proxium-repo). Without either header it is the key, the app, the repository and the first system and user messages.

    The routing uses the same anchor. It keeps the vendor of a conversation and the model that the classifier chose for it. Two sessions that send the same first message get two anchors when they send an id, a repository, or use two keys or two apps.

  3. It cuts the id to 200 characters and replaces each control character with a space. The id becomes a field of a Redis key, so it holds no control character.

  4. If neither the header nor the anchor gives an id, the conversation is unattributed. Every call with no id shares that one conversation, and it is a loop only when it repeats.

Where you read the loop​

WhatWhere
The settings of the project, one app or one key prefixGET and PUT /api/teams/{slug}/loop, proxium_read_loop, proxium_set_loop
The conversations that count now, and the runs that finishedGET /api/teams/{slug}/memory/loops, proxium_read_loops
The stop condition of a conversation, and its goal stateGET and PUT /api/teams/{slug}/memory/goal, proxium_read_goal_state, proxium_set_stop_condition
The changes of a goal that wait for a person, and their approvalGET /api/teams/{slug}/memory/goals/waiting, POST /api/teams/{slug}/memory/goal/approve, proxium_read_waiting_goals, proxium_approve_goal
The score of each verifier model, for each dayGET /api/teams/{slug}/memory/verifier, proxium_read_verifier_score
The same two lists, in the consoleLoops, see Console screens

The job verifier_replay runs every day. It asks the verifier again on the settled items of each project. Then it writes the score of each model for the day: the items it judged, the judgments that the truth agreed with, and the rate.

A conversation that stops calls for ten minutes becomes a run. The run keeps its calls, its cost, its stalls and its outcome, and the gateway reports it once.

Names and lists​

LimitAt the limit
Key name, on KeysA name that is too long is dropped. The key has no name
Endpoint name400 bad_name
Vendor name400 bad_name
Models of one vendor400 too_many_models

A vendor name holds only lower-case letters, digits and -, with no - at the start or the end.

How soon a change applies​

ChangeApplies
Routing and app limitsAt once on one server, and soon after on all
The model list of each vendorProxium reads it again from time to time