Skip to main content

Run batch jobs at the batch price

A batch is a group of chat requests that does not need an answer in seconds. The vendor answers it within 24 hours, at its batch price. OpenAI and Anthropic bill a batch at 50 % of the standard price. OpenRouter reports the cost of each batch, and Proxium bills that cost.

Use a batch for offline work: evaluations, classification, enrichment and extraction. Do not use a batch for an agent loop. Each turn of a loop needs the answer of the turn before, and a batch answers all its lines at the end.

Which vendors​

VendorHow Proxium sends the batchPrice
OpenRouterThe lines inline, to /batchesThe cost that OpenRouter reports
OpenAIA JSONL file to /files, then /batches50 % of the standard price
AnthropicThe lines as Messages requests, to /messages/batches50 % of the standard price
Another vendor of the OpenAI shapeAs OpenAIThe standard price

The batch uses your project's own key for the vendor when the project has one. Otherwise it uses the platform key, and the project needs access to the platform vendors. An endpoint of your project serves only your project.

Send a batch​

  1. Name the model as provider/model, for example openai/gpt-4o-mini.
  2. Give each line a custom_id that is unique in the batch, and a chat-completion body.
  3. Send the batch to POST /v1/batches with your virtual key.
curl https://api.proxium.tech/v1/batches \
-H "Authorization: Bearer $PROXIUM_KEY" \
-H "Content-Type: application/json" \
-d '{
"endpoint": "/v1/chat/completions",
"model": "openai/gpt-4o-mini",
"requests": [
{"custom_id": "q1", "body": {"messages": [{"role": "user", "content": "Classify: refund"}]}},
{"custom_id": "q2", "body": {"messages": [{"role": "user", "content": "Classify: invoice"}]}}
]
}'

Proxium answers 202 with the batch: its id, its status and its request_counts. Every line uses the model of the batch. Proxium removes stream from a line, because a batch line is answered whole.

A batch holds at most 50,000 lines and 100 MB. For Anthropic, a custom_id holds 1 to 64 letters, digits, _ and -.

Read the answers​

Read the batch with GET /v1/batches/{id}. Proxium asks the vendor for its state at each read, and also once a minute on its own.

When the vendor ends the batch, the status is completed, expired or cancelled, and results holds one line for each request:

{
"custom_id": "q1",
"response": {
"status_code": 200,
"body": {
"object": "chat.completion",
"choices": [{"index": 0, "message": {"role": "assistant", "content": "Paris"}, "finish_reason": "stop"}],
"usage": {"prompt_tokens": 12, "completion_tokens": 8, "total_tokens": 20}
}
},
"error": null
}

The results of every vendor have this shape, the shape of an OpenAI batch output line. A line that failed has "response": null and an error, or a response with an error status. The lines are not in the order that you sent them, so match them by custom_id.

Statuses​

StatusMeaning
validatingThe vendor checks the batch
in_progressThe vendor runs the batch
cancellingYou cancelled the batch, and the vendor has not stopped it yet
finalizingProxium writes the usage of the batch
completedThe vendor ran every line
expiredThe vendor did not finish in 24 hours. The lines that finished have results
cancelledThe vendor stopped the batch. The lines that finished have results
failedThe vendor did not take the batch

Cancel a batch​

Send POST /v1/batches/{id}/cancel. The status changes to cancelling, and then to cancelled when the vendor stops. The lines that finished before the stop have results, and Proxium bills them.

OpenRouter has no cancel for a batch. A cancel of an OpenRouter batch answers 409 batch_not_cancellable, and the batch runs to the end. A batch that ended also answers 409.

List your batches​

Send GET /v1/batches. The list holds the batches of your project, newest first, with no results. Use limit (1 to 100, default 20) and after (the last_id of the page before) to read the next page. has_more is true when another page exists.

What it costs​

Proxium writes one usage row for each line that answered, with the cost basis batch and the source batches. Each line is billed once.

Before Proxium sends a batch, it estimates the cost of each line. It divides the characters of the line by four, and counts that as input tokens. It counts max_tokens (4096 when not set) as output tokens, at the batch price. If the estimate is more than a cap of the project or the app, Proxium refuses the batch with 429 and does not send it.

Errors​

StatusCodeCause
400bad_batchThe endpoint is not /v1/chat/completions, the model is not provider/model, or a line has no custom_id or no body
400batch_not_availableThe provider is not an endpoint that your project can use, or it has no batch API
403platform_not_grantedThe batch needs the platform key, and your project has no access to the platform vendors
403model_not_allowedYour key may not call this model
409batch_not_cancellableThe batch ended, or the vendor has no cancel
429project_cappedThe estimate of the batch is more than a cap
502upstream_failedThe vendor refused the batch or did not answer