Run batch jobs at the batch price
A batch is a group of chat requests that does not need an answer in seconds. The vendor answers it within 24 hours, at its batch price. OpenAI and Anthropic bill a batch at 50 % of the standard price. OpenRouter reports the cost of each batch, and Proxium bills that cost.
Use a batch for offline work: evaluations, classification, enrichment and extraction. Do not use a batch for an agent loop. Each turn of a loop needs the answer of the turn before, and a batch answers all its lines at the end.
Which vendors
| Vendor | How Proxium sends the batch | Price |
|---|---|---|
| OpenRouter | The lines inline, to /batches | The cost that OpenRouter reports |
| OpenAI | A JSONL file to /files, then /batches | 50 % of the standard price |
| Anthropic | The lines as Messages requests, to /messages/batches | 50 % of the standard price |
| Another vendor of the OpenAI shape | As OpenAI | The standard price |
The batch uses your project's own key for the vendor when the project has one. Otherwise it uses the platform key, and the project needs access to the platform vendors. An endpoint of your project serves only your project.
Send a batch
- Name the model as
provider/model, for exampleopenai/gpt-4o-mini. - Give each line a
custom_idthat is unique in the batch, and a chat-completionbody. - Send the batch to
POST /v1/batcheswith your virtual key.
curl https://api.proxium.tech/v1/batches \
-H "Authorization: Bearer $PROXIUM_KEY" \
-H "Content-Type: application/json" \
-d '{
"endpoint": "/v1/chat/completions",
"model": "openai/gpt-4o-mini",
"requests": [
{"custom_id": "q1", "body": {"messages": [{"role": "user", "content": "Classify: refund"}]}},
{"custom_id": "q2", "body": {"messages": [{"role": "user", "content": "Classify: invoice"}]}}
]
}'
Proxium answers 202 with the batch: its id, its status and its request_counts.
Every line uses the model of the batch. Proxium removes stream from a line, because a
batch line is answered whole.
A batch holds at most 50,000 lines and 100 MB. For Anthropic, a custom_id holds 1 to 64
letters, digits, _ and -.
Read the answers
Read the batch with GET /v1/batches/{id}. Proxium asks the vendor for its state at each
read, and also once a minute on its own.
When the vendor ends the batch, the status is completed, expired or cancelled, and
results holds one line for each request:
{
"custom_id": "q1",
"response": {
"status_code": 200,
"body": {
"object": "chat.completion",
"choices": [{"index": 0, "message": {"role": "assistant", "content": "Paris"}, "finish_reason": "stop"}],
"usage": {"prompt_tokens": 12, "completion_tokens": 8, "total_tokens": 20}
}
},
"error": null
}
The results of every vendor have this shape, the shape of an OpenAI batch output line. A
line that failed has "response": null and an error, or a response with an error
status. The lines are not in the order that you sent them, so match them by custom_id.
Statuses
| Status | Meaning |
|---|---|
validating | The vendor checks the batch |
in_progress | The vendor runs the batch |
cancelling | You cancelled the batch, and the vendor has not stopped it yet |
finalizing | Proxium writes the usage of the batch |
completed | The vendor ran every line |
expired | The vendor did not finish in 24 hours. The lines that finished have results |
cancelled | The vendor stopped the batch. The lines that finished have results |
failed | The vendor did not take the batch |
Cancel a batch
Send POST /v1/batches/{id}/cancel. The status changes to cancelling, and then to
cancelled when the vendor stops. The lines that finished before the stop have results,
and Proxium bills them.
OpenRouter has no cancel for a batch. A cancel of an OpenRouter batch answers 409 batch_not_cancellable, and the batch runs to the end. A batch that ended also answers
409.
List your batches
Send GET /v1/batches. The list holds the batches of your project, newest first, with no
results. Use limit (1 to 100, default 20) and after (the last_id of the page before)
to read the next page. has_more is true when another page exists.
What it costs
Proxium writes one usage row for each line that answered, with the cost basis batch and
the source batches. Each line is billed once.
Before Proxium sends a batch, it estimates the cost of each line. It divides the characters of the line by four, and counts that as input tokens. It counts max_tokens (4096 when not set) as output tokens, at the batch price. If the estimate is more than a cap of the project or the app, Proxium
refuses the batch with 429 and does not send it.
Errors
| Status | Code | Cause |
|---|---|---|
| 400 | bad_batch | The endpoint is not /v1/chat/completions, the model is not provider/model, or a line has no custom_id or no body |
| 400 | batch_not_available | The provider is not an endpoint that your project can use, or it has no batch API |
| 403 | platform_not_granted | The batch needs the platform key, and your project has no access to the platform vendors |
| 403 | model_not_allowed | Your key may not call this model |
| 409 | batch_not_cancellable | The batch ended, or the vendor has no cancel |
| 429 | project_capped | The estimate of the batch is more than a cap |
| 502 | upstream_failed | The vendor refused the batch or did not answer |