Skip to main content

Use the Responses, embeddings, rerank and moderations APIs

Proxium serves five more OpenAI APIs with the same base URL and virtual key as chat:

RouteWhat it doesRouting typeStream
POST /v1/completionsThe legacy text completion APIChatYes
POST /v1/embeddingsTurns text into vectorsEmbeddingsNo
POST /v1/responsesThe OpenAI Responses APITextYes
POST /v1/rerankOrders documents by how well they match a queryRerankNo
POST /v1/moderationsChecks text against a content policyModerationNo

Proxium sends your request body to the same path at the vendor, for example <base URL of the vendor>/embeddings. It sends the answer of the vendor back without change. Jev answers rerank and moderations calls too. See Rerank and moderations.

Fig. 1 · How a passthrough call goes
Your appPOST /v1/embeddings
Proxiumchecks the key and the limits, picks the model
The vendorPOST <its base URL>/embeddings
Your appgets the answer of the vendor, unchanged
note

Each route goes only to a model of its routing type. A rerank call never goes to a chat model, and a moderations call never goes to a rerank model. When the project has no model of the type, the call gets 400 no_route at once.

Make embeddings​

  1. Add a vendor that serves embeddings, for example the OpenAI (embeddings) preset. See Add vendors and your own keys.
  2. Get the model id from GET /v1/models, for example t.acme.openai-embed/<model>.
  3. Send the id in model:
result = client.embeddings.create(
model="t.acme.openai-embed/<model>",
input=["The first text.", "The second text."],
)
print(len(result.data), len(result.data[0].embedding))

Proxium does not store or search the vectors. It records the tokens and the cost of the call.

To send a tier name in place of the id, set a chain for the tier embed in Routing › Embeddings routing.

Use the Responses API​

Send the call to a vendor that serves the Responses API, such as OpenAI. Set stream to true to get server-sent events.

response = client.responses.create(
model="t.acme.openai/<model>",
input="Write one sentence about queues.",
)
print(response.output_text)

Rerank and moderations​

Jev, the classifier that Proxium routes with, answers both routes. Jev reads the text and scores each question with a probability:

RouteWhat Jev scoresThe answer
/v1/moderationsEach category of the OpenAI moderation API, for each inputThe OpenAI moderation shape: flagged, categories, category_scores
/v1/rerankHow well each document answers the queryThe shape of Cohere and Jina: results with index and relevance_score, best first

A project with the pooled vendors of Proxium can use the Jev of Proxium. A project can also add its own TypeSafe vendor. The tiers moderation and rerank start with Jev, and a moderation with no model uses the tier moderation:

result = client.moderations.create(input="Text to check.")
print(result.results[0].flagged)
Terminal
curl https://proxium.tech/v1/rerank \
-H "Authorization: Bearer $PROXIUM_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "rerank", "query": "refund policy", "documents": ["Refunds take 5 days.", "We ship worldwide."], "top_n": 1}'

Jev reads text only. A moderation with an image part gets 400 bad_request.

A moderation takes at most 256 inputs. A request with more gets 400 too_many_inputs, and Jev makes no call. Each input is one Jev call, and the request takes one call slot for each input. Jev reads at most 24,000 characters of an input. A longer input keeps its start and its end, with a line that holds … between them. Its result carries "input_truncated": true.

You can also use a rerank or moderation model of your own vendor. Add the vendor with the request type Rerank or Moderation. Then send its id in model, or put it in the chain of the tier. Proxium sends your body to the vendor without change.

The answer of a rerank is read in one shape. Cohere and Jina answer with results, and Proxium returns it as it is. Voyage answers with data, the same list under another name. Proxium copies data to results and keeps the rest of the answer. A moderation answer is returned as the vendor sent it.

Each call is priced by the way its vendor bills:

Vendor billsHow Proxium prices the callWhat the model needs
Search units, like Cohere (meta.billed_units.search_units)Units times the price per 1,000 unitsThe rerank price per 1,000 units. Without it the call is recorded as a missing price, not as free
Tokens, like Jina and Voyage (usage.total_tokens)The tokens as input, at the input priceThe input price per million tokens
Nothing, like the OpenAI moderation APINo chargeNothing. The call is recorded at zero
Terminal
curl https://proxium.tech/v1/rerank \
-H "Authorization: Bearer $PROXIUM_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "t.acme.my-reranker/<model>", "query": "refund policy", "documents": ["Refunds take 5 days.", "We ship worldwide."]}'

What is the same as chat, and what is not​

Chat and MessagesThese four routes
Virtual key, ceilings, cost recordYesYes
Routing and failoverYesYes
Row in the log of RequestsYesYes
Response cacheYesNo. Each call goes to a vendor
x-proxium-memory: recallAdds memoriesNo effect. /v1/responses keeps the call in memory as write
A call without modelUses auto: the classifier chooses the tier, or the call uses standard400 missing_model. A moderation uses the tier moderation
A call with model: "auto"The classifier chooses the tier, or the call uses standard/v1/responses uses standard, rerank uses rerank, and moderations uses moderation. They have no classifier