Build a voice agent with realtime sessions

A voice agent sends speech and gets speech back over one WebSocket, with low latency. Proxium serves the OpenAI realtime API at wss://proxium.tech/v1/realtime. It relays the events of the session to the vendor and back with no change. Your app keeps its code. It changes the base URL and the key.

Proxium adds the same checks as on every call. Your vendor key stays on the server. The budget of the key applies. Proxium prices and records each response of the session.

Before you start

Your project needs a vendor of the request type Voice (realtime) that speaks the OpenAI realtime protocol.

1. On Providers › Vendors, add OpenAI with your own OpenAI key.
2. In Request type, select Voice (realtime).
3. In the models, list the realtime models that you use.

Proxium opens the vendor socket at the base URL of the vendor, followed by /realtime. See Add vendors and your own keys.

Open a session with the OpenAI SDK

voice.py

from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="https://proxium.tech/v1", api_key=PROXIUM_KEY)

async with client.realtime.connect(model="t.acme.my-voice/realtime-model") as conn:
    await conn.session.update(session={"type": "realtime", "instructions": "Be brief."})
    await conn.response.create()
    async for event in conn:
        if event.type == "response.done":
            break

The SDK opens wss://proxium.tech/v1/realtime?model=… and sends your Proxium key in Authorization. In model, send a model id of your vendor. A tier name also works, if Routing has a chain for Voice (realtime).

A browser cannot set a header on a WebSocket. It sends the key as a subprotocol, next to realtime:

browser.js

const ws = new WebSocket(
  "wss://proxium.tech/v1/realtime?model=t.acme.my-voice/realtime-model",
  ["realtime", "openai-insecure-api-key." + PROXIUM_KEY],
);

warning

Do not put a key with a large budget in a browser. A person who opens the page can read the key. Give the browser app its own key with a low budget, or open the session from your server.

What Proxium does in a session

When	What Proxium does
Before the session opens	Checks the key and the budget, and the number of open sessions of the key and of the project. Then it opens the vendor socket. If a vendor of the chain fails to open, Proxium tries the next one. A refusal is an HTTP error with a status and a code.
At each response	Every response takes one call of the budget, also a response that the vendor starts by itself when the user stops speaking.
When a response ends	Proxium records one usage row, priced by kind of token: text, audio and image, each with its own rate. The transcription of the user's speech gets its own row, at the price of the transcription model.
When the budget refuses a response	Proxium counts the response before it reaches your app, so no event of a refused response reaches your app. Proxium cancels the response, sends an error event with the code of the refusal, and closes the session.
On a text event of your app	The data protection of the project scans every JSON event that your app sends. A blocked event does not reach the vendor, and your app gets an error event with guardrail_blocked. Only the audio of an input_audio_buffer.append event and of an input_audio content part is left out of the scan. Any other audio field is scanned as text.
On a binary frame of your app	The vendor takes JSON events as text frames only. A binary frame is refused with an error event unsupported_frame, and it does not reach the vendor.
On a text frame that is not one JSON object	The frame is refused with an error event invalid_event, and it does not reach the vendor.
When your app leaves during a response	Proxium cancels the response and waits for the vendor to end it, so its usage is exact.
When the vendor socket breaks	Proxium records the response from the output that passed, and marks the row as estimated.

Proxium stores no audio. It stores the transcript of the session when your project stores prompts and answers.

Errors before the session opens

Status	Code	Cause
400	missing_model	The URL has no model.
400	no_route	No realtime vendor of the project serves the model, or the key may not use it.
401	missing_key, invalid_key	The key is missing or not valid.
426	upgrade_required	The request is not a WebSocket upgrade.
429	tenant_capped, too_many_sessions	The budget refused the session, or the key or the project has its limit of open sessions.
503	upstream_failed	No vendor of the chain opened the session.
503	shutting_down	The server is stopping. Open the session again.

How a session ends

Close code	Cause
1000	Your app closed the session, or the session reached its longest length.
1008	The budget refused a response. The error event before it has the code.
1011	The vendor closed the session or its socket broke.
1012	The server restarts.

Before a restart, your app gets an error event with the code session_closing. The session goes on for a short time, so a reply can end. When you get session_closing or close code 1012, open a new session and send the conversation again.

Related pages

• Add vendors and your own keys: add the vendor and its key.
• Set budgets and limits: the budget that each response takes from.
• Protect sensitive data: the classes that scan the text events.
• What Proxium does not do: the gaps of realtime sessions.