Lower your costs
The Improve screen reads the traffic of your project, and lists what to change. Each finding shows the numbers that it comes from.
Read the findings
Open Improve in the console. One sentence above the readouts says what the money is. It gives the dollars a month that the findings with a saving add up to. It also gives the count of the other findings, which fix a failure or show where the cost sits. Most findings fix a failure, so they carry no dollar figure.
The readouts on top count the findings:
| Readout | Shows |
|---|---|
| Findings | All findings |
| Worth doing first | The findings of high severity |
| Worth a look | The other findings |
| Worth fixing | The estimated saving of each month, in dollars, from the findings that name one |
Below the readouts, the findings are in groups. Each group has a count and the screen where you fix it:
| Group | Holds | Fix it on |
|---|---|---|
| Reliability | The vendors that fail many attempts now | Providers |
| Empty answers | The apps whose calls end empty at max_tokens | Routing |
| Wrong model names | The apps that ask for a model name that no vendor serves | Routing |
| Classifier chain | The hops of the classifier that answer almost nothing | Routing |
| Cost | The spend lines that can go down | Routing |
| Cache | The projects with real traffic where the cache saved little | Usage |
| Paused vendors and models | One folded table of the vendors and models that the failover policy paused | Routing |
| Other findings | A kind of finding that this page has no group for yet |
An agent applies a change of traffic with proxium_plan_routing_document, which shows the change, and then proxium_apply_routing_document. proxium_list_recommendations returns the same groups, with the same counts and the same sentence.
These are the kinds of finding:
| Kind | Proxium flags | Severity |
|---|---|---|
| Cost | One of your largest spend lines this month, with the price of a million tokens. When the same vendor has a chat model that costs much less, the finding names it | High when the saving is large |
| Cache | A project with real traffic this month, where the cache saved little | High when the saving is large |
| Reliability | A vendor where many recent attempts failed. When the vendor refuses the model name, the title of the finding names the model, the cause and the screen to fix it | High when most of them failed |
| Classifier hop | A hop of the classifier chain that answers almost none of its calls. The finding names the reason that most of the failed calls of the hop give | High when it answers none |
| Retired model | A model that your failover policy retired recently, with the last failure | High while it is retired |
| Retired vendor | A vendor that your failover policy retired recently: no credits, a rejected key, or failed calls across its models | High while it is retired |
| Unknown model | An app that asks for a model name that the vendor does not list, with its recent calls and the model that answered them | High |
| Cut answer | An app whose calls to one model end empty at max_tokens in at least 5% of attempts: the model spent the whole budget on thinking. Raise max_tokens, or use a model that does not reason | High from 25% |
If the screen says Nothing to flag, your traffic has none of these patterns.
Other findings holds a finding of a kind that this page has no group for yet. It keeps its own title on the page.
Fix a hop of the classifier chain
For a Classifier chain finding, read the reason that the finding gives. A hop that refuses the model name needs a fix on Routing. The same holds for a hop that cuts every answer at the length limit.
Move traffic to a cheaper model
For a cost finding, read the sentence of the finding first. It gives the spend of the line and the price of a million tokens. When the catalog has no cheaper model for that traffic, it says so, and the line needs no change.
- Read the cheaper model that the finding names, and its price.
- Open Routing › Model catalog, and compare Cost, per Mtok of the two models.
- Test the cheaper model on a few real prompts of the app.
- In Text routing, select Change on the tier of the app, and put the cheaper model first.
- Select Save for this project.
Your apps need no change, because they send the tier name. To move one app only, add a rule for that app. See Route requests to models.
Let the cache answer repeated calls
For a cache finding: the cache answers only a request that is the same, field for field, as an earlier one. These changes give more hits:
| Change | Reason |
|---|---|
| Keep the system prompt the same on each call | A changed character is a new request |
Send the same temperature and tools each time | Both fields are part of the cache key |
| Put a date or an id in a message only when the answer depends on it | Each new value is a new request |
Overview › Answer cache shows what the cache of Proxium saved. The response cache explains the key.
Keep the cache of the vendor
A vendor also caches the start of a long conversation, and reads that part at a lower price. An agent that sends a long history on each step gets most of its input from this cache.
- Overview › Vendor cache shows the share of the input that came from it, what that saved, and how many calls lost it.
- Requests shows the cached share of each call. Select Lost the cache to list the calls that read less than half of their input from the cache.
- A call that moves to another vendor loses the cache, because that vendor does not hold the conversation. Keep one vendor first in the chain of a tier.
- Change the start of the conversation as little as you can. A changed system prompt, or a changed order of the tools, makes the vendor read the whole input again.
Fix a vendor that fails often
For a reliability finding:
- Open Performance, and read Failures › Failure causes and Failures by provider for that vendor.
- If the reason is a rejected key or a used quota, fix the key at the vendor, or add it again on Providers.
- Else, move the vendor down in your chains, so that a better vendor answers first.
Each failed attempt adds time before the answer. Debug a failed call shows how to read the attempts.
Fix a retired model, a retired vendor or an unknown model
For a retired model finding: remove the model from the tiers that name it on Routing, or replace it with another model. Proxium does not change your tiers.
For a retired vendor finding, the fix depends on the reason:
- No credits: add credits at the vendor.
- A rejected key: replace the key on Providers.
- Failed calls: check the status of the vendor, or move its models down your tiers.
For an unknown model finding: fix the model name in the code of the app that the finding names. Or, to keep the name, make it a tier on Routing. Until you fix it, each call sends the name to the vendor first, until your failover policy retires it. See Retire a model or a vendor.
Related pages
- Track spend: where the money goes.
- Set budgets and limits: cap the spend of one app.