Skip to main content

Lower your costs

The Improve screen reads the traffic of your project, and lists what to change. Each finding shows the numbers that it comes from.

Read the findings​

Open Improve in the console. One sentence above the readouts says what the money is. It gives the dollars a month that the findings with a saving add up to. It also gives the count of the other findings, which fix a failure or show where the cost sits. Most findings fix a failure, so they carry no dollar figure.

The readouts on top count the findings:

ReadoutShows
FindingsAll findings
Worth doing firstThe findings of high severity
Worth a lookThe other findings
Worth fixingThe estimated saving of each month, in dollars, from the findings that name one

Below the readouts, the findings are in groups. Each group has a count and the screen where you fix it:

GroupHoldsFix it on
ReliabilityThe vendors that fail many attempts nowProviders
Empty answersThe apps whose calls end empty at max_tokensRouting
Wrong model namesThe apps that ask for a model name that no vendor servesRouting
Classifier chainThe hops of the classifier that answer almost nothingRouting
CostThe spend lines that can go downRouting
CacheThe projects with real traffic where the cache saved littleUsage
Paused vendors and modelsOne folded table of the vendors and models that the failover policy pausedRouting
Other findingsA kind of finding that this page has no group for yet

An agent applies a change of traffic with proxium_plan_routing_document, which shows the change, and then proxium_apply_routing_document. proxium_list_recommendations returns the same groups, with the same counts and the same sentence.

These are the kinds of finding:

KindProxium flagsSeverity
CostOne of your largest spend lines this month, with the price of a million tokens. When the same vendor has a chat model that costs much less, the finding names itHigh when the saving is large
CacheA project with real traffic this month, where the cache saved littleHigh when the saving is large
ReliabilityA vendor where many recent attempts failed. When the vendor refuses the model name, the title of the finding names the model, the cause and the screen to fix itHigh when most of them failed
Classifier hopA hop of the classifier chain that answers almost none of its calls. The finding names the reason that most of the failed calls of the hop giveHigh when it answers none
Retired modelA model that your failover policy retired recently, with the last failureHigh while it is retired
Retired vendorA vendor that your failover policy retired recently: no credits, a rejected key, or failed calls across its modelsHigh while it is retired
Unknown modelAn app that asks for a model name that the vendor does not list, with its recent calls and the model that answered themHigh
Cut answerAn app whose calls to one model end empty at max_tokens in at least 5% of attempts: the model spent the whole budget on thinking. Raise max_tokens, or use a model that does not reasonHigh from 25%

If the screen says Nothing to flag, your traffic has none of these patterns.

Other findings holds a finding of a kind that this page has no group for yet. It keeps its own title on the page.

Fix a hop of the classifier chain​

For a Classifier chain finding, read the reason that the finding gives. A hop that refuses the model name needs a fix on Routing. The same holds for a hop that cuts every answer at the length limit.

Move traffic to a cheaper model​

For a cost finding, read the sentence of the finding first. It gives the spend of the line and the price of a million tokens. When the catalog has no cheaper model for that traffic, it says so, and the line needs no change.

  1. Read the cheaper model that the finding names, and its price.
  2. Open Routing › Model catalog, and compare Cost, per Mtok of the two models.
  3. Test the cheaper model on a few real prompts of the app.
  4. In Text routing, select Change on the tier of the app, and put the cheaper model first.
  5. Select Save for this project.

Your apps need no change, because they send the tier name. To move one app only, add a rule for that app. See Route requests to models.

Let the cache answer repeated calls​

For a cache finding: the cache answers only a request that is the same, field for field, as an earlier one. These changes give more hits:

ChangeReason
Keep the system prompt the same on each callA changed character is a new request
Send the same temperature and tools each timeBoth fields are part of the cache key
Put a date or an id in a message only when the answer depends on itEach new value is a new request

Overview › Answer cache shows what the cache of Proxium saved. The response cache explains the key.

Keep the cache of the vendor​

A vendor also caches the start of a long conversation, and reads that part at a lower price. An agent that sends a long history on each step gets most of its input from this cache.

  • Overview › Vendor cache shows the share of the input that came from it, what that saved, and how many calls lost it.
  • Requests shows the cached share of each call. Select Lost the cache to list the calls that read less than half of their input from the cache.
  • A call that moves to another vendor loses the cache, because that vendor does not hold the conversation. Keep one vendor first in the chain of a tier.
  • Change the start of the conversation as little as you can. A changed system prompt, or a changed order of the tools, makes the vendor read the whole input again.

Fix a vendor that fails often​

For a reliability finding:

  1. Open Performance, and read Failures › Failure causes and Failures by provider for that vendor.
  2. If the reason is a rejected key or a used quota, fix the key at the vendor, or add it again on Providers.
  3. Else, move the vendor down in your chains, so that a better vendor answers first.

Each failed attempt adds time before the answer. Debug a failed call shows how to read the attempts.

Fix a retired model, a retired vendor or an unknown model​

For a retired model finding: remove the model from the tiers that name it on Routing, or replace it with another model. Proxium does not change your tiers.

For a retired vendor finding, the fix depends on the reason:

  • No credits: add credits at the vendor.
  • A rejected key: replace the key on Providers.
  • Failed calls: check the status of the vendor, or move its models down your tiers.

For an unknown model finding: fix the model name in the code of the app that the finding names. Or, to keep the name, make it a tier on Routing. Until you fix it, each call sends the name to the vendor first, until your failover policy retires it. See Retire a model or a vendor.