Why is your inference bill so high?
Compute is scarce.
Inference Providers will keep charging more and more money.
As your users grow, your agentic workflows will only consume more tokens from this point onwards.
Inference bill only goes up from now on.
As the Agents take over most enterprise workflows, teams that say they need more AI capacity.
What they don't realize is that they are underusing the capacity they already pay for. When an agent workload hits a wall, the obvious thing is to ask for a higher quota, buy provisioned throughput, or add a second provider. In my experience building agent evaluation pipelines and infrastructure, the blocker is usually somewhere else.
This piece covers five places where enterprise agent spend leaks: idle compute, mis-sized concurrency, evaluation bills that outgrow the agent, real-time inference for work that could wait, and self-hosting decisions made on the wrong math.
Each one has a fix that costs far less than asking for more capacity.
1. Your agents leave capacity idle 80% of the time
An agent is not a chatbot. In a typical task it reads the prompt, plans, calls the model, then spends most of its time running tools, editing files, and executing code. In our workflows, when I profiled the agent evaluation pipeline, agents were calling the infernece provider only about 20% of their lifetime.
That matters a lot whether your capacity is GPUs or quota. With self-hosted models or provisioned throughput, you pay by the hour, so an idle GPU is money spent on nothing. With pay-per-token APIs capped by a quota your provider won't raise, idle tokens-per-minute are also counted as work that never happened.
Suppose you run an pipeline where each task runs in a sandbox on a batch scheduler cluster. You cap concurrency at a fixed 25 tasks to avoid throttling. The scheduler can't see when the agent inside a sandbox is actually using the model, so tuning this won't help.
Instead, move the control point down to where the bottlenecked resource is consumed: the inference call. A token-aware rate limiter lets a model call through when capacity is free and queues the request when limit-cap has hit.
Run the numbers from the next section, and the same capacity can support around 500 concurrent tasks instead of 30, with no throttling. That is roughly 16x the throughput. On fixed-cost capacity, cost per task falls by about 94%. On a capped quota, agent task runs finish in a fraction of the time without a single extra token.
2. Size concurrency for the average, not the peak
Most task concurrency limits are set by by peak/avg load, and both leave money and time on the table.
The right number comes from three measurements: your quota, your average tokens per task, and your average task duration.
Let's say your,
| Measurement | Value |
|---|---|
| Inference provider Quota | 5M tokens per minute (TPM) |
| Tokens per task | 100k |
| Task duration | 10 minutes |
| Average consumption per task | 10k TPM |
| Share of time calling the LLM | 20% |
| Consumption while calling the model | about 50k TPM |
If you estimate for the peak, you assume every task burns 50k TPM at once and cap concurrency at about 100. If you size for the average, the ceiling is 5M divided by 10k, or about 500 tasks. That is 5x more work on the same quota.
The catch is that sometimes many tasks call the model at the same moment, and that burst can push you over your limit. An inference-level gating can work well in this scneario: it makes the extra calls wait instead of failing with throttling errors and retry storms. Going above the average-based ceiling doesn't add throughput, it just makes tasks wait longer.
Check which limit you are actually hitting before changing anything. Most providers enforce both TPM and requests per minute (RPM). Hitting RPM with plenty of TPM to spare usually points to retry storms or bursty clients, not a lack of capacity.
3. Your monitoring bill can outgrow your agent
Better models don't remove the need for evaluation. Agents act dynamically and non-deterministically, so any system serving hundreds of users needs an observability layer that checks what agents actually did. The problem is what that layer costs when every check is an LLM judge.
The mechanism most teams miss: a judge re-reads the whole trace for every criterion it scores. Score three criteria on a 20k-token trace and you spend about 60k judge input tokens to evaluate one run(assuming no cache hits). If the judge is the same class of model as the agent, evaluation can cost more than the work it is evaluating.
The fix is to stop treating every agent action the same way:
- Deterministic checks first. Schema validation, tool-call success, and policy rules cost almost nothing and filter out obvious failures before a judge sees them.
- Judge high-stakes actions immediately. Anything that touches money, customer data, or irreversible operations gets evaluated in an online fashion.
- Batch everything else. Low-stakes actions can be judged hours later at a steep discount (next section). This allows you to still obseve agent behaviour that you can steer in future iterations.
- Use smaller judges for simple semantics. Many criteria, like tone or whether the agent stayed on task, don't need a frontier model.
The same tiering is how an eval suite that takes 12 hours can be run under one hour: run cheap checks first, judge only high-stake components in live traces, and defer the rest.
4. The 50% discount most eval pipelines ignore
Batch inference cuts the price of latency-tolerant work in half, and most teams run all their evaluations in real time anyway. On Amazon Bedrock, select models from Anthropic, Meta, Mistral AI, and Amazon are offered for batch inference at 50% below on-demand pricing, with a typical job completing within 24 hours (AWS). Other major providers offer similar batch discounts.
Work that qualifies for batch:
- Evaluation of low-stakes agent actions, scored for trend monitoring rather than intervention
- Regression suites run before a release, not during a user session
- Re-scoring historical traces after you change a rubric or a judge model
- Offline data processing, labeling, and synthetic data generation
Work that shouldn't be run on batch inference: anything where the result changes what happens next in an ongoing session.
Batch inference helps with scarce quota too. On many providers, batch jobs are governed by their own quotas rather than your real-time limits, so moving deferred work there can free on-demand capacity for the calls that need it.
Check how your inference provider handles this.
5. When does self-hosting comes into picture
Self-hosting can cut inference cost substantially, but only at high, steady utilization. If your GPU is not running jobs at 2AM, you're still paying for it.
Here is the decision path I'd follow for evaluation workloads:
- Is the data customer-sensitive or regulated? If it can't leave your environment, self-hosting is a requirement, not a cost-optimization. Decide that first.
- No eval system yet? Start on an API. Build a small dataset, get human labels, and iterate on an LLM judge until its scores track real customer outcomes.
- Volume growing? Shadow, don't switch. Route 5โ10% of eval traffic to a self hosted open-weight model and compare its running average against the current Evals.
- Switch only when quality matches and utilization stays high. Idle GPUs erase the savings, which is why agent inference load profiling matters here too.
The operational cost of GPU management is also real. Serving infernece requests well means understanding prefill versus decode, batching, and kernel performance, even with frameworks like vLLM and SGLang handling much of it. For mid-size enterprises with steady volume and enough engineering people, that trade is often worth it. For spiky, early-stage workloads, it usually doesn't make sense.
To Summarize,
So, ask these questions before you buy more compute, or throw money at the problem(your infernece can consume infinite amount of money if you let it).
- What share of each task's lifetime is spent calling the model?
- Is our concurrency sized for the peak, the average, or a guess?
- Are we hitting TPM or RPM, and are retries part of the problem?
- Which evaluations need a judge right now, and which could run in batch at half the price?
- If we self-host, will utilization stay high enough to pay for the GPUs and the Ops cost?