Why is your inference bill so high?

ยท 6 min read

Compute is scarce.

Inference Providers will keep charging more and more money.

As your users grow, your agentic workflows will only consume more tokens from this point onwards.

Inference bill only goes up from now on.

As the Agents take over most enterprise workflows, teams that say they need more AI capacity.

What they don't realize is that they are underusing the capacity they already pay for. When an agent workload hits a wall, the obvious thing is to ask for a higher quota, buy provisioned throughput, or add a second provider. In my experience building agent evaluation pipelines and infrastructure, the blocker is usually somewhere else.

This piece covers five places where enterprise agent spend leaks: idle compute, mis-sized concurrency, evaluation bills that outgrow the agent, real-time inference for work that could wait, and self-hosting decisions made on the wrong math.

Each one has a fix that costs far less than asking for more capacity.

1. Your agents leave capacity idle 80% of the time

An agent is not a chatbot. In a typical task it reads the prompt, plans, calls the model, then spends most of its time running tools, editing files, and executing code. In our workflows, when I profiled the agent evaluation pipeline, agents were calling the infernece provider only about 20% of their lifetime.

That matters a lot whether your capacity is GPUs or quota. With self-hosted models or provisioned throughput, you pay by the hour, so an idle GPU is money spent on nothing. With pay-per-token APIs capped by a quota your provider won't raise, idle tokens-per-minute are also counted as work that never happened.

Suppose you run an pipeline where each task runs in a sandbox on a batch scheduler cluster. You cap concurrency at a fixed 25 tasks to avoid throttling. The scheduler can't see when the agent inside a sandbox is actually using the model, so tuning this won't help.

Instead, move the control point down to where the bottlenecked resource is consumed: the inference call. A token-aware rate limiter lets a model call through when capacity is free and queues the request when limit-cap has hit.

Run the numbers from the next section, and the same capacity can support around 500 concurrent tasks instead of 30, with no throttling. That is roughly 16x the throughput. On fixed-cost capacity, cost per task falls by about 94%. On a capped quota, agent task runs finish in a fraction of the time without a single extra token.

2. Size concurrency for the average, not the peak

Most task concurrency limits are set by by peak/avg load, and both leave money and time on the table.

The right number comes from three measurements: your quota, your average tokens per task, and your average task duration.

Let's say your,

Measurement Value
Inference provider Quota 5M tokens per minute (TPM)
Tokens per task 100k
Task duration 10 minutes
Average consumption per task 10k TPM
Share of time calling the LLM 20%
Consumption while calling the model about 50k TPM

If you estimate for the peak, you assume every task burns 50k TPM at once and cap concurrency at about 100. If you size for the average, the ceiling is 5M divided by 10k, or about 500 tasks. That is 5x more work on the same quota.

The catch is that sometimes many tasks call the model at the same moment, and that burst can push you over your limit. An inference-level gating can work well in this scneario: it makes the extra calls wait instead of failing with throttling errors and retry storms. Going above the average-based ceiling doesn't add throughput, it just makes tasks wait longer.

Check which limit you are actually hitting before changing anything. Most providers enforce both TPM and requests per minute (RPM). Hitting RPM with plenty of TPM to spare usually points to retry storms or bursty clients, not a lack of capacity.

3. Your monitoring bill can outgrow your agent

Better models don't remove the need for evaluation. Agents act dynamically and non-deterministically, so any system serving hundreds of users needs an observability layer that checks what agents actually did. The problem is what that layer costs when every check is an LLM judge.

The mechanism most teams miss: a judge re-reads the whole trace for every criterion it scores. Score three criteria on a 20k-token trace and you spend about 60k judge input tokens to evaluate one run(assuming no cache hits). If the judge is the same class of model as the agent, evaluation can cost more than the work it is evaluating.

The fix is to stop treating every agent action the same way:

  1. Deterministic checks first. Schema validation, tool-call success, and policy rules cost almost nothing and filter out obvious failures before a judge sees them.
  2. Judge high-stakes actions immediately. Anything that touches money, customer data, or irreversible operations gets evaluated in an online fashion.
  3. Batch everything else. Low-stakes actions can be judged hours later at a steep discount (next section). This allows you to still obseve agent behaviour that you can steer in future iterations.
  4. Use smaller judges for simple semantics. Many criteria, like tone or whether the agent stayed on task, don't need a frontier model.

The same tiering is how an eval suite that takes 12 hours can be run under one hour: run cheap checks first, judge only high-stake components in live traces, and defer the rest.

4. The 50% discount most eval pipelines ignore

Batch inference cuts the price of latency-tolerant work in half, and most teams run all their evaluations in real time anyway. On Amazon Bedrock, select models from Anthropic, Meta, Mistral AI, and Amazon are offered for batch inference at 50% below on-demand pricing, with a typical job completing within 24 hours (AWS). Other major providers offer similar batch discounts.

Work that qualifies for batch:

Work that shouldn't be run on batch inference: anything where the result changes what happens next in an ongoing session.

Batch inference helps with scarce quota too. On many providers, batch jobs are governed by their own quotas rather than your real-time limits, so moving deferred work there can free on-demand capacity for the calls that need it.

Check how your inference provider handles this.

5. When does self-hosting comes into picture

Self-hosting can cut inference cost substantially, but only at high, steady utilization. If your GPU is not running jobs at 2AM, you're still paying for it.

Here is the decision path I'd follow for evaluation workloads:

  1. Is the data customer-sensitive or regulated? If it can't leave your environment, self-hosting is a requirement, not a cost-optimization. Decide that first.
  2. No eval system yet? Start on an API. Build a small dataset, get human labels, and iterate on an LLM judge until its scores track real customer outcomes.
  3. Volume growing? Shadow, don't switch. Route 5โ€“10% of eval traffic to a self hosted open-weight model and compare its running average against the current Evals.
  4. Switch only when quality matches and utilization stays high. Idle GPUs erase the savings, which is why agent inference load profiling matters here too.

The operational cost of GPU management is also real. Serving infernece requests well means understanding prefill versus decode, batching, and kernel performance, even with frameworks like vLLM and SGLang handling much of it. For mid-size enterprises with steady volume and enough engineering people, that trade is often worth it. For spiky, early-stage workloads, it usually doesn't make sense.

To Summarize,

So, ask these questions before you buy more compute, or throw money at the problem(your infernece can consume infinite amount of money if you let it).