LLM Cost Optimization: Cost per Task, Routing, Batching and Context

How to cut LLM costs without hurting quality: measure cost per task, route to smaller models, trim context, cache, batch offline work and cap agent loops.

Intermediate 3 min read · Updated 29 October 2026

LLM features can be expensive in a way most software isn't: every request has a visible marginal cost, agents can multiply it unpredictably, and costs scale with usage rather than servers. LLM cost optimization starts with measuring the right unit (cost per successful task, not per call), then applies the levers in order of impact: model routing, context trimming, caching, batching, output control and bounding agent loops, always checked against the evaluation suite so quality doesn't silently drop.

Measure cost per task

A cheap call that fails and triggers three retries and a human escalation isn't cheap. Attribute every model call (tokens in, tokens out, model, cached tokens) to a task id through tracing, and compute cost per resolved task alongside quality.

cost per resolved conversation =
    Σ model calls (input_tokens × in_price + output_tokens × out_price − cache discounts)
  + embedding + reranking + tool/API costs
  ÷ conversations resolved without human escalation

Track it per feature, per customer tier and per model version, next to quality metrics.

Where the tokens go

An illustrative token breakdown for one turn of a RAG chat assistant. Input context, not the answer, is usually the largest cost, which is why trimming and caching context pay off.

The levers, in rough order of impact

Lever	How	Typical effect	Risk
Model routing	Send easy requests to a small model; escalate hard or low-confidence ones	Large: small models can cost an order of magnitude less	Quality drop on misrouted requests
Context trimming	Fewer, better retrieved chunks (reranking), summarised history, compact tool outputs	Large on input-heavy apps	Missing evidence
Prompt caching	Stable prefix first so repeated tokens are billed at a discount	Large for long system prompts	None to quality
Response caching	Exact or semantic caches for repeated questions	Depends on repetition	Stale or wrong hits
Batch processing	Use asynchronous batch APIs or offline GPU batches for non-interactive work	Often a large discount	Hours of latency
Output control	Ask for concise answers, set max tokens, structured output instead of prose	Moderate (output tokens cost more)	Truncation
Bound agent loops	Step and token budgets, better tools so fewer steps are needed	Prevents worst cases	Unfinished tasks
Fine-tune or distil	Train a small model to do a narrow task the big one did	Large at high volume	Lifecycle cost
Self-host	Run open models on your GPUs at high utilisation	Large at steady, high volume	Ops burden, idle capacity

Routing with a cascade

A model cascade: most requests are answered by the small model; a confidence or validation check escalates the rest to the large model. Cost falls sharply while quality on hard requests is kept.

Before: 10M requests/day, all on the large model at $0.012 each   = $120,000/day
After : router sends 70% to the small model at $0.0012 each       = $8,400
        10% of those fail validation and escalate (0.7M × $0.012)  = $8,400
        30% routed straight to large (3M × $0.012)                 = $36,000
        total ≈ $52,800/day (−56%), with eval scores held within 1 point
(illustrative per-request costs)

Cost controls in production

Budgets per tenant, team and feature, enforced in the gateway, with alerts at 50/80/100%.
Per-task caps on tokens and steps for agents; log the tasks that hit them.
Abuse protection: input length limits and per-user rate limits stop someone pasting a book into a free tier.
Cost dashboards by feature and model; review the top 1% most expensive tasks weekly, since they're usually bugs (loops, huge tool outputs).
Pricing alignment: if a feature's cost per user exceeds its revenue per user, product limits (quotas, tiers) are part of the design.

API vs self-hosted economics

Self-hosted GPU at $2/hour serving a small model at ~2,000 output tokens/s aggregate
  → 7.2M tokens/hour → ~$0.28 per 1M output tokens at 100% utilisation
  → at 30% average utilisation: ~$0.93 per 1M (idle time is the hidden cost)
Compare to the API price for an equivalent model, add engineering and on-call cost,
and self-host only when traffic is high and steady enough to keep GPUs busy.
(illustrative numbers)

See LLM serving for what drives GPU throughput, and model compression for quantization.

Where it’s used

Consumer assistants

Routing, prompt caching and concise outputs keep cost per active user below revenue per user.

Back-office automation

Batch APIs and distilled small models process millions of documents cheaply overnight.

Agents

Step budgets, better tools and summarised context bound the long tail of expensive runs.

In an interview

Measure cost per successful task, attributed through traces.
Show where tokens go and trim the biggest bucket first (usually context).
Propose a routing cascade and quantify its effect.
Add budgets, per-task caps and guardrails against abuse, and check quality with evals.

Frequently asked questions

How do you reduce LLM costs?

Measure cost per successful task, then route easy requests to smaller models, trim retrieved context and history, use prompt and response caching, batch offline work, limit output length, cap agent loops, and fine-tune or distil small models for high-volume tasks.

What is a model cascade?

A routing pattern where a small, cheap model handles a request first and only escalates to a larger model when a confidence or validation check fails.