Where does LLM API spend actually go in a backend?
This guide is for backend and platform engineers who own LLM-backed features and have to explain why the monthly bill doubled. The answer usually lies in request shape, not request count.
Each call is billed roughly as:
cost = (uncached input tokens × input price) + (cached input tokens × cached input price) + (output tokens × output price)
Output tokens are often priced several times higher than input tokens. A feature that generates long answers can therefore cost more than one that sends large prompts. Cached input is usually discounted, which is why prompt structure matters in the next section.
Illustrative estimate (assumed numbers, not a benchmark): assume $1 per million input tokens, $4 per million output tokens, and a request with 3,000 input and 500 output tokens. Cost = 3,000 × $1/1M + 500 × $4/1M = $0.003 + $0.002 = $0.005. At 200,000 requests a day, that is $1,000/day. Output is about 14% of the tokens but 40% of the cost.
Before changing anything, log input, cached input and output tokens for every request, tagged by feature, tenant and model, and aggregate them into a cost-per-feature view. Without that view you will optimize the endpoint you suspect instead of the one that actually spends. That is often a background job or a single heavy tenant.
The order matters: measure, then cache, then batch, then route. Caching doesn't change outputs. Batching only adds latency to work that can wait. Routing changes which model answers, so it depends on the quality baselines your instrumentation provides.
import json, logging, time
from collections import defaultdict
log = logging.getLogger("llm_usage")
# Assumed prices in USD per million tokens (illustrative, not real rates).
# Keep this table in config so price changes don't require a deploy.
PRICES = {
"model-large": {"input": 1.00, "cached_input": 0.10, "output": 4.00},
"model-small": {"input": 0.20, "cached_input": 0.02, "output": 0.80},
}
def request_cost(model, uncached_in, cached_in, out):
p = PRICES[model]
return (uncached_in * p["input"] + cached_in * p["cached_input"]
+ out * p["output"]) / 1_000_000
def record_usage(feature, tenant, model, uncached_in, cached_in, out):
# Emit one structured line per LLM call. Map these fields from your
# provider's usage object. Tag with feature, tenant and model so spend
# can be grouped by any of them later.
event = {"ts": time.time(), "feature": feature, "tenant": tenant,
"model": model, "uncached_input_tokens": uncached_in,
"cached_input_tokens": cached_in, "output_tokens": out,
"cost_usd": request_cost(model, uncached_in, cached_in, out)}
log.info(json.dumps(event))
return event
def cost_per_feature(events):
# Roll up logged events into the view you optimize from.
totals = defaultdict(lambda: {"requests": 0, "cost_usd": 0.0})
for e in events:
t = totals[(e["feature"], e["model"])]
t["requests"] += 1
t["cost_usd"] += e["cost_usd"]
return sorted(totals.items(), key=lambda kv: -kv[1]["cost_usd"])
events = [record_usage("support-chat", "tenant-42", "model-large", 3000, 0, 500),
record_usage("nightly-summaries", "tenant-7", "model-large", 1200, 0, 1500)]
for (feature, model), t in cost_per_feature(events):
print(f"{feature:20} {model:12} {t['requests']:>4} req ${t['cost_usd']:.4f}")
A provider-agnostic Python sketch that logs uncached input, cached input and output tokens per call, tagged by feature, tenant and model, prices each call from an assumed price table, and rolls the events up into a cost-per-feature view.
How does prompt caching work, and how do you structure prompts for it?
Providers store the intermediate computation for a prompt's leading tokens. When a later request starts with exactly the same token sequence, that work is reused instead of recomputed, and it is billed at a reduced rate. The match is a strict prefix match. If a single token differs at position N, nothing after N can be reused. That is why ordering matters more than anything else. Put the system prompt first, then tool definitions, few-shot examples and static reference documents. Put per-user and per-request content last: conversation history, retrieved chunks and the user's message.
Most cache misses come from small, accidental differences near the top of the prompt:
| Cache breaker | Why it breaks the prefix | Fix |
|---|---|---|
| Timestamp or request ID in the system prompt | Changes on every call | Move it to the end, or drop it |
| Tool list built from a set or dict | Order can vary between processes | Sort tools by name before sending |
| JSON with unordered keys or varying whitespace | Same data serializes to different bytes | Sort keys and fix the separators |
| User name or locale injected early | Splits the cache per user | Append it after the static block |
Deterministic serialization means the same logical object always produces byte-identical output. In practice, sort the keys, fix the separators and keep the encoding stable.
Providers differ on several points: whether caching is automatic or needs explicit markers, the minimum cacheable length, how long entries live, whether cache writes cost extra and how large the read discount is. Check each provider's current documentation before relying on any of these.
Treat cache hit rate as a first-class metric. Track cached input tokens as a share of total input tokens, per route. Alert when the share drops, because a refactor that reorders a prompt can quietly multiply your input costs.
import json
# Static content that must be byte-identical on every request.
SYSTEM_PROMPT = "You are a support assistant for an internal billing tool."
tools = {
"lookup_invoice": {"description": "Fetch an invoice by ID", "params": {"invoice_id": "string"}},
"refund": {"description": "Issue a refund", "params": {"amount": "number", "invoice_id": "string"}},
}
def stable_json(obj):
# Sorted keys, fixed separators and no ASCII escaping variation:
# the same logical object always yields the same bytes.
return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
def build_prompt(user_name, history, retrieved_chunks, user_message, now_iso):
# Sort tools by name so dict/set ordering across processes cannot reorder them.
tool_block = stable_json([{"name": n, **tools[n]} for n in sorted(tools)])
# Cacheable prefix: identical for every user and every request.
static_prefix = SYSTEM_PROMPT + "\n\nTOOLS:\n" + tool_block
# Per-user and per-request content goes last, including the timestamp.
dynamic_suffix = (
"\n\nUSER: " + user_name
+ "\nTIME: " + now_iso
+ "\nHISTORY:\n" + stable_json(history)
+ "\nCONTEXT:\n" + "\n".join(retrieved_chunks)
+ "\nMESSAGE: " + user_message
)
return static_prefix, dynamic_suffix
def cache_hit_rate(cached_input_tokens, total_input_tokens):
# Track per route; alert when this share regresses from its baseline.
return cached_input_tokens / total_input_tokens if total_input_tokens else 0.0
a, _ = build_prompt("alice", [], ["doc A"], "Where is my refund?", "2026-11-10T09:00:00Z")
b, _ = build_prompt("bob", [], ["doc B"], "Cancel invoice 42", "2026-11-10T09:05:00Z")
assert a == b # Different users and times still share the same prefix.
Builds a prompt with a byte-stable static prefix (system prompt plus sorted, deterministically serialized tools) and appends user-specific content and timestamps at the end, plus a helper for computing cache hit rate per route.
When should you avoid prompt caching and response caching?
Prompt caching only pays off when requests share an identical prefix within the provider's cache lifetime. An internal tool that gets a few calls an hour, or an endpoint that puts user-specific data early in the prompt, may rarely produce a hit. Where providers charge more for writing a cache entry than for normal input, a low hit rate costs more than not caching. Caching also limits how you change prompts. A one-word edit to a system prompt, or an A/B test that serves two variants, splits the shared prefix and lowers the hit rate without any error showing up.
Response caching is a separate layer that you run in your own application. Exact-match caching stores the output under a key built from the normalized prompt, model identifier and sampling parameters. It is safe for deterministic, non-personalized calls, such as classifying a fixed product description at temperature zero. Its main risk is stale answers, so set a TTL that matches how often the underlying data changes.
Semantic caching returns a stored answer when a new prompt is close enough in embedding space. Similar wording does not mean the user wants the same thing: "cancel my order" and "can I cancel my order?" sit close together but call for different actions. A loose similarity threshold produces confident wrong answers. Without tenant scoping in the cache key, one customer can receive another customer's data.
| Technique | Safe when | Main risk |
|---|---|---|
| Provider prompt caching | High-volume, stable prefixes | Write surcharge with low hit rate |
| Exact-match response cache | Deterministic, non-personalized calls | Stale answers |
| Semantic response cache | Strict threshold, per-tenant scope | Wrong answers, cross-tenant leaks |
import hashlib
import json
import time
# In-process store for illustration; production would use a shared cache with TTL support.
_cache: dict[str, tuple[float, str]] = {}
TTL_SECONDS = 3600 # match how often the underlying product data changes
def normalize(prompt: str) -> str:
# Collapse whitespace so trivial formatting differences map to one key.
return " ".join(prompt.split())
def cache_key(tenant_id: str, model: str, params: dict, prompt: str) -> str:
# Tenant, model and every sampling parameter are part of the key:
# a change to any of them must never reuse another entry's output.
payload = json.dumps(
{"tenant": tenant_id, "model": model, "params": params, "prompt": normalize(prompt)},
sort_keys=True,
)
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
def cached_completion(tenant_id, model, params, prompt, call_model):
# Only cache deterministic calls; sampled output is not a stable answer.
if params.get("temperature", 1.0) != 0:
return call_model(model, params, prompt)
key = cache_key(tenant_id, model, params, prompt)
hit = _cache.get(key)
if hit and time.time() - hit[0] < TTL_SECONDS:
return hit[1]
result = call_model(model, params, prompt)
_cache[key] = (time.time(), result)
return result
An exact-match response cache that keys on tenant, model, sampling parameters and a normalized prompt, caches only temperature-zero calls, and expires entries after a TTL to limit stale answers.
When should you use a batch API instead of synchronous calls?
Batch APIs accept a file or list of requests, process them asynchronously, and return results later at a substantial per-token discount. Providers can offer the discount because they schedule batch work into spare capacity. The tradeoff is the completion window. It is long, often measured in hours, and it is a ceiling rather than a promise. Some items may still be unfinished when the window closes.
That makes batching a good fit for work where nobody is waiting on the result:
- Evaluation runs against fixed datasets
- Backfills after a prompt or schema change
- Bulk classification and tagging
- Embedding generation for new or re-chunked corpora
- Nightly summarization of tickets, logs or documents
- Moderation queues that tolerate delayed review
It does not belong on user-facing paths. A request that might take hours, or might not finish at all, cannot back an interactive response, and you cannot retry your way to a latency SLA.
Treat a batch as a distributed job:
- Idempotent request IDs. Give every item a stable ID derived from its input, such as a hash of the record key and prompt version. Results then map back unambiguously, and resubmitting an item never creates a duplicate downstream write.
- Durable job tracking. Store the batch ID, submission time and per-item state in your database, not in process memory. Pollers can then restart safely.
- Partial-failure handling. Expect a mix of succeeded, errored and expired items in the same batch.
- Reconciliation. Diff the returned IDs against the submitted ones. Treat any item that is missing from the results as failed.
- Resubmission. Collect expired or retryable failures into a new batch, with a retry cap.
This is real operational overhead: a scheduler, a state table, a poller and alerting on stuck jobs. It pays off when volume is large enough that the discount outweighs the cost of maintaining that machinery.
import hashlib
import json
PROMPT_VERSION = "summarize-v3" # bump when the prompt changes
MAX_ATTEMPTS = 3
def request_id(record_key: str) -> str:
# Stable ID from input + prompt version: same record, same ID, so retries never duplicate writes
raw = f"{record_key}:{PROMPT_VERSION}".encode()
return hashlib.sha256(raw).hexdigest()[:32]
def reconcile(submitted: dict, results_jsonl: str) -> tuple[dict, list]:
# submitted maps request_id -> attempt count, as loaded from the durable state table
succeeded, retry = {}, []
returned = set()
for line in results_jsonl.splitlines():
item = json.loads(line)
rid = item["custom_id"]
returned.add(rid)
if item.get("status") == "succeeded":
succeeded[rid] = item["output"]
else:
retry.append(rid) # errored or expired: candidate for resubmission
# Anything submitted but absent from the results is treated as failed
retry.extend(rid for rid in submitted if rid not in returned)
# Enforce the retry cap; items over the cap go to alerting, not another batch
resubmit = [rid for rid in retry if submitted[rid] < MAX_ATTEMPTS]
return succeeded, resubmit
# Example with a provider-agnostic results file
submitted = {request_id(k): 1 for k in ["ticket-101", "ticket-102", "ticket-103"]}
results = "\n".join([
json.dumps({"custom_id": request_id("ticket-101"), "status": "succeeded", "output": "..."}),
json.dumps({"custom_id": request_id("ticket-102"), "status": "expired"}),
]) # ticket-103 is missing entirely
done, again = reconcile(submitted, results)
print(len(done), "succeeded;", len(again), "to resubmit") # 1 succeeded; 2 to resubmit
A provider-agnostic sketch that derives idempotent request IDs from the record key and prompt version, then reconciles batch results against submitted IDs. Errored, expired and missing items are queued for resubmission, up to a retry cap.
How does model routing work, and what are the alternatives to a single model?
Model routing makes a similar trade: it saves money per request but needs infrastructure and evaluation work to stay correct. The aim is to send each request to the cheapest model that handles it well, so that the large model is used only where its extra capability matters.
| Strategy | How it decides | Main risk |
|---|---|---|
| Static by task type | Endpoint or job type maps to a fixed model | Coarse; one hard input type drags a whole route up or down |
| Input-length rules | Token count thresholds pick the model | Length is a weak proxy for difficulty |
| Classifier routing | A small model or heuristic predicts difficulty | Misroutes are silent; the classifier itself needs evals |
| Cascade | Try small model, escalate on low confidence or failed validation | Pays twice on escalation; adds latency |
Cascades work best when failure can be detected mechanically. Examples are JSON that fails schema validation, tool calls that don't parse, or a refusal pattern. Self-reported confidence is much less reliable. The code example that follows shows this escalation logic.
A cascade only saves money when the escalation rate stays low. Every escalated request pays for both calls and waits for both. If a large share of traffic escalates, calling the large model directly is cheaper and faster.
Smaller models also tend to fail in ways that aggregate scores hide:
- edge cases
- non-English input
- reasoning over long contexts
A 2% drop in overall accuracy can mean a 30% drop for one customer segment.
For that reason, back every routing decision with offline eval sets built from real, anonymized traffic. Before switching a route, confirm the result with shadow traffic or an A/B comparison. Evals also go stale. Re-run them whenever a provider updates, re-points or deprecates a model, because routes that held up last quarter can quietly regress.
Which token-level cuts reduce LLM API costs without hurting quality?
Routing changes the price per token. The cuts below reduce how many tokens you send and receive.
- Cap output length. Output tokens usually cost more than input tokens, and an unbounded generation can run long. Set a max output limit per endpoint based on the observed response length for that task, not one global value.
- Ask for structured or concise output. A JSON schema or "answer in two sentences" removes preamble, restated questions and closing summaries the client discards anyway. Structured output also makes parsing failures easier to detect.
- Trim conversation history. Resending every prior turn makes cost grow with conversation length. Keep a sliding window of recent turns and fold older ones into a running summary, regenerated occasionally rather than on every turn.
- Deduplicate RAG context. Overlapping chunks and near-duplicate documents often fill the retrieved context. Deduplicate by content hash or similarity before assembly, and cap the number of chunks.
- Send only the tools you need. Large tool schemas count as input tokens on every call. Attach only the tools relevant to the current step.
Aggressive trimming can backfire. An output cap set too low truncates JSON mid-object. A window set too short drops the constraint the user stated ten turns ago. Each failure produces a retry, a follow-up message or an escalation to a larger model. The effective cost per task is:
cost per call × (1 + retry rate)
A 30% token cut is lost if the retry rate rises enough to offset it. Track truncation stop reasons and retry rates alongside token counts, per endpoint.
Illustrative estimate: how caching, batching and routing compound
Illustrative estimate (assumed numbers, not a benchmark)
All inputs are assumed for illustration:
- 1,000,000 requests per month.
- Each request has a 3,000-token fixed system prompt, 500 variable input tokens and 300 output tokens.
- Large model prices: $3 per million input tokens and $15 per million output tokens.
- Small model prices: one sixth of the large model's prices.
- Cached input is billed at 10% of the input price. The cache write premium is ignored.
- 80% of system-prompt tokens hit the cache.
- 30% of traffic is batchable at a 50% discount.
- 60% of traffic is routed to the small model.
Per segment, the formula is:
cost = requests × (cached tokens × cache price + uncached input × input price + output × output price) × batch multiplier
Sum the segments to get the monthly cost.
| Step | What changes | Monthly cost | Saving vs. baseline |
|---|---|---|---|
| Baseline | All synchronous, large model | $15,000 | — |
| + Prompt caching | 2,400 of 3,500 input tokens cached | $8,520 | 43% |
| + Batch (stacks with cache) | 30% of traffic at half price | $7,242 | 52% |
| + Batch (does not stack) | Batched share pays list price × 0.5, no cache discount | $8,214 | 45% |
| + Routing (on stacked case) | 60% of traffic at 1/6 price | $3,621 | 76% |
The savings do not add. Each technique shrinks the base the next one works on:
- After routing, the dollars caching saves on the small-model share are one sixth of what they were.
- Routing also splits cache warmth across two models, so the 80% hit rate is optimistic.
- Batched jobs may not hit a warm cache at all.
- Check whether your provider applies batch and cache discounts together. In this example, the difference between the two batch rows is almost $1,000 a month.
Each technique also has an ongoing engineering cost:
- Caching requires keeping the prompt prefix stable.
- Batching requires an asynchronous job pipeline with result reconciliation.
- Routing requires an evaluation set that you rerun whenever either model changes.
Do the cheapest one first, and measure before adding the next.
How can a central LLM gateway save engineering time?
Each of those controls is easier to build once than to build in every service. If you put them in a thin internal gateway or shared client library, feature teams get them by default and don't have to reimplement budget checks, retry logic, or prompt assembly themselves.
service ──► gateway ─┬─ log (feature, model, tokens in/out, cached tokens)
├─ normalize prompt (fixed prefix order, stable whitespace)
├─ budget check (per-feature spend; reject or downgrade)
├─ router (pick model tier, fallback on error/timeout)
│
├──► sync path ──► provider A / provider B
└──► batch path ─► job queue ─► provider batch API
└─► results store ─► callback/poll
Each stage exists for a specific reason. Normalization keeps cache hit rates from depending on how carefully each team orders its system prompt. Per-feature budgets turn a runaway loop into a rejected request instead of a surprise invoice. Usage logs tagged by feature are what let you check whether routing changes actually lowered cost. Fallbacks sit in one place, so a provider outage is handled consistently across services.
The trade-offs are real:
- Extra hop. A remote gateway adds latency. A client library avoids the hop, but you then have to roll out upgrades to every service.
- Single point of failure. Run it redundantly, and decide in advance whether it fails open or fails closed when the budget store is unreachable.
- Ownership. Without a named owning team, the gateway decays into an unmaintained proxy.
- Bottleneck and lag. Teams wait on the gateway for new provider features. Provide a documented passthrough so teams aren't blocked.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to reduce llm api costs, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.