LLM Cost Control: Token Budgets, Caching and Routing in 2026

LLM Cost Control: Token Budgets, Caching and Routing in 2026 — cover image

Your AI feature works beautifully in the demo. Then real users arrive, conversations get longer, and the monthly invoice from your model provider looks nothing like the spreadsheet you built during planning. This is one of the most common surprises we see when founders move from prototype to production, and it is almost always fixable without making the product worse.

This guide explains where LLM costs actually come from, and how to control them with token budgets, caching, model routing and monitoring. It is written for founders and product teams at startups and SMEs who are shipping AI features and want predictable unit economics.

Where LLM Costs Really Come From

Providers typically charge per token, with input tokens and output tokens priced differently. That sounds simple, but a single user action in a modern AI product often triggers many model calls. Cost grows from five places:

Before optimising anything, measure. If you cannot say which feature, which tenant and which prompt version produced yesterday's spend, you are guessing. Our guide to testing AI features with an evaluation suite pairs well with this one, because every cost cut needs a quality check behind it.

A Real-World Style Example

Consider an illustrative scenario. A logistics startup adds an AI assistant that answers shipment questions for dispatchers. The first version sends a 2,000 word system prompt, the last 20 chat messages and eight retrieved documents on every turn. For example, if each request carried roughly 12,000 input tokens and the team served a few thousand requests a day, the monthly bill could easily end up several times higher than the founders expected.

After a review, the team might make four changes. They trim the system prompt and move rarely used rules into retrieved snippets. They summarise old chat turns instead of resending them. They cache the stable prompt prefix. And they route simple status lookups to a small, fast model while keeping the stronger model for exception handling. In a scenario like this, it is realistic to expect a large reduction in token volume per request, though the exact saving depends on your traffic and provider pricing. The point is the method, not the number.

Treat every token like a line item. If it does not change the answer, it should not be in the prompt.

Step-by-Step: Bring LLM Spend Under Control

  1. Instrument every call. Log model, feature name, prompt version, input tokens, output tokens, latency, tenant ID and outcome. Without this, no later step is measurable.
  2. Find the top three cost drivers. Sort spend by feature and by prompt. In most products a small number of prompts account for the majority of cost. Start there.
  3. Set token budgets per feature. Decide a maximum input and output size for each feature, for example a cap on retrieved chunks and a cap on answer length. Enforce it in code, not in a wiki page.
  4. Shrink the prompt. Remove duplicated instructions, replace long examples with one or two good ones, and ask for structured, concise output. If you use JSON, see our notes on structured outputs and schema reliability so shorter responses stay valid.
  5. Add prompt caching. Put stable content (instructions, policies, reference documents) at the start of the prompt and variable content at the end, so the provider can reuse the cached prefix where it supports this.
  6. Cache answers where it is safe. For repeated, non-personal questions such as pricing FAQs, store the answer keyed by a normalised question and a document version. Expire it when the source changes.
  7. Route by difficulty. Send classification, extraction and short rewrites to a small model. Escalate to a stronger model only when confidence is low or the task needs multi-step reasoning.
  8. Cap loops and retries. Limit agent steps, tool calls and total tokens per task. Use exponential backoff on retries and never retry blindly on a validation failure without changing the input.
  9. Use batch and async modes. For work that does not need an instant answer, such as nightly summaries or bulk tagging, batch endpoints are often priced lower than real-time calls.
  10. Alert on change, not only on totals. Alert when average tokens per request jumps, when a single tenant spikes, or when daily spend passes a threshold.

Token Budgets in Practice

A token budget is simply a promise your code keeps. A practical budget has four numbers: the maximum system prompt size, the maximum retrieved context, the maximum history window and the maximum output length. When the assembled prompt exceeds the budget, a trimming policy decides what to drop first. A sensible order is oldest chat turns, then lowest ranked retrieved chunks, and only last the instructions that define behaviour.

Budgets also protect you from abuse. Without them, one user pasting a huge document can cost more than a hundred normal users. Combine budgets with rate limits and per-tenant caps. Our article on API rate limiting and abuse protection for SaaS covers the same ideas for the rest of your API surface.

Model Routing Without Breaking Quality

Routing sounds complicated but can start very simply. Begin with a rules-based router: if the task type is "extract fields" or "classify intent", use the small model. If the request mentions multiple documents or requires comparison, use the larger one. Later you can add a lightweight classifier or use the small model's own confidence score to decide when to escalate.

Always keep a fallback path. If the small model returns an invalid format or a low-confidence answer, retry once with the stronger model rather than showing the user a bad result. Keep your routing logic behind a single gateway so you can swap providers later, a point we explore in our guide to avoiding LLM vendor lock-in.

Retrieval Hygiene: Fewer, Better Chunks

In RAG systems, retrieved context is often the largest part of every prompt. More chunks do not always mean better answers. Tune the number of chunks, use reranking so that the best passages come first, and drop passages below a relevance threshold. Smaller, cleaner context is cheaper and often more accurate, because the model has less irrelevant text to be distracted by.

Key Benefits of Controlling LLM Costs

Common Mistakes to Avoid

How Mavani Approaches AI Cost Design

Across the 37+ products Mavani has delivered, we have found that cost design works best when it starts in the architecture phase, not after launch. We add logging, budgets and a model gateway from the first sprint, so that scaling never requires a rewrite. If you are planning an AI feature and want an architecture that keeps spend predictable, explore our AI development services or use the app cost calculator to get a first estimate for the wider project.

Frequently Overlooked Cost Levers

Beyond the big moves, several smaller habits add up. First, ask for shorter outputs. Many prompts invite long explanations that users never read, and output tokens are usually priced higher than input tokens. A clear instruction such as "answer in three sentences unless asked for detail" often improves the experience as well as the bill. Second, pre-process documents before they reach the model. Strip navigation text, repeated headers and boilerplate from pages and PDFs, because every wasted token is paid for on every call that includes it.

Third, separate interactive work from background work. A support chat needs a fast reply, but weekly report summaries can wait for a cheaper batch run. Fourth, review your embeddings and search settings. Re-embedding unchanged documents on every sync is a surprisingly common source of waste, so store a content hash and only process what changed. Finally, set a monthly review. Prices, model options and your own traffic all change, so a routing rule that was right in January may be wrong by June. Put the review on the calendar, compare cost per successful task over time, and share the numbers with product and finance so everyone understands the trade-offs.

It also helps to define a simple unit metric, such as cost per resolved conversation or cost per processed document. Raw monthly spend rises naturally as you grow, which is a good sign. Cost per unit of value is the number that tells you whether the system is becoming healthier or more wasteful.

Conclusion

LLM cost control is not about using the weakest model or starving your product of context. It is about measuring where tokens go, setting clear budgets, reusing work through caching and sending each task to the right model. Teams that do this early enjoy faster products, calmer finance conversations and the freedom to ship more AI features. Start by logging every call this week. Once you can see the spend, the savings usually become obvious.

Frequently Asked Questions

Why do LLM API bills grow faster than expected?
Most bills grow because of long prompts, repeated context, retries and agent loops that call the model many times per user action. Cost per request looks small in testing, but multiplies quickly once real users arrive and conversations get longer.
What is prompt caching and when does it help?
Prompt caching lets a provider reuse the processed form of a long, stable prompt prefix such as system instructions or reference documents. It helps most when many requests share the same large prefix and only a small part changes per call.
Should every request use the cheapest model?
No. Cheap models are great for classification, extraction and short rewrites, but they may struggle with complex reasoning. A routing layer that sends easy tasks to small models and hard tasks to stronger ones usually gives the best balance of cost and quality.
How do I stop a runaway agent from burning money?
Set hard limits on steps, tokens and wall-clock time per task, and add a per-user and per-day spend cap. When a limit is hit, the agent should stop, summarise what it did and hand off to a human or ask for confirmation.
How can a small team monitor AI spend?
Log tokens in and out, model name, feature name and user or tenant ID for every call. Send those logs to a simple dashboard and set alerts on daily spend and on sudden changes in average tokens per request.