LLM cost optimization: three levers that cut your bill

You can cut most of your LLM spend without losing quality. The route is routing, caching, then batching, in that order. Get the measurement right first and the savings hold.

Where does LLM cost optimization actually start?

Your bill is not one number. It is a stack of smaller costs, and most teams only ever see the total.

Break it apart and the picture gets clear fast.

Input tokens versus output tokens. You pay for both. Output usually costs more per token. A model that talks a lot costs more than one that answers in three lines. If your prompts invite long, chatty replies, you are paying a premium for words nobody reads twice.

System prompts and history. Every message drags baggage with it:

• The system prompt, the instructions that set how the model behaves
• The full conversation history, resent on every turn
• Any RAG context you have bolted on to give the model extra facts

None of it is free. A 500-token system prompt, sent on every one of 10,000 daily chats, is 5 million tokens a day before a single user question gets answered.

The hidden multipliers. These are the ones that quietly blow budgets:

• Retry loops when a call fails or times out
• Validation retries when the output does not match your schema
• Logging that duplicates full prompts and responses
• Embedding generation and vector-store queries on every RAG lookup

Here is a worked example. Your system prompt runs 800 tokens. The average user question is 50. You retry failed calls twice on average. That is not one call at 850 tokens. It is closer to three calls at 850 tokens each, because every retry resends the whole context. Your real cost per resolved question just tripled and nothing on your dashboard said a word.

What metrics should you measure before optimising?

You cannot fix what you do not measure. Track five numbers from day one:

• Cost per request. The baseline. Everything else measures against it.
• Tokens in versus tokens out. Shows you where the bloat lives.
• Cache hit rate. The share of requests served from cache instead of a fresh call. Below 20% on repeat-heavy work usually means the cache keys need attention.
• Model routing distribution. How much traffic goes to the cheap model versus the expensive one.
• Retry rate. A high retry rate means you are buying the same answer more than once.

Pro Tip: Track cost per outcome, not raw token counts. Cost per resolved ticket. Cost per generated report. Cost per completed session. That framing stops you cutting tokens in ways that quietly wreck the output.

Instrument every call and attribute spend by team or feature. It is the same discipline FinOps teams already apply to cloud, and the AI token cost management approach treats tokens as a variable cost of goods sold rather than a fixed line item.

You do not need clever tooling to start. A simple dashboard will show you the waste inside a week.

The three highest-impact cost levers explained

Three tactics do most of the work. Get them right before you touch anything else.

1. Model routing. Send easy work to a cheap model and hard work to an expensive one. Two ways to build it:

• Rule-based routing. If the query matches a known pattern, a simple FAQ or a short classification, send it to your cheapest model. Quick to build. Works on predictable traffic.
• Classifier-based routing. A lightweight model decides where each task goes. Better once your traffic gets too varied for rules.

The strongest pattern is cheap model plus verifier. Let the cheap model answer, then run a quick check. If the verifier flags a problem, escalate. Academic work on quality-aware routing and token optimisation reports cuts of 40 to 90% on document-processing costs while quality holds or improves.

2. Caching. Three flavours, easiest first:

• Exact-match caching. Same input, same output, served instantly.
• Semantic caching. Catches near-duplicate questions worded differently. Much higher hit rates on support and FAQ work.
• Prefix or prompt-aware caching. Caches the stable part of your prompt so you only pay full price for the part that changes.

Claude gives you explicit prompt caching. Put the cache breakpoint right after your static prefix and the cached portion reprocesses at 0.1 times the normal input rate, roughly 90% off that part of every call. Support work is where that lands hardest, and we ran the cost-per-ticket numbers on it in the $1.84 support ticket.

3. Batching. Not every task needs an instant answer. Batch APIs run work asynchronously and knock roughly 50% off the cost of anything that can wait. Anthropic's own Message Batches API bills every token at half the standard rate, with most batches finishing inside an hour. Overnight reports, bulk classification and content queues all qualify.

Here is the bit most teams miss. Batching and caching stack.

Which mix wins depends on your traffic. Support bots with lots of repeat questions lean hardest on caching. Document pipelines lean on routing. Report generation and bulk content lean on batching. Most production systems end up running all three.

Secondary tactics: when trimming, quantising, and self-hosting pay off

These cost real engineering time. Leave them until routing, caching and batching are live.

Prompt trimming and compression. Cut filler from your system prompt. Summarise long history instead of resending it word for word. How much you save depends entirely on how bloated the prompt was, so measure your own before and after rather than trusting a headline number. The more dependable input-side win is caching, where providers discount cached input by 50 to 90% depending on the model and the tier.

Right-sizing and quantisation. Quantisation shrinks a model's precision so it runs smaller and cheaper. It suits classification, extraction and simple Q&A. Always run a quality check against your current model before you switch. A small accuracy drop on a high-volume task can cost more in rework than it saves.

Fine-tuning and distillation. Training a smaller model to copy a bigger one, or tuning a model on your own data, both carry real upfront cost in engineering time, data prep and compute. They pay off once volume is high and predictable, not before.

Self-hosting. Running the model on your own infrastructure. Only worth it once your monthly API spend consistently beats what it costs to run and monitor that infrastructure.

Tactic	Upfront effort	Typical break-even signal
Prompt trimming	Low	Immediate, no volume threshold
Right-sizing/quantisation	Medium	Steady, repetitive task volume
Fine-tuning/distillation	High	High, predictable monthly volume
Self-hosting	High	API spend exceeds infra + ops cost

How should you sequence LLM cost optimization over time?

Order matters. Run these in sequence.

Week 1:

1. Instrument every call. Log tokens, cost, latency, cache status.
2. Set quotas per feature or team so nobody can run away with spend.
3. Trim the obvious prompt fat.
4. Turn on provider-side caching wherever it already exists.

Weeks 2 to 8:

1. Build rule-based routing for your most predictable, highest-volume traffic.
2. Add exact-match caching for repeat queries.
3. Split batchable work into its own queue, away from real-time traffic.
4. Watch cache hit rate and cost per request weekly. If the hit rate stalls below 20%, your cache keys are wrong.

Month 3 and beyond:

1. Once volume is high and predictable, look at fine-tuning or distillation.
2. If monthly spend consistently beats infrastructure cost, take self-hosting seriously.
3. Re-run your routing rules. Pricing and quality shift fast, and Devwiz keeps a comparison of current LLM API pricing worth checking against your own mix. Last quarter's call may already be wrong.

Pro Tip: Provider caching and batch APIs are configuration changes, not rewrites. Start there. You get a return in days, before you touch anything as risky as self-hosting economics.

The thresholds are rough but useful. Move to routing once you are past a few thousand queries a day. Consider fine-tuning once volume is steady enough to amortise the training cost. Only self-host once your API bill has outgrown what a lean infrastructure team could run it for.

Why do you need a cost gateway or control layer?

Without one control point, every team picks its own models. Spend sprawls.

A gateway sits between your app and the model providers. It is where you enforce routing rules, caching and quotas once, instead of chasing them across ten codebases.

Think of a kitchen pass. Every dish leaves through one window, so the head chef catches the mistake before it reaches the table. No pass, and plates go straight from any station to any customer with nobody checking.

Build in:

• Circuit breakers. Halt or downgrade a feature automatically when its spend spikes past a threshold.
• Budget alerts. Real-time warnings as a team or feature approaches its cap.
• Per-feature cost attribution. Know which feature is burning the budget, not just the company total.

Consumption billing behaves like a variable manufacturing cost. Without breakers and quotas, spend runs away in days.

Ownership matters too. Someone owns the routing decisions and someone reviews them. A quarterly look at your default model and your routing rules catches the week a cheaper model caught up, or a provider quietly repriced.

How The AI Orchestrators build cost-efficient AI systems

We run one method: Explore, Map, Transform.

Explore finds where the founder's judgment actually sits in the business. Map turns that judgment into a written architecture: the roles, the decision rules, the handoffs. Transform builds it, with Claude Code, into an AI Operating System of AI employees that run real workflows.

Cost discipline is built into that Transform step, not bolted on after. Every AI employee gets a model tier assigned to its job, a cache boundary around its stable context, and a queue if its work can wait. That is routing, caching and batching, decided at the architecture stage rather than retrofitted once the bill arrives. Building it with Claude Code is what makes that possible for a founder who does not have an engineering team.

The businesses we work with are usually $1M+ education and consulting firms where the founder is the bottleneck. Cost matters there for a specific reason: every workflow we hand to an AI employee has to run cheaply enough that scaling it does not eat the margin. Njin makes the same argument from the hiring side in automate before you hire.

Here is a before and after any team can copy. Track cost per resolved client interaction before you touch routing or caching. Track the same number 30 days later. If both are working, it should drop by at least a third. If it has not, your sequencing is off, usually a cache that is not hitting or a router sending too much to the expensive model.

What actually separates teams that cut costs from those that don't?

Here is the honest bit. Most teams skip measurement and jump to the fun part, fine-tuning or self-hosting, because it feels like real engineering. That is backwards.

Fix the free stuff before you build the expensive stuff. Caching and routing are configuration. Fine-tuning and self-hosting are projects with real risk attached.

Starting with a new client, I would do two things on day one. Turn on provider caching and check the hit rate a week later. Then log cost per outcome instead of cost per call, because that is the number that tells you whether you are actually winning.

James Killick

Want help doing this properly, without the guesswork?

Most guides stop at "here is what to try". We build it with you.

The AI Orchestrators runs a 90-day program that maps how your business actually makes decisions, then builds the AI employees that run it. Routing, caching and monitoring go into the architecture, so the system your team uses every day is cheap to run by design.

It fits $1M+ education and consulting businesses where the founder is the single point of failure and the method needs to scale without the inference bill scaling with it. If that sounds like you, start with the assessment to see whether you are a fit. Want the terms first? Browse the AI orchestration glossary or read our original research.

Sources

• LLM cost optimization: 7 strategies to cut inference spend
• QC-Opt: quality-aware cost-optimised LLM routing and token optimisation (arXiv)
• AI token cost management for engineering teams (Faros AI)
• Message Batches API pricing (Anthropic docs)
• LLM Cost Optimization: How to Cut Spend 50-90% (LeanLM)