Reducing LLM costs without hurting quality
By TechlyUpUpdated 2 min readDevelopers and engineering leads
Quick answer
LLM costs are driven mainly by model choice and token volume. Measure quality on your evaluation set, then try the smallest model that passes, trim prompts and context, cache repeated work, batch offline jobs, and route easy requests to cheaper models. Change one lever at a time and confirm quality holds.
Measure first
Track tokens in and out, requests, and cost per feature. Without per-feature numbers, you can't tell where savings are.
The main levers
Apply in roughly this order.
- Model choice: test smaller models on your eval set.
- Prompt size: remove redundant instructions and examples.
- Context size: retrieve fewer, better passages.
- Caching: reuse results for identical or repeated requests, and use provider prompt caching where available.
- Batching: run non-urgent jobs in batch modes where offered.
- Routing: send simple requests to cheaper models, hard ones to stronger ones.
Protect quality
Re-run your evaluation after each change. A cheaper model that causes more retries or human corrections may cost more overall.
Set guardrails on spend
Use per-user and per-feature limits, alerts on unusual usage, and maximum output lengths to prevent runaway costs.
Cost-cutting mistakes
Some savings cost more than they save.
- Switching to a cheaper model without re-running evaluations.
- Cutting context that was essential for accuracy.
- Caching responses that should vary per user or over time.
- Optimising a feature that is a tiny share of total spend.
Worked example: trimming a support assistant's costs
A team finds most spend comes from a support assistant resending full conversation histories. They summarise older turns, move stable instructions to the start of the prompt to benefit from prompt caching, and route simple FAQ-type questions to a smaller model.
They run their evaluation set after each change. Quality holds on all but the routing change, which misroutes some complex questions; they tighten the routing rules before keeping it. Each change is kept only when its effect is measured.
Try it yourself
For one feature, record baseline cost and quality, then test one smaller model and one prompt trim. Keep the change only if quality holds.
Frequently asked questions
Are open-source models cheaper?
They can be, depending on hosting and scale, but include infrastructure and operations costs in the comparison.
Does prompt caching reduce cost?
Many providers discount repeated prompt prefixes. Structure prompts so stable parts come first.
What's a common hidden cost?
Long conversation histories resent on every turn; summarise or trim history.
Want a suggested next step for your situation?
Share a few details and someone from TechlyUp will get back to you. No automated sequences.
Sources and further reading
Examples are authored practice material, not measured learner outcomes. Tool behavior can change. Found an error? Contact TechlyUp with the page URL and correction.