LLM inference costs add up fast. A customer-facing application processing thousands of requests per hour can easily generate six-figure monthly bills. The good news is that most LLM deployments have significant optimization opportunities. The key is reducing cost without degrading the quality your users experience.

Measure Before Optimizing

Before cutting costs, instrument your system to understand where money goes. Track per-request costs by breaking down input tokens, output tokens, and model used. Identify your most expensive use cases, your highest-volume endpoints, and your average prompt size. Optimization without measurement is guesswork.

Prompt Optimization

Reduce prompt length. Every token costs money. Audit your system prompts for redundancy, excessive examples, and unnecessary instructions. A system prompt that shrinks from 2000 tokens to 800 tokens saves 60% on input costs for every request.

Optimize few-shot examples. If you include examples in your prompts, test whether fewer examples achieve the same quality. Often two well-chosen examples work as well as five.

Compress retrieved context. In RAG systems, the retrieved documents dominate input token count. Summarize or extract relevant passages rather than including entire documents. Rank by relevance and include only the top results.

Caching

Exact match caching. Cache responses for identical prompts. This is most effective for applications with repetitive queries: FAQ bots, document classification, and template-based generation. Even a modest cache hit rate dramatically reduces costs.

Semantic caching. Cache responses for semantically similar prompts. When a new query is sufficiently similar to a cached query, return the cached response. This requires careful threshold tuning to balance cache hits against response quality.

Provider prompt caching. Hosted APIs cache repeated prompt prefixes for you and bill cache hits at a steep discount: Anthropic charges 10% of the base input price for a cache read on most Claude models (5% on Claude Opus 5.5), and OpenAI discounts cached input tokens as well. Put stable content (system prompt, tool definitions, reference documents) at the start of the prompt and keep it byte-identical between calls so the cache hits.

KV cache reuse. For self-hosted models, reuse key-value caches across requests that share prompt prefixes. This reduces computation for the shared portion. Effective when many requests use the same system prompt.

Model Routing

Not every request needs your most capable (and expensive) model. Implement a router that directs requests to the appropriate model based on complexity:

Simple requests (classification, extraction, short answers) go to smaller, cheaper models. A well-prompted small model often matches a large model on straightforward tasks.

Complex requests (multi-step reasoning, creative generation, nuanced analysis) go to the most capable model available.

Router implementation. Start with rule-based routing (task type, input length) and evolve to a classifier that estimates task difficulty. Monitor quality by model tier to ensure routing decisions are not degrading user experience.

Batching and Async Processing

Batch non-urgent requests. If requests do not need real-time responses, batch them and process during off-peak hours or with batch API pricing, which is typically 50% cheaper.

Parallelize where possible. For complex tasks that involve multiple LLM calls (planning, critique, refinement), evaluate whether all steps are necessary. Sometimes a single well-crafted prompt replaces a multi-step chain.

Output Optimization

Limit output tokens. Set the output limit to the minimum needed for your use case (max_tokens on Anthropic’s Messages API; max_completion_tokens or max_output_tokens on OpenAI’s reasoning models). A classification task does not need 4096 output tokens.

Control reasoning effort. Most current frontier models reason before they answer, and those hidden reasoning tokens are billed as output tokens. Use the lowest effort or thinking setting that passes your evaluations for each route; a high setting on a simple extraction task can multiply output cost without improving the answer.

Use structured output. JSON mode or function calling produces more concise outputs than free-form text, reducing output token costs.

Stop sequences. Configure stop sequences to end generation as soon as the useful output is complete, rather than letting the model generate unnecessary closing text.

Infrastructure Optimization

For self-hosted models:

  • Right-size GPU instances. Profile your model’s actual VRAM and compute requirements rather than over-provisioning.
  • Use quantized models. 4-bit or 8-bit quantization reduces memory requirements and increases throughput with minimal quality loss for most applications.
  • Autoscale aggressively. Scale down during low-traffic periods. LLM serving infrastructure is expensive when idle.
  • Consider spot/preemptible instances for batch processing workloads that can tolerate interruption.

Monitoring Cost Continuously

Add cost tracking to your observability stack. Alert on cost anomalies (a new feature that unexpectedly triples token usage) and track cost per user action as a key metric alongside latency and quality.

Sources

  1. Anthropic, “Pricing” (prompt caching multipliers; fetched 25 September 2026): https://platform.claude.com/docs/en/about-claude/pricing
  2. OpenAI, “Reasoning models” guide (reasoning tokens billed as output tokens): https://developers.openai.com/api/docs/guides/reasoning