LLM inference costs add up fast. A customer-facing application processing thousands of requests per hour can easily generate six-figure monthly bills. The good news is that most LLM deployments have significant optimization opportunities. The key is reducing cost without degrading the quality your users experience.

Measure Before Optimizing

Before cutting costs, instrument your system to understand where money goes. Track per-request costs by breaking down input tokens, output tokens, and model used. Identify your most expensive use cases, your highest-volume endpoints, and your average prompt size. Optimization without measurement is guesswork.

Prompt Optimization

Reduce prompt length. Every token costs money. Audit your system prompts for redundancy, excessive examples, and unnecessary instructions. A system prompt that shrinks from 2000 tokens to 800 tokens saves 60% on input costs for every request.

Optimize few-shot examples. If you include examples in your prompts, test whether fewer examples achieve the same quality. Often two well-chosen examples work as well as five.

Compress retrieved context. In RAG systems, the retrieved documents dominate input token count. Summarize or extract relevant passages rather than including entire documents. Rank by relevance and include only the top results.

Caching

Exact match caching. Cache responses for identical prompts. This is most effective for applications with repetitive queries: FAQ bots, document classification, and template-based generation. Even a modest cache hit rate dramatically reduces costs.

Semantic caching. Cache responses for semantically similar prompts. When a new query is sufficiently similar to a cached query, return the cached response. This requires careful threshold tuning to balance cache hits against response quality.

KV cache reuse. For self-hosted models, reuse key-value caches across requests that share prompt prefixes. This reduces computation for the shared portion. Effective when many requests use the same system prompt.

Model Routing

Not every request needs your most capable (and expensive) model. Implement a router that directs requests to the appropriate model based on complexity:

Simple requests (classification, extraction, short answers) go to smaller, cheaper models. A well-prompted small model often matches a large model on straightforward tasks.

Complex requests (multi-step reasoning, creative generation, nuanced analysis) go to the most capable model available.

Router implementation. Start with rule-based routing (task type, input length) and evolve to a classifier that estimates task difficulty. Monitor quality by model tier to ensure routing decisions are not degrading user experience.

Batching and Async Processing

Batch non-urgent requests. If requests do not need real-time responses, batch them and process during off-peak hours or with batch API pricing, which is typically 50% cheaper.

Parallelize where possible. For complex tasks that involve multiple LLM calls (planning, critique, refinement), evaluate whether all steps are necessary. Sometimes a single well-crafted prompt replaces a multi-step chain.

Output Optimization

Limit output tokens. Set max_tokens to the minimum needed for your use case. A classification task does not need 4096 output tokens.

Use structured output. JSON mode or function calling produces more concise outputs than free-form text, reducing output token costs.

Stop sequences. Configure stop sequences to end generation as soon as the useful output is complete, rather than letting the model generate unnecessary closing text.

Infrastructure Optimization

For self-hosted models:

  • Right-size GPU instances. Profile your model’s actual VRAM and compute requirements rather than over-provisioning.
  • Use quantized models. 4-bit or 8-bit quantization reduces memory requirements and increases throughput with minimal quality loss for most applications.
  • Autoscale aggressively. Scale down during low-traffic periods. LLM serving infrastructure is expensive when idle.
  • Consider spot/preemptible instances for batch processing workloads that can tolerate interruption.

Monitoring Cost Continuously

Add cost tracking to your observability stack. Alert on cost anomalies (a new feature that unexpectedly triples token usage) and track cost per user action as a key metric alongside latency and quality.