Performance engineering for AI systems differs fundamentally from traditional software optimization. In conventional systems, bottlenecks are typically CPU or I/O bound. In AI systems, the interplay between model size, GPU memory, batch size, and numerical precision creates a multidimensional optimization space that demands specialized techniques.

Origins and History

Performance engineering as a discipline traces back to capacity planning in mainframe computing, but its application to AI systems accelerated with the rise of deep learning. Google’s 2017 paper “Attention Is All You Need” (Vaswani et al.) introduced the Transformer architecture, which created new performance challenges due to quadratic attention complexity [1]. NVIDIA’s development of TensorRT in 2017 and Microsoft’s release of ONNX Runtime in 2019 established the modern toolkit for AI inference optimization [2][3]. The shift from research to production AI forced engineering teams to treat latency percentiles with the same rigor as traditional web services.

Latency Optimization

Measure latency at P50, P95, and P99 percentiles, not averages. A system with 50ms average latency may have a P99 of 800ms, creating unacceptable user experiences for one in a hundred requests.

GPU memory management. GPU out-of-memory errors are the most common production failure. Profile memory usage with nvidia-smi and PyTorch’s torch.cuda.memory_summary(). Use gradient checkpointing during training and KV-cache optimization during inference. PagedAttention, introduced by vLLM, reduces memory waste from 60-80% to near zero by borrowing virtual memory concepts from operating systems [4].

Batch size tuning. Larger batches improve throughput but increase latency per request. Start with batch size 1 for latency-sensitive applications and increase until P99 latency exceeds your SLA. Dynamic batching frameworks collect incoming requests over a short window (typically 5-50ms) and process them together.

Model quantization. Reduce precision from FP32 to FP16, INT8, or INT4. FP16 halves memory usage with negligible quality loss for most models. INT8 quantization via GPTQ or AWQ can reduce model size by 4x while retaining 95-99% of original quality. Always benchmark quantized models against your specific evaluation dataset.

ONNX Runtime and TensorRT. Convert models to ONNX format for cross-platform optimization. ONNX Runtime applies graph optimizations (operator fusion, constant folding) automatically. For NVIDIA GPUs, TensorRT provides additional kernel auto-tuning and can deliver 2-6x speedups over unoptimized PyTorch inference [3].

Throughput Engineering

When total requests per second matters more than individual request latency, shift to throughput-oriented strategies.

Horizontal scaling. Deploy multiple model replicas behind a load balancer. Use Kubernetes with GPU-aware scheduling to bin-pack replicas efficiently. Separate GPU nodes by capability (A10G for inference, A100 for training) to avoid resource contention.

Request batching. Server-side batching frameworks like Triton Inference Server and vLLM collect requests into batches transparently. Continuous batching (iteration-level scheduling) processes new requests as existing ones complete individual decode steps, improving GPU utilization from 30-50% to 80-95%.

Async inference. For non-interactive workloads, decouple request submission from result retrieval. Use message queues (SQS, Kafka) to buffer requests and process them at optimal batch sizes. Return results via webhooks or polling endpoints. This pattern absorbs traffic spikes without over-provisioning GPU capacity.

Putting It Together

Start with profiling. Identify whether your bottleneck is compute-bound (GPU utilization near 100%), memory-bound (large models, small batches), or I/O-bound (data loading, network). Apply optimizations in order of impact: quantization first, then batching, then infrastructure scaling. Measure after every change. Optimization without measurement is guesswork.

Sources

  1. Vaswani, A., et al. “Attention Is All You Need.” Advances in Neural Information Processing Systems 30 (NeurIPS 2017).
  2. Microsoft. “ONNX Runtime: Cross-platform, high performance ML inferencing and training accelerator.” GitHub, 2019. https://github.com/microsoft/onnxruntime
  3. NVIDIA. “TensorRT: Programmable Inference Accelerator.” NVIDIA Developer, 2017. https://developer.nvidia.com/tensorrt
  4. Kwon, W., et al. “Efficient Memory Management for Large Language Model Serving with PagedAttention.” Proceedings of the 29th Symposium on Operating Systems Principles (SOSP 2023).