Batch vs Real-Time Inference Patterns

ML models can serve predictions in two modes: batch (process a dataset at once) and real-time (respond to individual requests on demand). The choice affects infrastructure, cost, latency, and system architecture. Many production systems use both modes for different parts of their prediction pipeline.

Overview

Batch Inference

Batch inference processes a dataset through a model in a single job. You provide input data (typically in Amazon S3, a database, or a data lake), the job applies the model to every record, and results are written back to storage. Amazon SageMaker AI Batch Transform, Spark ML inference, and custom scripts on Amazon EMR are common batch inference patterns on AWS. (Amazon renamed the core ML platform from Amazon SageMaker to Amazon SageMaker AI at re:Invent 2024; the inference options described here are unchanged.)

Batch is most cost-effective because compute resources run only during processing. GPU instances process data at maximum throughput without idle time. You can use spot instances for further savings. Processing 10 million predictions overnight in batch costs a fraction of maintaining a real-time endpoint to serve those same predictions over the course of a day.

Real-Time Inference

Real-time inference deploys a model behind an endpoint that responds to individual prediction requests. SageMaker AI real-time endpoints, custom APIs on Amazon ECS or Amazon EKS, and serverless options (AWS Lambda with model artifacts, or SageMaker AI Serverless Inference) are common patterns. The model is loaded in memory and responds to requests with low latency. SageMaker AI real-time endpoints accept request payloads up to 25 MB, while the fully managed Serverless Inference option (which scales to zero between traffic spikes but adds cold starts) accepts payloads up to 4 MB.

Real-time inference is necessary when predictions must reflect current state, when predictions are triggered by user actions, or when the input data is not known in advance. Product recommendations on a website, fraud detection for transactions, and chatbot responses all require real-time inference.

Near Real-Time: The Middle Ground

Many use cases do not need sub-second latency but cannot tolerate hours of staleness. Near real-time patterns process micro-batches every few minutes. Streaming inference with Amazon Kinesis or Apache Kafka processes events in small batches with seconds to minutes of latency. This provides a cost-effective middle ground.

Architecture Patterns

Pre-compute and cache: Run batch inference on all likely inputs and cache results. Serve predictions from cache at request time. This gives real-time latency with batch economics but only works when the input space is bounded and predictable.

Batch + real-time hybrid: Use batch inference for baseline predictions (updated nightly) and real-time inference for adjustments based on current context. Recommendation systems often use this pattern - batch-computed candidate lists refined by real-time scoring.

Asynchronous inference: Accept prediction requests, queue them, and return results when ready. Amazon SageMaker AI Asynchronous Inference handles this pattern, autoscaling the instance count to zero when there are no requests (so you pay only while requests are processing). It is built for large payloads (up to 1 GB) and long-running models (processing times up to one hour), which makes it a good fit between real-time endpoints and offline batch jobs. This works when callers can tolerate seconds to minutes of latency.

Batch vs Real-Time for LLMs and Foundation Models

The same trade-off now appears at the foundation-model layer, where the economics are explicit. Hosted model providers offer asynchronous batch endpoints at a discount to their synchronous (real-time) APIs:

• OpenAI Batch API - submit a file of requests and receive results within 24 hours (often sooner) at a 50% discount versus synchronous calls, with a separate, higher rate-limit pool.
• Anthropic Message Batches API - asynchronously process large volumes of requests at a 50% discount on input and output tokens, with results available when the batch completes or after 24 hours, whichever comes first.
• Amazon Bedrock batch inference - submit large prompt datasets in Amazon S3 and retrieve results as a managed job. Select foundation models run at 50% of on-demand inference pricing.

The decision rule is the same as for self-hosted models: if the work is not user-facing and can tolerate up to a day of latency (evaluations, bulk classification, embedding generation, document processing), batch roughly halves the token bill. Reserve the real-time API for interactive features like chat and live search.

Cost Comparison

For a model serving 1 million predictions per day, batch inference on a spot GPU instance might cost a few dollars. The same volume through a real-time endpoint with an always-on GPU instance costs significantly more. The real-time premium is the cost of low latency and immediate availability.

SageMaker AI Serverless Inference and auto-scaling endpoints reduce real-time costs for variable traffic but still cost more than batch for predictable workloads.

When to Choose Batch

Choose batch inference when predictions are needed on a schedule rather than on demand, when the input dataset is known in advance, when latency of minutes to hours is acceptable, when cost optimization is a priority, or when you are scoring large datasets for analytics or reporting.

When to Choose Real-Time

Choose real-time inference when predictions must be immediate (user-facing interactions), when inputs are not known until request time, when predictions depend on current state (fraud detection, pricing), or when the application architecture is request-response.

Practical Recommendation

Default to batch unless you have a clear latency requirement. Many teams deploy real-time endpoints for workloads that could be served more efficiently with batch inference and caching. Start with batch, measure whether the prediction staleness is acceptable, and move to real-time only for the predictions that genuinely need it. For the platform choice underneath these patterns, see /comparisons/sagemaker-vs-bedrock/; for the serverless compute trade-offs behind real-time endpoints, see /comparisons/lambda-vs-fargate-ai/.

Sources

• Deploy models for inference - Amazon SageMaker AI Developer Guide
• Asynchronous inference - Amazon SageMaker AI Developer Guide
• Batch transform for inference - Amazon SageMaker AI Developer Guide
• Amazon Bedrock offers select FMs for batch inference at 50% of on-demand price
• OpenAI Batch API guide
• Anthropic Message Batches API (batch processing)