AI applications have unique rate limiting requirements. Model APIs impose their own limits (requests per minute, tokens per minute), costs scale with usage, and request processing times are orders of magnitude longer than traditional API calls. Effective rate limiting protects both your budget and your service quality.

Token-Based Rate Limiting

Traditional rate limiting counts requests. AI applications need to count tokens because a single request can consume vastly different amounts of capacity depending on input and output size.

Input token estimation - Before sending a request to the model, estimate its token count using a tokenizer. If the request would exceed the remaining token budget for the current window, queue or reject it. This prevents a single large request from consuming the entire token budget.

Output token budgeting - Output tokens are harder to predict because the model determines the output length. Set max_tokens per request based on the expected output size plus a buffer. Track actual output token usage and adjust estimates over time.

Combined limits - Enforce both request-per-minute and token-per-minute limits. A system might handle 100 small requests per minute but only 10 large ones. Both dimensions must be within limits for the request to proceed.

Adaptive Throttling

Static rate limits waste capacity during quiet periods and cause unnecessary rejections during temporary spikes.

Dynamic limits - Adjust rate limits based on current system load, model API response times, and error rates. When the model API starts responding slowly (indicating it is approaching capacity), reduce the request rate proactively rather than waiting for errors.

Backpressure - When the model API returns rate limit errors (429), implement exponential backoff and reduce the sending rate. Gradually increase the rate once errors stop. This automatically discovers and respects the actual available capacity.

Priority-based throttling - When approaching limits, throttle lower-priority requests before higher-priority ones. A customer-facing chatbot request should be served before a background batch processing request. Implement priority levels and shed load from the lowest priority first.

Queue Management

When requests arrive faster than they can be processed, a well-designed queue prevents both request loss and unbounded wait times.

Bounded queues - Set a maximum queue size. When the queue is full, reject new requests with a clear error message rather than queuing them for an unacceptable wait time. Users waiting 30 seconds is acceptable; users waiting 5 minutes is not.

Queue timeout - Set a maximum wait time for queued requests. If a request has been waiting longer than the timeout, remove it from the queue and return an error. The caller can retry or take alternative action.

Queue visibility - Provide queue depth and estimated wait time to callers. This enables callers to make informed decisions about waiting, retrying, or falling back to a cheaper model.

Cost Protection

Rate limiting serves as a cost protection mechanism in addition to its reliability function.

Budget caps - Set daily and monthly spending caps per application, per tenant, and per feature. When a cap is reached, reject new requests or route to cheaper alternatives. This prevents runaway costs from bugs, abuse, or unexpected traffic spikes.

Cost alerting - Alert when spending reaches 50%, 75%, and 90% of the budget cap. This gives operators time to investigate and adjust before the cap is hit.

Abuse detection - Monitor for usage patterns that indicate abuse: sudden volume spikes from a single user, requests with unusually large inputs, or repeated requests with identical content. Automatically throttle suspected abuse and alert for investigation.

Implementation Patterns

API Gateway integration - Implement rate limiting at the API gateway layer for centralized enforcement. This catches all traffic regardless of which application component originates the request. AWS API Gateway, Kong, and similar tools provide built-in rate limiting.

Distributed rate limiting - For multi-instance deployments, rate limits must be enforced across all instances collectively. Use a shared counter in Redis or a similar distributed data store. Local-only rate limiting allows each instance to independently reach the limit, exceeding the global limit by a factor of the instance count.