Human-in-the-loop (HITL) is not a single pattern but a spectrum of human involvement in AI-driven workflows. The right level of human involvement depends on the cost of errors, the maturity of the model, and the regulatory environment. Getting this balance wrong in either direction - too much human involvement (negating automation value) or too little (allowing unchecked errors) - is the most common failure mode in production AI systems.

Review Queue Pattern

The most common HITL implementation. The AI system processes inputs and produces outputs, and a human reviewer approves, rejects, or corrects each output before it takes effect.

When to use it - When errors have significant consequences (financial, legal, reputational) and the AI is not yet proven reliable enough for full automation. Document classification that drives regulatory filing, medical data extraction, and financial transaction categorization are typical review queue candidates.

Implementation - Route AI outputs to a review queue with the original input, the AI’s output, and a confidence score. Present them in a UI that makes review fast: show the source document alongside the extracted data, highlight fields with low confidence, and provide one-click approve/reject actions. Prioritize the queue by confidence: low-confidence items first (they are most likely to need correction) or high-value items first (errors are most costly).

The key metric - Track the correction rate (percentage of items where the reviewer changes the AI output). If correction rate is below 5%, you are probably over-reviewing. If it is above 30%, the model needs improvement before HITL is cost-effective.

Confidence Threshold Pattern

Instead of reviewing everything, only route items below a confidence threshold to human review. Items above the threshold are auto-approved.

Threshold calibration - Set the threshold by analyzing the relationship between confidence scores and actual accuracy. Plot confidence versus correction rate on historical data. Choose the threshold where auto-approved items would have had an acceptable error rate. For most business applications, a 2-3% error rate in the auto-approved stream is acceptable.

Dynamic thresholds - The threshold should not be static. As the model improves (from retraining on corrected data), the threshold can be lowered, reducing human review volume. Track this reduction as a key success metric.

Exception Handling Pattern

Some inputs are genuinely outside the model’s capability. Rather than producing a low-confidence output that wastes reviewer time, the system should recognize its limitations and route the entire input to a human for manual processing.

Detection strategies - Out-of-distribution detection identifies inputs that differ significantly from training data. This is more reliable than low confidence scores, which can be confidently wrong. Document type classifiers that encounter an unseen format, or extractors that find no recognizable fields, should route to exception handling rather than guessing.

Active Learning Pattern

Human corrections are not just quality control - they are training data. Each correction teaches the model something it got wrong, creating a feedback loop that improves model accuracy over time and reduces the human review burden.

Implementation - Store every human correction with the original input and the model’s output. Periodically retrain or fine-tune the model on corrected examples, weighted toward recent corrections. Measure model accuracy improvement per batch of corrections to track the learning rate.

Diminishing returns - Active learning produces rapid improvement initially and then plateaus. Track when correction rates stop declining to identify when additional human review is no longer improving the model and should be reduced to sampling-based quality monitoring.

Sampling Pattern

For high-volume, mature systems where full review is impractical, review a statistical sample of auto-approved outputs. This catches systematic drift without the cost of full review.

Sample sizing - The sample size depends on the acceptable detection latency for quality degradation. A 5% random sample catches a 10% increase in error rate within approximately 200 items. For most systems, reviewing 2-5% of outputs provides adequate quality monitoring.

Stratified sampling - Random sampling may miss errors concentrated in specific input types. Stratify the sample by input characteristics (document type, source, complexity) to ensure coverage across the full input distribution.

Escalation Pattern

Multi-tier review where straightforward corrections are handled by junior reviewers and complex or ambiguous cases are escalated to senior domain experts. This is cost-effective for systems that produce a wide range of output complexity.

Tier design - Tier 1 reviewers handle clear accept/reject decisions and simple corrections. Tier 2 handles ambiguous cases requiring domain expertise. Tier 3 handles cases that reveal systematic model failures requiring prompt or model changes. Each tier should have clear escalation criteria to prevent bottlenecks.