Image classification assigns labels to images. Modern approaches range from dedicated computer vision models (Amazon Rekognition) to multi-modal LLMs that can reason about image content. The choice depends on the classification task’s specificity, volume, and accuracy requirements.

Classification Approaches

Pre-built classification services - Amazon Rekognition, Google Vision, and similar services provide pre-trained classifiers for common categories: objects, scenes, faces, text, and content moderation. No training required. Best for standard classification tasks where the pre-built categories match your needs.

Custom labels - Services like Amazon Rekognition Custom Labels let you train classifiers for your specific categories using your own labeled images. You provide 50-250 labeled images per category, and the service trains and hosts a custom model. Best for domain-specific classification (defect detection, product categorization, medical imaging) where pre-built categories are insufficient.

Multi-modal LLM classification - Send images to a vision-capable LLM with classification instructions. The model can handle nuanced, context-dependent classification that pre-built services cannot: “Is this product photo suitable for the homepage?” or “Does this document appear to be a signed contract?” Best for complex classification requiring judgment or context.

Confidence and Thresholds

Multi-class confidence - Image classifiers typically return scores for multiple categories. Interpret the score distribution, not just the top category. A confident classification has a high top score and low scores for alternatives. An uncertain classification has similar scores across multiple categories.

Threshold per category - Different categories may need different confidence thresholds. A content moderation classifier should have a low threshold for flagging potentially unsafe content (high recall) and a higher threshold for auto-removing content (high precision).

Rejection class - Include an explicit “unknown” or “other” category for images that do not match any defined class. Without this, every image is forced into an existing category regardless of fit.

Pre-Processing

Image normalization - Resize images to the model’s expected input dimensions. Normalize brightness and contrast for consistency. Convert formats as needed (PNG, JPEG). For batch processing, standardize all images to a consistent format and resolution.

Quality filtering - Detect and reject images that are too blurry, too dark, too small, or corrupted before sending to the classifier. These produce unreliable results and waste API costs. A simple quality check (file size, resolution, format validation) catches most issues.

Orientation correction - Images may be rotated or mirrored. Auto-detect and correct orientation using EXIF data or an orientation detection model before classification.

Production Patterns

Batch vs. real-time - Use real-time classification for user-facing features (content upload moderation, search). Use batch classification for processing existing image libraries, periodic quality audits, and reprocessing after model updates.

Caching - Cache classification results by image hash. The same image uploaded multiple times does not need reclassification. This reduces costs for systems where image duplication is common.

Human review integration - For high-stakes classifications, route low-confidence results to human reviewers. Build the review interface to show the image, the model’s classification with confidence scores, and relevant examples from each candidate category to help reviewers decide quickly.

Model versioning - When updating the classification model, re-classify a sample of historical images and compare results against previous classifications. Flag any systematic changes (a category that suddenly gets more or fewer assignments) for investigation before rolling out the new model broadly.