Customers frequently know what they want a product to look like but cannot describe it in words. “A blue dress like the one in that Instagram post” is a common shopping intent that text search cannot serve. Visual search enables customers to upload a photo, screenshot, or camera capture and find visually similar products in the retailer’s catalog. Retailers with visual search report 2-4x higher conversion rates on visual search sessions compared to text search.

The Problem

Text-based product search depends on product descriptions, titles, and tags that may not match customer vocabulary. A customer searching for “boho maxi dress” may not find a product tagged as “long bohemian sundress.” Visual attributes like pattern, silhouette, texture, and color combination are difficult to express in words but instantly recognizable in images.

Fashion, home decor, and furniture are the most natural categories for visual search, but the use case extends to any product category where visual similarity is a meaningful dimension of product discovery: auto parts, building materials, art supplies, and collectibles.

AI Approach

Feature extraction - SageMaker hosts a convolutional neural network (typically ResNet or EfficientNet) fine-tuned on the retailer’s product catalog. The model extracts a high-dimensional feature vector from each product image that captures visual attributes: color, pattern, shape, texture, and style. These vectors are computed offline for the entire catalog and stored for similarity search.

Similarity search - Amazon OpenSearch with k-nearest-neighbor (kNN) search indexes the product feature vectors. When a customer uploads a query image, the same model extracts its feature vector, and OpenSearch returns the most visually similar products from the catalog. Search latency is typically under 200ms even for catalogs with millions of products.

Object detection and cropping - Query images often contain multiple objects, backgrounds, or irrelevant context. Amazon Rekognition detects and isolates the primary product in the image before feature extraction. For fashion, this means detecting the garment and excluding the model, background, and accessories. For home decor, it means isolating the specific item from the room context.

Attribute extraction - Beyond visual similarity, the system extracts searchable attributes from images: dominant colors, pattern types (stripes, floral, solid), material appearance (leather, cotton, wood), and style categories. These attributes enable hybrid search: visual similarity filtered by specific attributes (“similar but in red”).

Architecture

Product catalog images are processed through the feature extraction pipeline on SageMaker and indexed in OpenSearch. Customer query images are uploaded to S3 via API Gateway, processed through Rekognition for object detection, then through SageMaker for feature extraction. OpenSearch returns the top-N similar products. Results are filtered by availability, size, and business rules before presentation. The entire pipeline executes in under 500ms.

Key Considerations

Catalog image quality - Visual search accuracy depends on consistent, high-quality catalog images. Products photographed on white backgrounds with standardized lighting produce better features than lifestyle images with complex backgrounds. A preprocessing pipeline that normalizes catalog images improves search quality.

Cross-category search - A query image of a patterned cushion might return similar patterns across categories (curtains, rugs, clothing). This can be desirable for inspiration-driven shopping or undesirable for intent-driven shopping. Category filtering should be available to the customer.

Feedback loop - Customer interactions with visual search results (clicks, purchases, skips) provide implicit feedback for model fine-tuning. Products that are visually similar according to the model but never purchased together may indicate a feature mismatch that fine-tuning can correct.

Mobile optimization - Visual search is primarily a mobile use case. Camera integration, image compression, and fast upload paths are essential for good mobile UX.

Next Steps

Build the product catalog feature index using a pretrained vision model fine-tuned on the product image dataset. Deploy a visual search prototype for the highest-impact category (typically fashion or home decor). A/B test against text-only search to measure conversion lift. Iterate on the model based on click-through and purchase data from search results.