Amazon EMR (Elastic MapReduce) is a managed big data platform that runs Apache Spark, Hadoop, Hive, Presto, and other open-source frameworks on scalable clusters of EC2 instances or on EKS containers. For AI projects, EMR is the workhorse for large-scale data processing tasks that exceed what Lambda, Glue, or single-machine tools can handle: transforming terabytes of raw data into training datasets, computing features across billions of records, and running distributed ML algorithms.

Official documentation: https://docs.aws.amazon.com/emr/ Pricing: https://aws.amazon.com/emr/pricing/ Service quotas: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-limits.html

Core Concepts

Cluster - A collection of EC2 instances running the selected frameworks. A cluster has three node types: a primary node (coordinates the cluster), core nodes (store data in HDFS and run tasks), and task nodes (run tasks only, no storage, can be spot instances for cost savings).

EMR on EKS - Run Spark workloads on existing EKS clusters. This eliminates the need for separate EMR clusters and allows sharing Kubernetes infrastructure between ML training, inference, and data processing workloads.

EMR Serverless - A serverless option where you submit Spark or Hive jobs without provisioning clusters. EMR Serverless automatically provisions, scales, and decommissions resources. Best for intermittent batch jobs where maintaining a cluster is wasteful.

EMR Studio - A managed Jupyter notebook environment for interactive data exploration and development. Notebooks connect to EMR clusters and support PySpark, SparkR, and SparkSQL kernels.

Spark for ML Feature Engineering

Apache Spark on EMR is the standard tool for large-scale feature engineering. Spark DataFrames and SQL handle the transformations that prepare raw data for model training:

Aggregation features - Compute statistics (mean, sum, count, percentiles) grouped by entity and time window. For example, calculate each customer’s average order value, purchase frequency, and return rate over the last 30, 90, and 365 days.

Join enrichment - Join event data with reference data (product catalogs, customer profiles, geographic data) to create enriched feature vectors. Spark handles joins across datasets with billions of rows that would be impractical in single-machine tools.

Text and sequence features - Use Spark’s MLlib for TF-IDF, word2vec, and n-gram computation across large text corpora. For sequence features, window functions compute ordered statistics (time between events, sequence patterns).

EMR Serverless for AI Workloads

EMR Serverless is increasingly the default choice for ML data processing because it eliminates cluster management entirely. Submit a Spark job with your application code, dependencies, and configuration. EMR Serverless provisions workers, runs the job, and releases resources when done. You pay only for the compute consumed.

The trade-off is less control: you cannot SSH into nodes, install custom packages at the OS level, or use HDFS for intermediate storage. For most feature engineering and data transformation workloads, these limitations do not matter.

Integration with SageMaker

The standard pipeline connects EMR data processing to SageMaker model training. EMR reads raw data from S3, transforms it into training-ready features, and writes the output back to S3 in a format SageMaker can consume (CSV, Parquet, RecordIO). SageMaker training jobs then read directly from S3.

For iterative development, EMR Studio notebooks and SageMaker Studio notebooks can access the same S3 data, allowing data engineers and data scientists to work with the same datasets in their preferred environments.

Cost Optimization

Spot instances for task nodes can reduce compute costs by 60-90%. Task nodes are stateless and can tolerate interruption. EMR automatically handles spot interruptions by reassigning tasks to remaining nodes.

Graviton instances (ARM-based) offer better price-performance for Spark workloads. EMR supports Graviton across all node types with no code changes required.

Auto-scaling adjusts the number of task nodes based on YARN metrics. Configure scale-up when pending containers exceed a threshold and scale-down when idle capacity persists.

Right-size clusters by analyzing Spark UI metrics from previous runs. Over-provisioned clusters waste money on idle resources. Under-provisioned clusters waste money on longer runtimes.

Pricing

EMR charges an EMR fee on top of the underlying EC2 instance costs. The EMR fee is approximately 25% of the on-demand EC2 price. For EMR Serverless, pricing is per vCPU-hour and per GB-hour of memory consumed during job execution. Compare the cost of a persistent cluster (for continuous workloads) against serverless (for intermittent batch jobs) to determine the cheaper option for your usage pattern.