Amazon EMR (Elastic MapReduce) is a managed big data platform that runs Apache Spark, Hadoop, Hive, Presto, and other open-source frameworks on scalable clusters of EC2 instances, on EKS containers, or as a serverless service. For AI projects, EMR is the workhorse for large-scale data processing tasks that exceed what Lambda, Glue, or single-machine tools can handle: transforming terabytes of raw data into training datasets, computing features across billions of records, and running distributed ML algorithms.

Official documentation: https://docs.aws.amazon.com/emr/ Pricing: https://aws.amazon.com/emr/pricing/ Service quotas: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-limits.html

Beginner foundations

Before the specifics, a few plain-language ideas make EMR easier to understand.

Big data processing - When a dataset is too large to open in a spreadsheet or process on one computer, the work is split across many machines that each handle a slice in parallel, then the results are combined. This is the core idea behind frameworks like Apache Spark and the older Hadoop MapReduce model that gave EMR its name.

Managed service - You do not install, patch, or babysit the underlying software. AWS provisions the machines, installs the frameworks, and handles failures. You bring your code and data and pay for what you use.

Cluster - A group of computers working together as one. EMR can spin up a cluster of EC2 instances, run your job, and shut the cluster down so you stop paying.

Useful background reading: /tools/apache-spark/ (the engine most EMR jobs run on), /glossary/data-lake/ (where the raw data usually lives, normally in /tools/aws-s3/), /glossary/etl/ (extract, transform, load, the kind of work EMR does), and /glossary/serverless/ (the pay-per-job model behind EMR Serverless).

Core Concepts

Cluster - A collection of EC2 instances running the selected frameworks. A cluster has three node types: a primary node (coordinates the cluster), core nodes (store data in HDFS and run tasks), and task nodes (run tasks only, no storage, can be spot instances for cost savings).

EMR on EKS - Run Spark workloads on existing EKS clusters. This eliminates the need for separate EMR clusters and allows sharing Kubernetes infrastructure between ML training, inference, and data processing workloads.

EMR Serverless - A serverless option where you submit Spark or Hive jobs without provisioning clusters. EMR Serverless automatically provisions, scales, and decommissions resources. Best for intermittent batch jobs where maintaining a cluster is wasteful.

EMR Studio - A managed, web-based integrated development environment for interactive data exploration and development. Notebooks (delivered as EMR Studio Workspaces, which replace the older standalone EMR Notebooks) connect to EMR clusters, EMR on EKS, or EMR Serverless and support PySpark, SparkR, and SparkSQL kernels, plus the Spark UI and YARN Timeline Service for debugging.

Release version - EMR ships curated bundles of open-source frameworks under a version string. The current generation is built around Apache Spark 4.0, which AWS made generally available across EMR on EC2, EMR on EKS, and EMR Serverless on 9 June 2026 (the AWS Spark runtime release emr-spark-8.0). Spark 4.0 adds Spark Connect, the VARIANT data type for semi-structured JSON, SQL scripting, the Python Data Source API, and streaming improvements.

Spark for ML Feature Engineering

Apache Spark on EMR is the standard tool for large-scale feature engineering. Spark DataFrames and SQL handle the transformations that prepare raw data for model training:

Aggregation features - Compute statistics (mean, sum, count, percentiles) grouped by entity and time window. For example, calculate each customer’s average order value, purchase frequency, and return rate over the last 30, 90, and 365 days.

Join enrichment - Join event data with reference data (product catalogs, customer profiles, geographic data) to create enriched feature vectors. Spark handles joins across datasets with billions of rows that would be impractical in single-machine tools.

Text and sequence features - Use Spark’s MLlib for TF-IDF, word2vec, and n-gram computation across large text corpora. For sequence features, window functions compute ordered statistics (time between events, sequence patterns).

The features EMR computes are often published to a /glossary/feature-store/ so that training and serving read the same definitions, which is a common source of bugs when the two paths drift apart. For background on what this stage is doing, see /glossary/etl/ and the broader /glossary/mlops/ practice.

EMR Serverless for AI Workloads

EMR Serverless is increasingly the default choice for ML data processing because it eliminates cluster management entirely. Submit a Spark job with your application code, dependencies, and configuration. EMR Serverless provisions workers, runs the job, and releases resources when done. You pay only for the compute consumed.

The trade-off is less control: you cannot SSH into nodes, install custom packages at the OS level, or use HDFS for intermediate storage. For most feature engineering and data transformation workloads, these limitations do not matter.

Two recent additions narrow that gap. EMR Serverless storage (EMR releases 7.12 and higher) removes the need to provision local disk yourself: AWS manages shuffle, disk spill, and disk caching automatically, which simplifies sizing and can lower cost for shuffle-heavy jobs. Spark Connect (EMR releases 7.13 and higher, Apache Spark 3.5.6 and later) lets you drive a remote EMR Serverless Spark application from a local IDE or notebook over a thin client, so interactive PySpark development no longer requires a long-lived cluster. As of June 2026, Amazon SageMaker Unified Studio notebooks can run PySpark and Spark SQL directly against an EMR Serverless application using Spark Connect.

Integration with SageMaker

The standard pipeline connects EMR data processing to SageMaker model training. EMR reads raw data from S3, transforms it into training-ready features, and writes the output back to S3 in a format SageMaker can consume (CSV, Parquet, RecordIO). SageMaker training jobs then read directly from S3.

For iterative development, EMR Studio notebooks and SageMaker Studio notebooks can access the same S3 data, allowing data engineers and data scientists to work with the same datasets in their preferred environments.

Cost Optimization

Spot instances for task nodes can cut compute costs significantly (AWS prices EC2 Spot at up to 90% off On-Demand). Task nodes are stateless and can tolerate interruption. EMR automatically handles spot interruptions by reassigning tasks to remaining nodes.

Graviton instances (ARM-based) offer better price-performance for Spark workloads. EMR supports Graviton across all node types with no code changes required.

Auto-scaling adjusts the number of task nodes based on YARN metrics. Configure scale-up when pending containers exceed a threshold and scale-down when idle capacity persists.

Right-size clusters by analyzing Spark UI metrics from previous runs. Over-provisioned clusters waste money on idle resources. Under-provisioned clusters waste money on longer runtimes.

Pricing

For EMR on EC2, AWS charges an EMR fee on top of the underlying EC2 (and EBS) instance costs. The fee varies by instance type but is typically in the region of 25 percent of the On-Demand EC2 price, billed per second with a one-minute minimum. EMR on EKS charges a smaller per-vCPU-hour and per-GB-hour uplift on top of your EKS compute. For EMR Serverless, pricing is per vCPU-hour and per GB-hour of memory consumed during job execution (for example, on Linux x86 as of June 2026, 0.052624 USD per vCPU-hour and 0.0057785 USD per GB-hour in US East), also billed per second with a one-minute minimum per worker; Graviton (ARM) rates are lower. Always check the live pricing page for your Region and instance family, since rates differ. Compare the cost of a persistent cluster (for continuous workloads) against serverless (for intermittent batch jobs) to determine the cheaper option for your usage pattern.

Best practices

EMR is covered by the AWS Well-Architected Framework, the standard methodology for reviewing cloud workloads. For EMR the most relevant pillars are cost optimization (instance choice, Spot for task nodes, Graviton, and right-sizing all map directly to spend) and performance efficiency (cluster shape, file format, and partitioning). See the /foundations/well-architected/ overview and the /glossary/cost-optimization-pillar/, and the AWS EMR best practices guides linked in Sources for Spark tuning and cost guidance.

Sources