ML pipelines need orchestration: run data ingestion, then preprocessing, then training, then evaluation, then conditionally deploy. Apache Airflow and AWS Step Functions are the two most common orchestrators for these workflows on AWS.

Platform Overview

Apache Airflow is an open-source workflow orchestration platform. Workflows (DAGs) are defined in Python. Amazon MWAA (Managed Workflows for Apache Airflow) provides managed Airflow on AWS. Airflow has a rich ecosystem of operators for integrating with external services. Apache Airflow 3.0 became generally available in April 2025 and is a significant release: a rewritten React and FastAPI web UI, DAG versioning (a run completes against the DAG version it started on), a new Task SDK, event-driven scheduling, and a Task Execution API that decouples task execution from the scheduler. Amazon MWAA added Airflow 3 support on October 1, 2025 and, as of mid-2026, offers it alongside the 2.x line; Airflow 3 environments on MWAA run on Python 3.12.

AWS Step Functions is a serverless workflow orchestration service. Workflows are defined in Amazon States Language (JSON or YAML), in Workflow Studio, or using the AWS Cloud Development Kit (CDK). Through AWS SDK integrations, Step Functions can call over nine thousand API actions across more than 200 AWS services, and the HTTP Task lets a workflow call external HTTPS APIs (for example Stripe or Salesforce) without a Lambda function in between.

Feature Comparison

FeatureAirflow (MWAA)Step Functions
Workflow definitionPython (DAGs)JSON/YAML (ASL) or SDK
SchedulingBuilt-in scheduler (cron, intervals)EventBridge rules (separate)
Visual designerAirflow UI (DAG visualization)Workflow Studio (visual builder)
AWS integrationsVia operators (amazon provider package)Native plus AWS SDK (200+ services, 9,000+ actions)
Error handlingRetry, on_failure callbacksRetry, Catch, fallback states
Parallel executionYes (parallel tasks)Yes (Parallel, Map states)
Human approvalCustom operator (e.g., Slack)Manual approval via callback
CostMWAA environment ($0.49/hour minimum)Per state transition ($0.025/1000)
Max execution timeUnlimited1 year (Express: 5 minutes)
State managementXCom (limited), external storageBuilt-in state passing

ML Pipeline Fit

Data Processing Pipelines

Airflow excels at scheduled data pipelines. Define a DAG that runs daily: pull data from sources, validate, transform, load into the feature store. Airflow’s scheduler handles the cadence, retries, and backfills. The Python-native DAG definition makes complex branching logic natural.

Step Functions handles data pipelines through integration with Glue, Lambda, and ECS. The visual workflow builder makes simple pipelines easy to create. Complex branching logic is more verbose in ASL than in Python.

Advantage: Airflow for complex, scheduled data pipelines

Model Training Pipelines

Airflow can trigger Amazon SageMaker AI training jobs via the SageMaker operators, wait for completion, and proceed to evaluation. (The classic build, train, and deploy service was renamed Amazon SageMaker AI in late 2024; the name Amazon SageMaker now refers to the broader unified data and AI platform.) The SageMaker operators are mature and well-documented.

Step Functions has native SageMaker AI integration: CreateTrainingJob, CreateTransformJob, CreateEndpoint. No custom code needed. The integration is direct and handles polling and error cases automatically.

Advantage: Step Functions for simple SageMaker AI pipelines; Airflow for complex pipelines with many custom steps

Deployment Pipelines

Airflow can orchestrate multi-stage deployments but is not primarily a deployment tool. Using Airflow for blue-green or canary deployments requires custom operators.

Step Functions integrates with CodeDeploy, ECS, and Lambda for deployment. Parallel deployment to multiple environments and wait-for-approval states are built in.

Advantage: Step Functions for deployment orchestration

Operational Considerations

MWAA runs a persistent Airflow environment (scheduler, web server, workers). The smallest environment (mw1.small) is billed at about $0.49 per hour, which works out to roughly $360 per month if it runs continuously, before any extra workers, schedulers, web servers, or metadata database storage. The environment runs continuously, regardless of pipeline activity. Scaling workers for burst workloads requires configuration.

Step Functions is truly serverless. No infrastructure to manage. You pay only when workflows execute. Idle cost: zero. Scales automatically to any workload.

For teams running pipelines frequently (daily or more), MWAA’s fixed cost is amortized. For teams running pipelines weekly or monthly, Step Functions’ pay-per-execution model is more cost-effective.

Developer Experience

Airflow DAGs are Python code. ML engineers and data scientists are comfortable writing Python. The DAG defines the workflow clearly, and custom logic is just Python functions. Testing DAGs locally is straightforward. The Airflow community provides operators for almost every service.

Step Functions workflows are defined in JSON (ASL) or using the SDK. ASL is verbose for complex logic. The Workflow Studio visual builder helps for simple workflows but becomes unwieldy for complex ones. Custom logic requires Lambda functions, adding a layer of indirection.

For ML teams that think in Python, Airflow is more natural. For platform teams building reusable workflow patterns, Step Functions’ declarative approach is cleaner.

Common Patterns

Pattern 1: Airflow for Everything

Use MWAA to orchestrate all ML pipelines: data ingestion, training, evaluation, deployment. Works well for teams with Airflow expertise and complex, frequently-running pipelines.

Pattern 2: Step Functions for Everything

Use Step Functions for all workflow orchestration. Works well for AWS-native teams with simpler pipeline logic and variable execution frequency.

Pattern 3: Airflow + Step Functions

Airflow handles complex, scheduled data pipelines (daily data processing, feature engineering). Step Functions handles event-driven workflows (model deployment triggered by model registry update, inference pipeline triggered by API call).

Recommendation

Choose Airflow (MWAA) when you have complex ML pipelines with many steps, need Python-native workflow definition, run pipelines frequently (daily or more), or the team already knows Airflow.

Choose Step Functions when you want serverless operation with zero idle cost, need tight AWS service integration, run pipelines infrequently, or prefer visual workflow design for simple pipelines.

If you have decided on Airflow but are choosing an engine, see also /comparisons/airflow-vs-dagster/.

Sources and Further Reading