Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.

Apache Airflow DAG Patterns Production ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies. When to Use This Skill Creating data pipeline orchestration with Airflow Designing DAG structures and dependencies Implementing custom operators and sensors Testing Airflow DAGs locally Setting up Airflow in production Debugging failed DAG runs Core Concepts 1. DAG Design Principles Principle Description Idempotent Running twice produces same result Atomic Tasks succeed or fail completely Incremental Process only new/changed data Observable Logs, metrics, alerts at every step 2. Task Dependencies Quick Start Detailed patterns and worked examples Detailed pattern documentation lives in references/details.md . Read that file when the navigation tier above is insufficient. Best Practices Do's Use TaskFlow API Cleaner code, automatic XCom Set timeouts Prevent zombie tasks Use mode='reschedule' For sensors, free up workers Test DAGs Unit tests and integration tests Idempotent tasks Safe to retry Don'ts Don't use depends on past=True Creates bottlenecks Don't hardcode dates Use {{ ds }} macros Don't use global state Tasks should be stateless Don't skip catchup blindly Understand implications Don't put heavy logic in DAG file Import from modules

airflow-dag-patterns