Deploys Airflow DAGs and projects. Use when deploying Airflow or answering anything about deployment - deploying DAGs/projects, pushing code, setting up CI/CD, deploying to production or deployment strategies for Airflow.

Deploying Airflow This skill covers deploying Airflow DAGs and projects to production, whether using Astro (Astronomer's managed platform) or open source Airflow on Docker Compose or Kubernetes. Choosing a path: Astro is a good fit for managed operations and faster CI/CD. For open source, use Docker Compose for dev and the Helm chart for production. Astro (Astronomer) Astro provides CLI commands and GitHub integration for deploying Airflow projects. Deploy Commands Command What It Does astro deploy Full project deploy — builds Docker image and deploys DAGs astro deploy dags DAG only deploy — pushes only DAG files (fast, no image build) astro deploy image Image only deploy — pushes only the Docker image (for multi repo CI/CD) astro deploy dbt dbt project deploy — deploys a dbt project to run alongside Airflow Full Project Deploy Builds a Docker image from your Astro project and deploys everything (DAGs, plugins, requirements, packages): Use this when you've changed requirements.txt , Dockerfile , packages.txt , plugins, or any non DAG file. DAG Only Deploy Pushes only files in the dags/ directory without rebuilding the Docker image: This is significantly faster than a full deploy since it skips the image build. Use this when you've only changed DAG files and haven't modified dependencies or configuration. Image Only Deploy Pushes only the Docker image without updating DAGs: This is useful in multi repo setups where DAGs are deployed separately from the image, or in CI/CD pipelines that manage image and DAG deploys independently. dbt Project Deploy Deploys a dbt project to run with Cosmos on an Astro deployment: GitHub Integration Astro supports branch to deployment mapping for automated deploys: Map branches to specific deployments (e.g., main production, develop staging) Pushes to mapped branches trigger automatic deploys Supports DAG only deploys on merge for faster iteration Configure this in the Astro UI under Deployment Settings CI/CD . CI/CD Patterns Common CI/CD strategies on Astro: 1. DAG only on feature branches : Use astro deploy dags for fast iteration during development 2. Full deploy on main : Use astro deploy on merge to main for production releases 3. Separate image and DAG pipelines : Use image and dags in separate CI jobs for independent release cycles Deploy Queue When multiple deploys are triggered in quick succession, Astro processes them sequentially in a deploy queue. Each deploy completes before the next one starts. Reference [Astro Deploy Documentation](https://www.astronomer.io/docs/astro/deploy code) Open Source: Docker Compose Deploy Airflow using the official Docker Compose setup. This is recommended for learning and exploration — for production, use Kubernetes with the Helm chart (see below). Prerequisites Docker and Docker Compose v2.14.0+ The official apache/airflow Docker image Quick Start Download the official Airflow 3 Docker Compose file: This sets up the full Airflow 3 architecture: Service Purpose airflow apiserver REST API and UI (port 8080) airflow scheduler Schedules DAG runs airflow dag processor Parses and processes DAG files airflow worker Executes tasks (CeleryExecutor) airflow triggerer Handles deferrable/async tasks postgres Metadata database redis Celery message broker Minimal Setup For a simpler setup with LocalExecutor (no Celery/Redis), create a docker compose.yaml : Airflow 3 architecture note : The webserver has been replaced by the API server ( airflow api server ), and the DAG processor now runs as a standalone process separate from the scheduler. Common Operations Installing Python Packages Add packages to requirements.txt and rebuild: Or use a custom Dockerfile: Update docker compose.yaml to build from the Dockerfile: Environment Variables Configure Airflow settings via environment variables in docker compose.yaml : Open Source: Kubernetes (Helm Chart) Deploy Airflow on Kubernetes using the official Apache Airflow Helm chart. Prerequisites A Kubernetes cluster kubectl configured helm installed Installation Key values.yaml Configuration Upgrading DAG Deployment Strategies on Kubernetes 1. Git sync (recommended): DAGs are synced from a Git repository automatically 2. Persistent Volume : Mount a shared PV containing DAGs 3. Baked into image : Include DAGs in a custom Docker image Useful Commands Related Skills setting up astro project : For initializing a new Astro project managing astro local env : For local development with astro dev authoring dags : For writing DAGs before deployment testing dags : For testing DAGs before deployment

deploying-airflow