Annotate Airflow tasks with data lineage using inlets and outlets. Use when the user wants to add lineage metadata to tasks, specify input/output datasets, or enable lineage tracking for operators without built-in OpenLineage extraction.

Annotating Task Lineage with Inlets & Outlets This skill guides you through adding manual lineage annotations to Airflow tasks using inlets and outlets . Reference: See the [OpenLineage provider developer guide](https://airflow.apache.org/docs/apache airflow providers openlineage/stable/guides/developer.html) for the latest supported operators and patterns. On Astro Lineage annotations defined with inlets and outlets are visualized in Astro's enhanced Lineage tab , which provides cross DAG and cross deployment lineage views. This means your annotations are immediately visible in the Astro UI, giving you a unified view of data flow across your entire Astro organization. When to Use This Approach Scenario Use Inlets/Outlets? Operator has OpenLineage methods ( get openlineage facets on ) ❌ Modify the OL method directly Operator has no built in OpenLineage extractor ✅ Yes Simple table level lineage is sufficient ✅ Yes Quick lineage setup without custom code ✅ Yes Need column level lineage ❌ Use OpenLineage methods or custom extractor Complex extraction logic needed ❌ Use OpenLineage methods or custom extractor Note: Inlets/outlets are the lowest priority fallback. If an OpenLineage extractor or method exists for the operator, it takes precedence. Use this approach for operators without extractors. Supported Types for Inlets/Outlets You can use OpenLineage Dataset objects or Airflow Assets for inlets and outlets: OpenLineage Datasets (Recommended) Airflow Assets (Airflow 3+) Airflow Datasets (Airflow 2.4+) Basic Usage Setting Inlets and Outlets on Operators Multiple Inputs and Outputs Tasks often read from multiple sources and write to multiple destinations: Setting Lineage in Custom Operators When building custom operators, you have two options: Option 1: Implement OpenLineage Methods (Recommended) This is the preferred approach as it gives you full control over lineage extraction: Option 2: Set Inlets/Outlets Dynamically For simpler cases, set lineage within the execute method (non deferrable operators only): Dataset Naming Helpers Use the [OpenLineage dataset naming helpers](https://openlineage.io/docs/client/python/best practices dataset naming helpers) to ensure consistent naming across platforms: Note: Always use the naming helpers instead of constructing namespaces manually. If a helper is missing for your platform, check the [OpenLineage repo](https://github.com/OpenLineage/OpenLineage) or request it. Precedence Rules OpenLineage uses this precedence for lineage extraction: 1. Custom Extractors (highest) User registered extractors 2. OpenLineage Methods get openlineage facets on in operator 3. Hook Level Lineage Lineage collected from hooks via HookLineageCollector 4. Inlets/Outlets (lowest) Falls back to these if nothing else extracts lineage Note: If an extractor or method exists but returns no datasets, OpenLineage will check hook level lineage, then fall back to inlets/outlets. Best Practices Use the Naming Helpers Always use OpenLineage naming helpers for consistent dataset creation: Document Your Lineage Add comments explaining the data flow: Keep Lineage Accurate Update inlets/outlets when SQL queries change Include all tables referenced in JOINs as inlets Include all tables written to (including temp tables if relevant) Outlet only and inlet only annotations are valid. One sided annotations are encouraged for lineage visibility even without a corresponding inlet or outlet in another DAG. Limitations Limitation Workaround Table level only (no column lineage) Use OpenLineage methods or custom extractor Overridden by extractors/methods Only use for operators without extractors Static at DAG parse time Set dynamically in execute() or use OL methods Deferrable operators lose dynamic lineage Use OL methods instead; attributes set in execute() are lost when deferring Related Skills creating openlineage extractors : For column level lineage or complex extraction tracing upstream lineage : Investigate where data comes from tracing downstream lineage : Investigate what depends on data

annotating-task-lineage