Create custom OpenLineage extractors for Airflow operators. Use when the user needs lineage from unsupported or third-party operators, wants column-level lineage, or needs complex extraction logic beyond what inlets/outlets provide.

Creating OpenLineage Extractors This skill guides you through creating custom OpenLineage extractors to capture lineage from Airflow operators that don't have built in support. Reference: See the [OpenLineage provider developer guide](https://airflow.apache.org/docs/apache airflow providers openlineage/stable/guides/developer.html) for the latest patterns and list of supported operators/hooks. When to Use Each Approach Scenario Approach Operator you own/maintain OpenLineage Methods (recommended, simplest) Third party operator you can't modify Custom Extractor Need column level lineage OpenLineage Methods or Custom Extractor Complex extraction logic OpenLineage Methods or Custom Extractor Simple table level lineage Inlets/Outlets (simplest, but lowest priority) Important: Always prefer OpenLineage methods over custom extractors when possible. Extractors are harder to write, easier to diverge from operator behavior after changes, and harder to debug. On Astro Astro includes built in OpenLineage integration — no additional transport configuration is needed. Lineage events are automatically collected and displayed in the Astro UI's Lineage tab . Custom extractors deployed to an Astro project are automatically picked up, so you only need to register them in airflow.cfg or via environment variable and deploy. Two Approaches 1. OpenLineage Methods (Recommended) Use when you can add methods directly to your custom operator. This is the go to solution for operators you own. 2. Custom Extractors Use when you need lineage from third party or provider operators that you cannot modify . Approach 1: OpenLineage Methods (Recommended) When you own the operator, add OpenLineage methods directly: OpenLineage Methods Reference Method When Called Required get openlineage facets on start() Task enters RUNNING No get openlineage facets on complete(ti) Task succeeds No get openlineage facets on failure(ti) Task fails No Implement only the methods you need. Unimplemented methods fall through to Hook Level Lineage or inlets/outlets. Approach 2: Custom Extractors Use this approach only when you cannot modify the operator (e.g., third party or provider operators). Basic Structure OperatorLineage Structure Extraction Methods Method When Called Use For execute extraction() Before operator runs Static/known lineage extract on complete(task instance) After success Runtime determined lineage extract on failure(task instance) After failure Partial lineage on errors Registering Extractors Option 1: Configuration file ( airflow.cfg ) Option 2: Environment variable Important: The path must be importable from the Airflow worker. Place extractors in your DAGs folder or installed package. Common Patterns SQL Operator Extractor File Transfer Extractor Dynamic Lineage from Execution Common Pitfalls 1. Circular Imports Problem: Importing Airflow modules at the top level causes circular imports. 2. Wrong Import Path Problem: Extractor path doesn't match actual module location. 3. Not Handling None Problem: Extraction fails when operator properties are None. Testing Extractors Unit Testing Precedence Rules OpenLineage checks for lineage in this order: 1. Custom Extractors (highest priority) 2. OpenLineage Methods on operator 3. Hook Level Lineage (from HookLineageCollector ) 4. Inlets/Outlets (lowest priority) If a custom extractor exists, it overrides built in extraction and inlets/outlets. Related Skills annotating task lineage : For simple table level lineage with inlets/outlets tracing upstream lineage : Investigate data origins tracing downstream lineage : Investigate data dependencies

creating-openlineage-extractors