Use when designing or modifying Elasticsearch ingest pipelines, including single-path parsing, branching logic, sub-pipelines, enrichment processors, and robust on_failure handling.

ingest pipelines Skill authority The rules and patterns defined in this skill and its reference files are the authoritative source of truth . When examining existing integrations in the elastic/integrations repository for reference, you may encounter patterns that conflict with what is specified here — many integrations contain legacy patterns that predate current standards. Always follow this skill over patterns observed in other integrations. If a reference integration uses a deprecated or prohibited pattern, do not copy it. When to use Use this skill when tasks include: building or modifying elasticsearch/ingest pipeline/default.yml for a data stream choosing parser and normalization processors ( grok , dissect , json , kv , date , convert ) designing conditional branches and sub pipeline routing with pipeline processors implementing resilient error handling with top level on failure tuning processor order for ingest performance and maintainability When not to use Do not use this skill as the primary guide for: ECS field selection, categorization values, and field mapping strategy ( ecs field mappings ) elastic package command and stack lifecycle workflows ( elastic package cli ) test fixture authoring and expected output workflows ( integration testing → references/pipeline testing.md ) Pipeline anatomy In integration packages, ingest pipelines live under: data stream/<stream /elasticsearch/ingest pipeline/ Every stream usually has a default.yml with: description processors list optional pipeline level on failure Keep default.yml readable and focused. Move large format specific logic into sub pipelines where needed. ECS version Set the pipeline ECS reference version explicitly at the top of processors (after any introductory processors you already use). Use 9.3.0 — do not pin an older ECS version. Exception: when the orchestrator specifies this is an entity data stream ( event.kind: asset ), use 9.5.0 instead — entity leaf fields ( entity.attributes. , entity.lifecycle. , entity.relationships. ) do not exist at 9.3.0 and cause field is undefined build failures at that pin. Rename vs set (mapping to ECS) When moving a value from a custom or vendor field into an ECS field , prefer the rename processor so the source field is removed and you avoid duplicate data. Use set with copy from only when you must keep the source field or when rename is not applicable. Processor tags Every processor in the pipeline should have a tag (not only processors that can fail). Tags make failures and telemetry attributable to a specific step. CEL only opening processors (Agentless metadata and error only documents) For CEL based integrations only, include these before the standard message → event.original handling when they apply: remove : drop Agentless metadata fields ( organization , division , team ) when all are strings, so they do not collide with ECS. Use ignore missing: true and a conditional if . terminate : stop processing when the document is an error placeholder from the collector ( ctx.error?.message != null && ctx.message == null && ctx.event?.original == null ). Non CEL integrations (logs, syslog, filebeat style inputs) must not copy this block blindly — those fields and error shapes are specific to the CEL/Agentless path. See the create integration skill: the orchestrator must only expect this block when the data stream uses CEL input. Standard opening: ECS, optional CEL block, JSE00001, then parse event.original After the optional CEL only processors, the pipeline should follow this shape. All parsing ( json , csv , grok , etc.) runs on event.original . Never overwrite or mutate event.original in later processors — derive structured fields into other paths (for example json , temp. , ECS fields). Single path pattern (linear pipeline) Use this pattern when one parser flow handles all events. Combine the standard opening (ECS version, optional CEL only block, JSE00001 rename/remove, parse from event.original without mutating it), middle processors with tags on every step , and the pipeline level on failure and conditional append for preserve original event shown above. Example middle section (illustrative): Branching pattern (router + sub pipelines) Use branching when event formats or object models diverge: format based branching (for example JSON vs text) class/category based branching (for example OCSF class/category routing) object presence branching ( ctx.ocsf.user != null ) Pattern: In large integrations, keep default.yml as the router and put branch logic in files like: pipeline object <name .yml pipeline category <name .yml See references/branching patterns.md for full patterns from amazon security lake . Sub pipeline routing for multi log type integrations When a data stream receives multiple distinct log types (for example a firewall that emits traffic, auth, and DNS logs in the same stream), do not implement all parsing in a single monolithic default.yml . Use default.yml as a thin router that detects the log type and delegates to a dedicated sub pipeline per type. File layout Router pattern in default.yml Use the same ecs.version , JSE00001 rename / remove pair for message , and full pipeline level on failure as in the standard opening. The router only branches sub pipelines; it does not parse payloads. Rules default.yml must contain only routing logic and on failure handling — no field parsing. Each sub pipeline handles parsing, ECS mapping, and categorization for its own log type. Each sub pipeline must have its own on failure block. Name sub pipeline files pipeline <type .yml where <type matches the log type identifier used in the routing condition. Each log type gets its own pipeline test fixture file following the naming convention test <package <datastream <type sample.log . Processor ordering and performance run cheap existence checks before expensive operations drop early if records are out of scope prefer dissect over grok for stable delimited formats never use a script processor when a built in processor can do the job — set , rename , remove , append , convert , dissect , grok , gsub , lowercase , uppercase , and trim are all faster than Painless and easier to review. See the cost tiers in references/processor cookbook.md → Processor performance guide . use enrichment processors ( geoip , user agent ) only when needed always anchor grok patterns with ^ and $ — without anchors the regex engine scans the entire input string looking for a partial match, which is slow and can produce incorrect results on noisy log lines Mustache template syntax in processor values Ingest pipeline processors use Mustache templates to reference field values in value , message , and similar string parameters. Use triple braces {{{field}}} with single quotes — never double braces or double quotes: Why: Mustache double braces {{...}} HTML encode the value (e.g., & becomes & ), which corrupts data in ingest pipelines. Triple braces {{{...}}} emit the raw value. Single quotes prevent YAML from interpreting braces. Exception: {{ IngestPipeline "..." }} in pipeline.name is a Go template directive processed at build time, not a Mustache template — it correctly uses double braces. Error handling essentials Use pipeline level on failure as the main error reporting mechanism. Recommended baseline (order matters): append contextual error.message first using ingest.on failure variables (full template in the standard opening example) set event.kind: pipeline error (with a tag on the set processor) append preserve original event to tags when you need to retain the failed document for triage give every processor a tag (not only processors that can fail) Use processor level on failure for local cleanup or fallback parsing, not as the primary global error message path. See references/error handling patterns.md for full examples and tradeoffs ( ignore failure , fail , processor level on failure ). event.original handling (JSE00001) The elastic package build validator enforces that pipelines correctly handle the message to event.original rename. This check is known as JSE00001. New packages must comply; some legacy packages exclude it via validation.yml . Required two processor pattern Every pipeline that consumes a message field must include both processors (typically after ecs.version and after any CEL only remove / terminate steps when applicable): Step 1 ( rename ): moves message into event.original , but only when event.original is not already populated (idempotent when a prior pipeline or Logstash has already set it). Step 2 ( remove ): removes the redundant message field when event.original is present (after rename or from an upstream producer). Do NOT add an event.original removal processor at the end of the pipeline Some existing integrations contain a remove processor that deletes event.original at the end of the pipeline when preserve original event is not in tags . This pattern is deprecated and must not be used in new pipelines. The removal of event.original for storage optimization is now handled by a separate final pipeline outside the integration. Do not copy this pattern from reference integrations that still have it — it is legacy. That stack side final pipeline honors the preserve original event tag, so a package exposing the toggle needs NO in pipeline remove for the toggle to work — the toggle is not a no op without one. Never recommend adding this processor; when reviewing, its presence is the finding, never its absence. Reference The two processor JSE00001 pattern (rename + remove of message ) shown above is required and complete. Do not add any additional event.original processors beyond those two. Timezone handling ( tz offset ) For data streams that include the tz offset manifest var (syslog streams where messages lack a timezone), set event.timezone from conf.tz offset early in the pipeline, before any date parsing: This ensures date processors can apply the correct timezone when parsing timestamps that have no timezone component. Syslog structured data (RFC 5424 SD ELEMENT) parsing For vendor key=value payloads and RFC 5424 SD ELEMENT blocks, three strategies are available: KV with trim value (simplest, Strategy 1), SYSLOG5424SD grok + KV with regex splits (Strategy 2), and Painless for edge cases with embedded equals or mixed quoting (Strategy 3). Prefer Strategy 1 or 2; use Painless only when KV edge cases demand it. See references/grok recipes.md → Syslog structured data strategies for full code examples, key settings, and reference implementations. Keyword fields delivered as numbers Fields that carry identifiers, protocol codes, or other opaque values must be declared as keyword in fields.yml — even when the source data delivers them as numbers. Common examples: network protocol numbers ( network.iana number ) port numbers used as identifiers error codes, result codes, status codes SNMP OIDs, event IDs, object class codes Do not add a convert processor to stringify these values. Elasticsearch silently coerces numbers into keyword strings at index time, so the pipeline can pass the raw numeric value through unchanged. The field declaration in fields.yml : Because the test runner compares raw value types against declared field types, it will flag 6 (long) as a mismatch for keyword . Declare the field in numeric keyword fields in the pipeline test config so the runner accepts the numeric representation without requiring the fixture to

ingest-pipelines