Use when defining field mappings for data streams, populating ecs.yml with ECS field references, selecting ECS categorization values, choosing custom field types, or troubleshooting mapping validation failures.

ecs field mappings When to use Use this skill when tasks include: adding or modifying files under data stream/<stream /fields/ populating ecs.yml with ECS field references selecting event.kind , event.category , event.type , and event.outcome values choosing field type and mapping properties ( metric type , dimension , multi fields , and related options) checking whether a field already exists in ECS before adding custom fields troubleshooting mapping validation/build failures from elastic package check , elastic package lint , or pipeline test schema checks ECS dependency configuration Every package needs dev/build/build.yml at the package root. This file pins the ECS schema version used for field resolution. This file is required whenever the package has any field file. The scaffold does not generate it — create it manually. If it is missing or uses an outdated version, tests report ECS fields as undefined (e.g., field "destination.ip" is undefined ). Field files and roles A data stream's fields/ directory contains a small set of YAML files with distinct responsibilities: base fields.yml Fixed routing constants and @timestamp . All six fields are ECS fields, so each entry uses external: ecs . Override type and value where the data stream needs a constant keyword with a fixed value — the description is inherited from ECS automatically. Do not add other fields here. Only these routing constants and @timestamp belong in base fields.yml . constant keyword candidates Fields that hold a single value for every document in a data stream should use constant keyword . Beyond the routing constants in base fields.yml , evaluate these: Field Why constant keyword event.dataset One value per data stream by definition event.module One value per package data stream.type Fixed per stream ( logs / metrics ) data stream.dataset Fixed per stream data stream.namespace Set at deployment, constant within index observer.vendor Package represents one vendor observer.product Package represents one product When a constant keyword field is also an ECS field (e.g., observer.vendor ), use external: ecs with the type override. This inherits the description from ECS and avoids manual duplication. Place the definition in the appropriate field file ( ecs.yml for most ECS fields, base fields.yml for routing constants): remove from source option: Because constant keyword stores the value once in index metadata, it does not need to appear in every document's source . Elasticsearch handles this automatically — no explicit source.excludes configuration is needed. This saves storage when the value is always the same. ecs.yml Populate this file with the ECS fields the pipeline sets that dynamic mapping cannot infer. Use only name and external: ecs for each entry — no type, no description. The type is resolved from the ECS schema via dev/build/build.yml . On packages whose conditions.kibana.version floor is = 8.13.0, the ecs@mappings component template dynamically maps standard keyword/date and standard prefix geo ECS fields, so explicit declarations are REQUIRED only for types it cannot infer ( geo point on non standard parent prefixes, geo shape , nested , flattened ) or where elastic package validation fails. The gate is the kibana version constraint, not the package spec version — ecs@mappings is applied by the stack at install time from 8.13 onward, and a spec 3.x package that still admits 8.11/8.12 installs does not get it there, so such packages still need every pipeline set ECS field declared. Declaring more is harmless for a builder, but is never something a reviewer should demand — and existing declarations are not removable noise findings either (see review integration/references/conflict resolutions.md ). When you DO declare a field whose name exists in ECS, external: ecs must be used ([wiki reference](https://github.com/elastic/integrations/wiki/Fleet Package Code Review Comments defining an ecs field without using an external definition)). This applies across field files — ecs.yml , base fields.yml , and any file that defines an ECS field. You may override properties (e.g., type: constant keyword , value: ) while still using external: ecs — the description is inherited from ECS. Do not use external: ecs in fields.yml , agent.yml , or beats.yml — those files define non ECS fields. When attaching extra metadata to an ECS field (for example making a field a TSDB dimension or a constant keyword with a fixed value), combine external: ecs with that metadata. The description is inherited from ECS. Place the definition in ecs.yml (or base fields.yml for routing constants): fields.yml Integration specific custom (non ECS) fields only. Use a nested group hierarchy for the vendor namespace: Groups do not need to be declared as type: object — defining a group with nested fields is sufficient. The object structure is implicit. labels. exception labels is a core ECS object ( type: object , object type: keyword ) designed for ad hoc key value metadata. Subkeys under labels. do not require vendor namespacing — this is the one exception to the vendor prefix rule. Use labels. for simple keyword flags or integration internal markers (e.g., labels.is ioc transform source ). Use the vendor namespace for structured or nested data from an upstream source. Flags vs structured data Boolean flags and simple tags can live flat under the vendor group: Structured data from the source should use sub groups for logical hierarchy: agent.yml Non ECS fields populated by the Elastic Agent or Beats framework but not covered by ECS. Include only when the input type emits these fields. Typical fields: cloud.image.id , cloud.instance.id , host.containerized , host.os.build , host.os.codename , input.type , log.offset . See references/root and core fields.md for full YAML samples. beats.yml Filebeat/Beats specific fields not covered by ECS. Minimal form contains input.type and log.offset . Some inputs also emit log.flags or log.file. sub fields. See references/root and core fields.md for full YAML samples. ECS field selection Prefer ECS fields whenever semantics match. If no ECS field exists for the data, add it under the package namespace in fields.yml . Categorization quick reference Field Type Notes event.kind keyword Highest level classification. event.category keyword[] Broad domain buckets — always an array. event.type keyword[] Sub buckets within category — always an array. event.outcome keyword success , failure , unknown ; only set when meaningful. event.kind : alert , asset , enrichment , event , metric , pipeline error , signal , state event.category : api , authentication , configuration , database , driver , email , file , host , iam , intrusion detection , library , malware , network , package , process , registry , session , threat , vulnerability , web event.type : access , admin , allowed , change , connection , creation , deletion , denied , device , end , error , group , indicator , info , installation , protocol , start , user Decision workflow: 1. event.kind : event for normal logs, metric for measurements, state for periodic categorical observations about a thing (e.g. CDR posture/compliance findings), asset for entity/inventory snapshot records — one document per entity per collection cycle (see entity mappings skill), pipeline error in on failure 2. event.category : one or more values (array) for the broad domain 3. event.type : one or more values (array) for operation style 4. event.outcome : only when a clear success/failure/unknown applies; omit for informational/metric events 5. If no allowed value fits, leave the field empty — do not invent values Use event.action for source specific verbs ( blocked , dropped , authenticated ). See references/categorization cheatsheet.md for full worked examples. Timestamp fields ECS defines several timestamp fields with distinct semantics. Use them correctly: Field When to use Set by @timestamp The primary event timestamp. Parse from the source event data. Required. Integration pipeline event.created When the event was first created or recorded by the source system, if different from @timestamp . Integration pipeline event.start When an activity or period began (e.g., session start, connection start). Integration pipeline event.end When an activity or period ended (e.g., session end, connection close). Integration pipeline event.ingested When the event was ingested into Elasticsearch. Elasticsearch (outside the integration) event.ingested must NEVER be set by an integration pipeline. It is managed automatically by Elasticsearch's final pipeline. Do not add a set processor for event.ingested . When the source data contains multiple timestamps: 1. Map the primary event timestamp to @timestamp . 2. If another timestamp represents when the event was first recorded/created, map it to event.created . 3. If timestamps represent the start or end of an activity, map them to event.start and event.end . 4. If a timestamp does not match the semantics of any of the above, map it to a custom field under the vendor namespace with type: date in fields.yml . Reusable fieldset nesting rules Some ECS field sets must be nested under a parent entity — they are not valid at document root. geo — must be nested under: client.geo , destination.geo , host.geo , observer.geo , server.geo , source.geo , threat.indicator.geo Root level geo. fields are not recognized and will appear unmapped. Always set target field on the geoip processor: as (Autonomous System) — nested under: client.as , destination.as , server.as , source.as When using geoip for geolocation, always also perform an ASN lookup using GeoLite2 ASN.mmdb and rename the raw output fields to ECS names. The geoip ASN processor outputs asn and organization name , which must be renamed to as.number and as.organization.name : See the ingest pipelines skill → references/processor cookbook.md for the full geo+ASN pattern with both source and destination. os — nested under: host.os , observer.os , user agent.os entity (beta; leaf fields require ECS v9.4.0+ ) — nested under: user.entity , host.entity , service.entity , cloud.entity , orchestrator.entity , root entity , entity.target . Never invent device.entity. — not a valid parent. For entity data streams (i.e. those with event.kind: asset ), see the entity mappings skill for the full field catalog, pipeline patterns, and ECS availability matrix. Nested (array of objects) ECS fields Some ECS fields use type: nested , meaning they hold an array of objects where each object groups related sub fields together. The pipeline must produce this structure — do not flatten these into parallel scalar arrays. ECS fields that use nested type: Field Contains email.attachments file.name , file.size , file.extension , file.mime type , file.hash. threat.enrichments indicator. , matched. threat.indicator.file.elf.sections name , physical size , virtual size , etc. threat.indicator.file.pe.sections name , physical size , virtual size , etc. process.elf.sections name , physical size , virtual size , etc. process.pe.sections name , physical size , virtual size , etc. Anti pattern — parallel arrays (WRONG): This loses the association between each attachment's name and size. Queries cannot isol

ecs-field-mappings