Use this skill when the user asks to "set up parsing", "create parsing rule", "extract fields from logs", "regex extraction", "log parsing", "enrich logs", "add context to logs", "custom enrichment table", "lookup table", "geo enrichment", "create metric from logs", "events to metrics", "convert log

Data Pipeline Skill Use this skill when configuring how Coralogix processes, enriches, and transforms data. It covers parsing rules (extract structured fields from raw logs), enrichments (add context from lookup tables), Events2Metrics (derive metrics from log/span events), and recording rules (precompute PromQL expressions). CLI Commands Command Subcommands Purpose cx parsing rules list , get , create , update , delete , bulk delete , usage limits Manage log parsing rules cx enrichments list , add , remove , overwrite , limit , settings Manage enrichment rules cx enrichments custom list , get , create , update , delete , search Manage custom enrichment tables cx e2m list , get , create , update , delete , labels cardinality , limits Manage Events2Metrics definitions cx recording rules list , get , create , update , delete Manage Prometheus recording rule groups Key flags: All create/update operations use from file <path (or for stdin) All commands support o json for structured output and p <profile for profile selection cx parsing rules update and cx recording rules update require both from file and the rule group ID cx enrichments custom search requires id <table id and query <text cx parsing rules bulk delete requires ids <id1 <id2 ... Working with JSON Payloads These commands use complex JSON structures. Always template from an existing resource to avoid format errors: This pattern applies to all create/update operations across all 4 commands. It prevents payload format errors that are the 1 cause of failed attempts. Parsing Rules Workflow 1. List Existing Rules 2. Get a Template 3. Create New Rule Group Edit the template for your new service, then: 4. Verify Parsing Query recent logs to confirm fields are extracted (load cx telemetry querying for log querying): 5. Check Usage Limits Enrichment Workflow 1. List Enrichment Rules 2. Create Custom Enrichment Table (if needed) table definition.json must use the v5 JSON shape (inline file content, not multipart file=@... ): For updates, include customEnrichmentId (number) plus the same fields. 3. Add Enrichment Rules enrichment rules.json must use requestEnrichments (not enrichments from list output). Each enrichmentType is an object, not a string: Other types: {"aws": {"resourceType": "ec2"}} , {"suspiciousIp": {}} , {"customEnrichment": {"id": 1}} . 4. Search Custom Table Data 5. Verify Enriched Fields Query logs on hot storage (FrequentSearch tier) to confirm enriched fields appear. Avoid querying archive for verification ingestion delays can cause false negatives. Events2Metrics Workflow E2M derives Prometheus metrics from log/span events. See [ references/e2m schemas.md ](references/e2m schemas.md) for the full JSON wire format, enums, and cardinality rules. How E2M is computed (read this first) E2M aggregates events as they stream through the real time ingestion pipeline into metric series (~1 min resolution). It is forward only — metrics start from the moment the E2M is created; there is no backfill . All ingested data flows through the pipeline; a TCO policy routes each stream into a tier, and the tier decides what's possible: TCO tier Storage E2M / alerts / dashboards High Frequent Search (hot, OpenSearch) ✅ available Medium S3 archive (not hot storage) ✅ available — still processed by the pipeline Low Compliance only ❌ no aggregation features Blocked dropped ❌ The axis is tier / processing level — NOT "Frequent Search vs archive" (Medium is archive and E2M works on it). Do not tell users to "point E2M at archive instead of Frequent Search" — that is incorrect. 1. Design the metric Choose logs2metrics vs spans2metrics , the source field(s) + aggregations, and labels (with cardinality in mind — see references/e2m schemas.md ). To scope the E2M to a dataset , set the optional dataSource field to "<dataspace /<dataset " ; this requires the account feature e2m dataset source enabled (otherwise the API rejects it with "dataSource is not enabled for this company"). Omit it for the standard logs/spans stream. 2. Size it: check limits & cardinality The labels cardinality endpoint is a draft forecast — given proposed labels + query it returns the per day distinct permutation count over the last 7 days, so you can size a design before creating it. But cx e2m labels cardinality currently takes no arguments , so it sends no draft and returns an empty list (a CLI gap — it can't forecast yet). Until that's wired up, forecast via the UI or estimate permutations manually (product of distinct label values) and set permutationsLimit . Never use high cardinality fields (IDs, raw URLs, IPs) as labels. Note the forecast only sees Frequent Search (High tier) data. 3. Template from an existing definition Only cx e2m get returns the full payload ( {"e2m": {...}} ); list prints a summary. Extract .e2m and drop read only fields: 4. Create the E2M 5. Verify the metric Confirm series are being produced (load cx telemetry querying for metrics querying): Troubleshooting: E2M produces no metric series 1. Check the source data's TCO tier — if it's routed to Low/compliance (or blocked), E2M cannot run. Fix with a TCO change ( cx tco list / cx cost optimization ), not an E2M change. 2. Verify the query matches streaming data — run the E2M's lucene filter as a live cx logs / cx spans query and confirm it returns recent results. Note cx logs queries Frequent Search (High tier) by default; for a Medium tier (archive) source add tier archive , since the data won't appear in a default Frequent Search query even though E2M still produces series. 3. Remember it's forward only — no series exist for data ingested before the E2M was created. Cost optimization: convert High tier logs to metrics When the aggregated/metric view is what the customer most cares about , convert High tier logs → metrics , then downgrade the raw logs High → Medium . Medium still supports E2M/alerts/dashboards and costs less (S3 archive, no hot storage) — you keep cheap, detailed metrics while dropping expensive Frequent Search retention. 1. Find high volume High tier sources: cx usage summary / cx tco list (see cx cost optimization ). 2. Confirm which fields drive dashboards/alerts (see cx telemetry querying ). 3. Build + verify the E2M first (steps above). 4. Then change the TCO policy to move the raw logs High → Medium. Keep data on High or Medium (both support E2M); do not drop it to Low/compliance if metrics or alerts are still needed. Recording Rules Workflow 1. List Existing Recording Rules 2. Get a Template 3. Create Recording Rule Group 4. Verify with PromQL Confirm the precomputed metric is available (load cx telemetry querying for metrics querying): Key Principles Always template from existing cx <command get <id o json template.json before any create Verify after create query logs/metrics to confirm the pipeline change took effect Use o json all payload inspection and creation should use JSON output Check limits first cx parsing rules usage limits and cx e2m limits before creating to avoid hitting caps Bulk operations use cx parsing rules bulk delete ids for cleanup, not individual deletes Additional Resources Reference Files [ references/e2m schemas.md ](references/e2m schemas.md) Complete Events2Metrics JSON wire format: type / aggType enum values, logsQuery / spansQuery filters, metric labels & fields, the TCO tier compute model, cardinality/permutations sizing, and gotchas Related Skills cx telemetry querying discover what data is available before configuring pipeline, and verify parsing results, enriched fields, and E2M metric series via log/metrics queries cx cost optimization find high volume High tier sources worth converting to metrics, and move the raw logs High→Medium (TCO) after the E2M is verified

cx-data-pipeline