Name: configuring-filters
Author: celigo

Configure Celigo filter rules on exports, imports, and flow branches that control which records continue through the pipeline. Use when adding filters, setting branch routing conditions, or choosing between expression and script filters.

<! TIER:1 Configuring Filters A filter is a record gate that decides which records pass a point in the pipeline. What happens to the rest depends on where it sits: an export's filter silently drops them from the flow, while a lookup's or an import's filter only skips that step for them. Filters are configured directly on exports, imports, and flow router branches not as standalone resources. Concerns when configuring a filter: Placement where in the pipeline the filter runs (output filter on export, input filter on lookup, pre import filter, branch routing condition) Mechanism expression based rules (declarative, no code) vs. script based (full JavaScript control) Expression syntax prefix notation S expressions with operators, field access ( extract ), type coercions, and logical combinators ( and / or / not ) Settings references dynamic filter values pulled from flow, export, import, or integration settings via settings accessor Script alternative when expression rules cannot handle the logic (loops, multi step conditionals, external lookups), use a filter script instead. Dates relative to now and arithmetic across fields don't need one ([Dates and Computed Values]( dates and computed values)) Four Places Filters Appear Output filter (export filter ) Applied after records are retrieved from the source system. Records that match continue through the flow; records that do not match are silently dropped. Available on all export types. Input filter (lookup export inputFilter ) Applied to incoming records before they trigger a lookup API call. Records that do not match skip the lookup step entirely they pass through without enrichment. This reduces unnecessary API calls. Only relevant when isLookup: true . Import filter (import filter ) Decides whether the import runs for a record. A record that doesn't match skips this import nothing is sent to the destination for it and continues to the next step with nothing from this import's response mapping, so a later step that should only see written records needs its own filter. Useful for conditional writes (e.g., only import records with a non empty email). AI agent and guardrail steps are imports, and their filter works the same way. Branch input filter (router branch inputFilter ) Applied per branch in a flow router to route records to different processing paths. Each branch has its own filter expression. The last branch can omit a filter to serve as a catch all. Configured on the flow's routers[].branches[] entries, not on exports or imports. Quick Reference Filter Type Decision Matrix Situation Use Configured on Skip records from the source based on field values Output filter ( filter ) Export resource Skip lookup calls for records missing required fields Input filter ( inputFilter ) Export resource ( isLookup: true ) Skip import for records that should not be written Import filter ( filter ) Import resource Route records to different branches by field values Branch filter ( inputFilter ) Flow routers[].branches[] Loops, multi step logic, external calls Script filter Export or import filter.type: "script" Expression vs. Script Mechanism When to use Config Expression ( type: "expression" ) Field comparisons, regular expressions, empty checks, dates relative to now, arithmetic, combining with and/or filter.expression.rules array Script ( type: "script" ) Complex business logic, cross record state, values epochtime can't read, external API calls filter.script. scriptId + filter.script.function Prefer expressions they are simpler, faster, and do not require a separate script resource. Use scripts only when expression operators cannot handle the logic. Operator Quick Reference Operator Meaning Operands equals Exact match field, value notequals Not equal field, value greaterthan Greater than field, value greaterthanequals Greater than or equal field, value lessthan Less than field, value lessthanequals Less than or equal field, value contains Substring match field, substring doesnotcontain No substring match field, substring startswith Prefix match field, prefix notstartswith No prefix match field, prefix endswith Suffix match field, suffix notendswith No suffix match field, suffix matches Regular expression match, ignoring case field, regular expression notempty Field has a value field empty Field is null/empty field and All conditions true condition, condition, ... or Any condition true condition, condition, ... not Negate condition condition To test that a field has a value, use notempty , not a pattern or an empty string comparison: a record without the field passes ["matches", ["string", ["extract", "email"]], ". \\S. "] , and one whose field is missing or null passes ["notequals", ["string", ["extract", "email"]], ""] . Operator names are case insensitive ( NOTEMPTY and endsWith work), but only the names in this table exist. notcontains , doesnotstartwith , doesnotendwith , notmatches , doesnotmatch and notequal are refused with 422 Invalid operation: <name . , on save and in the filter processor. To negate a condition the table has no opposite for, wrap it in not : ["not", ["matches", ...]] . Field access: ["extract", "fieldName"] for record fields, ["settings", "flow.settingName"] for configuration values. Paths are relative to the record itself ( customer.email for a nested field): ["extract", "record.status"] matches nothing. Only a Handlebars operand takes the record. prefix. Type coercions: ["number", ...] , ["string", ...] , ["boolean", ...] , ["epochtime", ...] the only four; datetime , date and timestamp are refused with 422 Invalid operation: <name . Every extract must sit inside one: a bare ["extract", ...] is refused with 422 "Field operation extract must be called by a Type Conversion Operation", both when the filter is saved and by the filter processor ( celigo processors invoke filter ). A bare settings is refused the same way. The value compared with a coercion must have its type: number with an unquoted number ( 50 ), string with a quoted string, boolean with true or false . A mismatch such as ["number", ...] against "50" is refused with 422 "Argument 2 for operation greaterthanequals is a different data type: string.", on save and in the filter processor. A quoted number is a string, though, so ["string", ...] against "50" saves and compares as text: "120.00" sorts before "50" , and the rule drops 120 and 1250 while passing 75. Compare amounts as ["number", ["extract", "total price"]] against 50 , even when the field arrives as a string. number reads a missing or empty field as 0, so ["lessthan", ["number", ["extract", "total"]], 100] passes records that have no total. Add ["notempty", ["string", ["extract", "total"]]] beside it with and when that matters. Transformations: ["lowercase", ...] , ["uppercase", ...] , ["floor", ...] , ["ceiling", ...] , ["abs", ...] . Arithmetic on number operands: ["add", a, b] , ["subtract", a, b] , ["multiply", a, b] , ["divide", a, b] , ["modulo", a, b] . Dates and Computed Values Compare a date as ["epochtime", ["extract", "<field "]] against epoch milliseconds, written as a number. epochtime reads ISO 8601 strings and epoch millisecond numbers; a string comparison of timestamps orders values with different UTC offsets wrongly. epochtime fails on a value it can't read an empty string, text that isn't a date, or epoch milliseconds sent as a string with Invalid date. Cannot convert given string or number to date time. In a run, that record fails with FILTER EVALUATION FAILED and the others go on; the filter processor and an export preview refuse the whole call with 422. A notempty test beside it doesn't help, because and and or evaluate every operand. A missing or null field doesn't match and raises nothing. When the field can arrive blank, use a script filter. To compare with the current time, use a Handlebars operand: ["number", "{{timestamp \"x\"}}"] is now in epoch milliseconds, so this passes records created in the last 7 days: A helper can also compute a value from fields: ["equals", ["number", "{{add record.a record.b}}"], 5] . A Handlebars operand is evaluated only when the whole string is one {{...}} expression text around it, or a second expression, makes it a literal. It reads the record as record.<field (an unprefixed {{status}} renders empty) and renders text, so wrap it in ["number", ...] before comparing it with a number (otherwise 422 ... is a different data type: number ). epochtime doesn't render it (422 invalid datetime format ), and an unknown helper fails the rule with 422 Failed to evaluate handlebars expression . Schema Index Schema Contents [filter.yml](references/schemas/filter.yml) Filter object type, expression (version, rules), script ( scriptId, function), operators, Handlebars operands and examples Related Skills [configuring exports Quick Reference](../configuring exports/SKILL.md quick reference) export configuration where output filter and input filter are wired [configuring imports Quick Reference](../configuring imports/SKILL.md quick reference) import configuration where import filter is wired [writing scripts Record Level Processors](../writing scripts/SKILL.md record level processors on export or import) script based filter hook signature and data shapes [building flows Flow Topologies](../building flows/SKILL.md flow topologies) branching routers with per branch input filters [writing handlebars Quick Reference](../writing handlebars/SKILL.md quick reference) Handlebars in dynamic filter values <! TIER:2 How to Configure a Filter 1. Determine filter placement Where in the pipeline should filtering happen? After export retrieval use an output filter on the export ( filter ) Before a lookup call use an input filter on the lookup export ( inputFilter ) Before import submission use a filter on the import ( filter ) For branch routing use branch input filters on the flow router Put the filter on the step that should skip the record. An export's output filter removes the record from every later step. When only one destination should skip it every customer goes to the ERP, only some to the newsletter filter that destination's import and leave the export and the other imports alone. When several steps share the condition, route them down a branch instead of repeating the filter on each. 2. Choose expression vs. script Can the logic be expressed as comparisons combined with and/or including dates relative to now and arithmetic across fields ([Dates and Computed Values]( dates and computed values))? Use an expression. Need loops, multi step logic, a date field that can arrive blank, or external API calls? Use a script. 3. Build the expression rules Expression rules use prefix (S expression) notation. Every rule is an array where the first element is the operator and remaining elements are operands. Pattern: Single condition Pattern: Multiple conditions (all must match) Pattern: Multiple conditions (any must match) Pattern: Numeric comparison (the value unquoted: 50 , not "50" ) Pattern: Boolean test passes true and the string "true" Test for the value you want: [notequals, [boolean, [extract, fieldName]], false] also passes records where the field is missing or null. Pattern: Dynamic value from settings 4. Set the filter on the resource The filter is a top level field of the export or imp