Name: building-flows
Author: celigo

Build Celigo flows -- pipelines that move data from source systems to destination systems on a schedule or in response to events. Covers scheduling, chaining, error management, and abstract/instance templating. Use when creating, editing, or debugging flows.

<! TIER:1 Building Flows A flow moves data from one or more source systems to one or more destination systems. It runs on a schedule, in response to events (webhooks, listeners), or when triggered by another flow. Flows are the primary way integrations get work done in Celigo. A flow has page generators (exports that fetch data) and page processors (imports and lookups that process each record). Processors run sequentially in a flat list, or conditionally through routers that branch records to different paths. These processing pipeline mechanics routers, branches, page processors, response mapping are shared with APIs and tools (see building apis and building tools ). How to Run the Steps This skill names each build step by what it reads or changes; the exact calls depend on how you reach the account. Read the one reference that matches your tools both follow the step numbers of How to Build a Flow below, then Duplicate Records and the headings of Flow Operations: Celigo MCP tools ( list flows , upsert flow , patch flow , run flow , ...) [references/mcp.md](references/mcp.md) The celigo CLI [references/cli.md](references/cli.md) Planning and checking what already exists only read the account. Creating or changing a resource, enabling a flow, running it, and retrying or resolving its errors all write to the account or to the connected systems. The two surfaces differ most on writes which fields can be changed on their own, and whether the read modify write of a full update is done for you so follow your reference there. What Starts a Flow Flows start themselves this is the biggest thing that separates them from APIs (invoked by an HTTP caller) and tools (invoked by a consumer). Every flow begins with one or more page generators, of two kinds: Scheduled exports the flow runs on a cron cadence and each run pulls from the source: everything (full sync), only what changed since the last run (delta sync), records matching a query, or files landed in an FTP/SFTP/S3 folder. The right primitive for batch work: nightly reconciliations, hourly delta syncs, backfills, off peak windows. Listeners the source pushes to the flow. A webhook fires (or a NetSuite/Salesforce native real time event triggers) and the payload immediately starts flowing. No schedule; the flow runs as events arrive. The right primitive for event driven work ("when X happens, do Y"), especially when latency matters. A flow can mix both, and multi generator designs are common: Real time plus batch safety net a listener catches events as they fire; a scheduled export reconciles at off peak hours, catching up after webhook outages Consolidating sources customers from Salesforce AND HubSpot, each with its own generator, feeding the same downstream pipeline Different slices of the same source one export pulls new records, another pulls updated records, when the API exposes them separately If the requirement is "every night at 2 AM, do X" or "when a webhook arrives, do Y" that lives on a flow. APIs and tools have no schedule and no listener; they only run when invoked. Fetched Data Needs a Downstream Consumer A common design mistake: ending a flow on an import that fetches data back from a remote system (a preview call, a query, a lookup shaped POST) and relying on response mapping to capture the result. Response mapping makes fields visible to the NEXT step if no next step exists, the captured data is discarded when the run ends and nobody sees it. When the requirement says "preview / estimate / retrieve / fetch / check / look up", the design needs at least one of: A write back import to the source system (most common) e.g. source export preview import update import that writes the captured fields onto the source record A persistent destination the user named (file to S3/SFTP, email, database) A router or AI agent step that consumes the captured data within the same run A two step export fetch shaped import flow with nothing after it is a smell re read the intent for where the fetched data should end up. The same applies in reverse: capturing a created record's ID via response mapping is only useful if a later step writes it somewhere. Flow Topologies Linear Flows A flat pageProcessors[] list with no routers. One or more page generators feed records through a sequential chain of page processors. Each processor is either an import ( type: "import" ) or a lookup export ( type: "export" ). Records pass through every step in order. Unique to flows APIs and tools always use routers. Older flows and Data Loader flows, which still produce it can carry the legacy two step shape instead: a top level exportId (the one source) and importId (the one destination) in place of pageGenerators and pageProcessors . The platform still accepts and serves it; read such a flow's steps from those two fields, and build new flows with pageGenerators and pageProcessors . Branching Flows Page generators feed records into routers[] instead of pageProcessors[] . Each router evaluates records against branch conditions and routes them to matching branches. Branches contain their own pageProcessors[] and can chain to other routers via nextRouterId . Records enter at routers[0] and reach every other router only through a nextRouterId , so the rest of the array isn't execution order (Flow Builder can save routers out of order): walk the chain from routers[0] when you describe or change a branching flow. Two routing modes (shared with APIs and tools): Input filters ( routeRecordsUsing: "input filters" ) S expression rules on each branch; last branch can omit filter as a catch all Script based ( routeRecordsUsing: "script" ) a JavaScript branching function returns, per record, an array of branch indices zero based positions in branches[] (e.g. [0] ); [] sends the record down no branch Flows support both first matching branch and all matching branches routing. APIs only support first matching branch . Tools support first matching branch only. first matching branch a record takes the first branch that matches, so a broad filter ahead of a narrow one takes its records: order branches from most to least specific. A script that returns more than one index fails that record with invalid script routing . all matching branches a record goes down every branch that matches, each as its own copy. No match a record that no branch takes leaves the run with no error and no ignore count. When records go missing, check each branch's filter, or read a test run's router stage, which lists the branches each record took. A flow uses EITHER pageProcessors (linear) OR routers (branching) at the top level not both. Turning a linear flow into a branching one moves its top level steps into the branches; a body that keeps them next to routers is refused with 422. A branching router sets both routeRecordsTo and routeRecordsUsing ; a router with only one of them is refused with 422. With input filters: The filter shape. Each branch's inputFilter is { version: "1", rules: [...] } , not the { type: "expression", expression: {...} } wrapper that export and import filters use. A wrapped filter saves without an error, but the branch is stored with no filter, and every record goes down it (live verified 2026 10 21) read the flow back and check each branch's inputFilter . Wrap every extract in a type conversion string , number , boolean , or epochtime . A bare ["extract", "<field "] is refused with 422. Order the branches. They are checked in order. A branch with no filter matches every record, so under first matching branch the catch all goes last; under all matching branches a catch all also receives a copy of every record the other branches take. A two branch router, wholesale orders to one import and everything else to another: On an existing flow, this replaces its top level pageProcessors in the full document you send back; every other field goes back as read. When a branching flow needs linear steps before the branch point (e.g., a lookup enrichment or AI classification that all branches depend on), use a pass through router : a single branch router with nextRouterId pointing to the branching router. Omit routeRecordsTo and routeRecordsUsing on the pass through router including them makes it appear as a filter based branch in the UI. Leaving both unset is the pass through shape; the server fills in no default. Setting only one of the two is refused (422 invalid flow router config ), and so is a pass through router with more than one branch (422 invalid flow router branch ). The same shape rejoins branches: point every branch's nextRouterId at one pass through router and put the shared steps in its branch. Don't repeat a step in each branch instead an import can be a step only once per flow, and a second entry fails with 422 invalid flow ("already included as a pageProcessor"). Abstract / Instance Flows A template/inheritance model. An abstract flow ( isAbstract: true ) defines the complete graph but cannot execute. Instance flows ( abstractFlowId ) inherit the graph and customize via an overrides object (connections, schedules, mappings, filters). Use when the same flow structure is deployed across multiple regions, tenants, or environments with different connections or parameters. Quick Reference Flow Type Decision Matrix Pattern Structure Key fields Read schema Linear Flat processor list pageGenerators[] , pageProcessors[] request.yml , page generator.yml , page processor.yml Branching (routers) Routers with conditional branches pageGenerators[] , routers[] + router.yml , branch.yml Abstract / Instance Template + per instance overrides isAbstract: true / abstractFlowId , overrides + overrides helper.yml , overrides.yml Minimum Required Fields Every flow needs at minimum: name display name integrationId parent integration. Without one the flow lands in the account's "Standalone flows" integration (or the create fails with 422 standalone flows not supported ) disabled: true always create disabled pageGenerators[] at least one entry with exportId Either pageProcessors[] (linear) or routers[] (branching) never both Which Schemas to Read Always read: [request.yml](references/schemas/request.yml) base flow fields [page generator.yml](references/schemas/page generator.yml) export sources, per generator schedules, delta coordination [page processor.yml](references/schemas/page processor.yml) import/export steps with responseMapping and hooks Add for branching flows: [router.yml](references/schemas/router.yml) routing strategy, record distribution mode [branch.yml](references/schemas/branch.yml) input filters, per branch processors, chaining Add if response mapping is needed: [response mapping.yml](references/schemas/response mapping.yml) extract/generate pairs for carrying data between steps All available schemas (in [references/schemas/](references/schemas/)): Base fields (all flows): [request.yml](references/schemas/request.yml) Response shape: [response.yml](references/schemas/response.yml) Page generators: [page generator.yml](references/schemas/page generator.yml) Page processors: [page processor.yml](references/schemas/page processor.yml) Response mapping: [response mapping.yml](references/schemas/response mapping.yml) Routers: [router.yml](references/schemas/router.yml) Branches: [branch.yml](references/schemas/branch.yml) Abstract flow helpers: [overrides helper.yml](references/schemas/overrides helper.yml) Instance overrides: [overrides.yml](references/schemas/overrides.yml) Cloning: [clone req