Build Celigo flows -- pipelines that move data from source systems to destination systems on a schedule or in response to events. Covers scheduling, chaining, error management, and abstract/instance templating. Use when creating, editing, or debugging flows.

<! TIER:1 Building Flows A flow moves data from one or more source systems to one or more destination systems. It runs on a schedule, in response to events (webhooks, listeners), or when triggered by another flow. Flows are the primary way integrations get work done in Celigo. A flow has page generators (exports that fetch data) and page processors (imports and lookups that process each record). Processors run sequentially in a flat list, or conditionally through routers that branch records to different paths. These processing pipeline mechanics routers, branches, page processors, response mapping are shared with APIs and tools (see building apis and building tools ). What Starts a Flow Flows start themselves this is the biggest thing that separates them from APIs (invoked by an HTTP caller) and tools (invoked by a consumer). Every flow begins with one or more page generators, of two kinds: Scheduled exports the flow runs on a cron cadence and each run pulls from the source: everything (full sync), only what changed since the last run (delta sync), records matching a query, or files landed in an FTP/SFTP/S3 folder. The right primitive for batch work: nightly reconciliations, hourly delta syncs, backfills, off peak windows. Listeners the source pushes to the flow. A webhook fires (or a NetSuite/Salesforce native real time event triggers) and the payload immediately starts flowing. No schedule; the flow runs as events arrive. The right primitive for event driven work ("when X happens, do Y"), especially when latency matters. A flow can mix both, and multi generator designs are common: Real time plus batch safety net a listener catches events as they fire; a scheduled export reconciles at off peak hours, catching up after webhook outages Consolidating sources customers from Salesforce AND HubSpot, each with its own generator, feeding the same downstream pipeline Different slices of the same source one export pulls new records, another pulls updated records, when the API exposes them separately If the requirement is "every night at 2 AM, do X" or "when a webhook arrives, do Y" that lives on a flow. APIs and tools have no schedule and no listener; they only run when invoked. Fetched Data Needs a Downstream Consumer A common design mistake: ending a flow on an import that fetches data back from a remote system (a preview call, a query, a lookup shaped POST) and relying on response mapping to capture the result. Response mapping makes fields visible to the NEXT step if no next step exists, the captured data is discarded when the run ends and nobody sees it. When the requirement says "preview / estimate / retrieve / fetch / check / look up", the design needs at least one of: A write back import to the source system (most common) e.g. source export preview import update import that writes the captured fields onto the source record A persistent destination the user named (file to S3/SFTP, email, database) A router or AI agent step that consumes the captured data within the same run A two step export fetch shaped import flow with nothing after it is a smell re read the intent for where the fetched data should end up. The same applies in reverse: capturing a created record's ID via response mapping is only useful if a later step writes it somewhere. Flow Topologies Linear Flows A flat pageProcessors[] list with no routers. One or more page generators feed records through a sequential chain of page processors. Each processor is either an import ( type: "import" ) or a lookup export ( type: "export" ). Records pass through every step in order. Unique to flows APIs and tools always use routers. Branching Flows Page generators feed records into routers[] instead of pageProcessors[] . Each router evaluates records against branch conditions and routes them to matching branches. Branches contain their own pageProcessors[] and can chain to other routers via nextRouterId . Two routing modes (shared with APIs and tools): Input filters ( routeRecordsUsing: "input filters" ) S expression rules on each branch; last branch can omit filter as a catch all Script based ( routeRecordsUsing: "script" ) a JavaScript function returns the branch name Flows support both first matching branch and all matching branches routing. APIs only support first matching branch . Tools support first matching branch only. A flow uses EITHER pageProcessors (linear) OR routers (branching) at the top level not both. When a branching flow needs linear steps before the branch point (e.g., a lookup enrichment or AI classification that all branches depend on), use a pass through router : a single branch router with nextRouterId pointing to the branching router. Omit routeRecordsTo and routeRecordsUsing on the pass through router including them makes it appear as a filter based branch in the UI. The API defaults are sufficient. Abstract / Instance Flows A template/inheritance model. An abstract flow ( isAbstract: true ) defines the complete graph but cannot execute. Instance flows ( abstractFlowId ) inherit the graph and customize via an overrides object (connections, schedules, mappings, filters). Use when the same flow structure is deployed across multiple regions, tenants, or environments with different connections or parameters. Quick Reference Flow Type Decision Matrix Pattern Structure Key fields Read schema Linear Flat processor list pageGenerators[] , pageProcessors[] request.yml , page generator.yml , page processor.yml Branching (routers) Routers with conditional branches pageGenerators[] , routers[] + router.yml , branch.yml Abstract / Instance Template + per instance overrides isAbstract: true / abstractFlowId , overrides + overrides helper.yml , overrides.yml Minimum Required Fields Every flow needs at minimum: name display name integrationId parent integration disabled: true always create disabled pageGenerators[] at least one entry with exportId Either pageProcessors[] (linear) or routers[] (branching) never both Which Schemas to Read Always read: [request.yml](references/schemas/request.yml) base flow fields [page generator.yml](references/schemas/page generator.yml) export sources, per generator schedules, delta coordination [page processor.yml](references/schemas/page processor.yml) import/export steps with responseMapping and hooks Add for branching flows: [router.yml](references/schemas/router.yml) routing strategy, record distribution mode [branch.yml](references/schemas/branch.yml) input filters, per branch processors, chaining Add if response mapping is needed: [response mapping.yml](references/schemas/response mapping.yml) extract/generate pairs for carrying data between steps All available schemas (in [references/schemas/](references/schemas/)): Base fields (all flows): [request.yml](references/schemas/request.yml) Response shape: [response.yml](references/schemas/response.yml) Page generators: [page generator.yml](references/schemas/page generator.yml) Page processors: [page processor.yml](references/schemas/page processor.yml) Response mapping: [response mapping.yml](references/schemas/response mapping.yml) Routers: [router.yml](references/schemas/router.yml) Branches: [branch.yml](references/schemas/branch.yml) Abstract flow helpers: [overrides helper.yml](references/schemas/overrides helper.yml) Instance overrides: [overrides.yml](references/schemas/overrides.yml) Cloning: [clone request.yml](references/schemas/clone request.yml), [clone response.yml](references/schemas/clone response.yml) Related Skills [configuring connections Quick Reference](../configuring connections/SKILL.md quick reference) connection types and auth methods for page generators and processors [configuring exports Quick Reference](../configuring exports/SKILL.md quick reference) building exports used as page generators and lookup processors [configuring imports Quick Reference](../configuring imports/SKILL.md quick reference) building imports used as page processors [writing mappings Mapper 2.0 Workflow](../writing mappings/SKILL.md mapper 20 workflow) field mappings on imports and response mapping between steps [writing scripts Data Pipeline Hooks](../writing scripts/SKILL.md data pipeline hooks) preSavePage, preMap, postMap, postSubmit, postResponseMap hooks [writing handlebars Quick Reference](../writing handlebars/SKILL.md quick reference) dynamic values in URIs, filters, delta tokens, SQL queries [troubleshooting flows Diagnostic Workflow](../troubleshooting flows/SKILL.md diagnostic workflow) diagnosing flow failures, errors, and performance issues <! TIER:2 How to Build a Flow 1. Plan the flow Before creating anything, decide what kind of operation this is: Decision tree: Modifying an existing flow's step config (export settings, import mappings, scripts) work on the step directly, not the flow. Use celigo exports set , celigo imports set , or the relevant skill (configuring exports, configuring imports, writing scripts, writing mappings) Modifying an existing flow's structure (add/remove steps, change schedule, rename) GET the flow, modify the structure, PUT it back. Don't rebuild from scratch Building a new flow where every step is known build directly, bottom up (skip to step 2) Any ambiguity about what steps are needed design first. List every system, every data direction, every step before writing any JSON Design checklist (when ambiguity exists): What source systems? What destination systems? What data moves between them, in which direction? How often? (cron schedule, webhook trigger, on demand) What happens when a step fails? ( proceedOnFailure , error notifications) Do downstream steps need data from upstream responses? (response mapping) Is this a one off or a reusable template? (abstract/instance flow) Sandbox or production? (never mix sandbox: true flows only use sandbox: true connections) 2. Identify the integration Every flow belongs to an integration (the container). Find or create the integration first. 3. Check for existing patterns Before building from scratch, check what already exists in the account and marketplace. The account index auto refreshes when stale ( 4 hours). Force a fresh snapshot with celigo account snapshot . 4. Build the connections, exports, and imports Flows reference existing resources. Build bottom up: connections first, then exports and imports that use those connections, then the flow that wires them together. For every step, match the adaptor to the target application raw HTTP is the fallback, not the default. Use the native adaptor when one exists (NetSuite, Salesforce, databases, FTP/S3); otherwise check for a pre built HTTP connector (550+ apps: celigo http connectors list ) and build the connection from it; hand write HTTP config from public API docs only when no connector exists or it doesn't cover the endpoint. See [configuring exports Check for a pre built connector](../configuring exports/SKILL.md 3 check for a pre built connector) and [configuring imports Check for a pre built connector](../configuring imports/SKILL.md 3 check for a pre built connector). See configuring exports and configuring imports for how to build each resource. 5. Choose the topology Scenario Topology All records follow the same path Linear ( pageProcessors ) Records need conditional routing by field values Branching with input filters Routing logic requires custom JavaScript Branching with script router Records should fan out to all matching paths Branching with all mat

building-flows