Name: configuring-exports
Author: celigo

Configure Celigo export resources -- the data source step that fetches records from external systems. Use when creating or editing exports, choosing the right adaptor type for a target application, setting up delta/incremental syncs, webhooks, file transfers, or lookups.

<! TIER:1 Configuring Exports An export is the data source in a Celigo integration. It connects to an external system and pulls data into the pipeline. Exports serve two roles: Source the starting point that fetches the primary batch of records Lookup a mid flow enrichment step ( isLookup: true ) that fetches additional data per record during processing Both roles are used across flows, APIs, and tools. Beyond fetching data, exports also handle post retrieval processing before records enter the pipeline: Output filter expression based filtering to skip records that don't match criteria Transform Transformation 2.0 expression rules to reshape/flatten response data before mapping preSavePage hook JavaScript processing on the full page of records before they enter the pipeline; the only stage that can change how many records a page carries (split one record into many, or merge several) One to many when used as a lookup, fan out a child array already on the input record. Set oneToMany: true and pathToMany to that array path so each child triggers a separate lookup. Once fanned out, the array element itself is the record, and the parent (without the array) is record. PARENT . This is not resourcePath , which locates records in an API response see [One to many fan out]( one to many fan out the array element is the record) Response mapping when used as a lookup, extract fields from the lookup response back into the record. Configured on the flow's pageProcessors[] entry, but planned when building the lookup export. The response contains a data array and an errors array. Use data[0].fieldName when you expect a single result (e.g., fetching one order by ID). When multiple results are expected, map the whole array with extract: "data" (downstream steps then read it as a normal JSON array, or fan out over it with one to many), or use a lists entry with data[ ].fieldName extracts to build a reshaped array. Do not put data[ ].fieldName in a top level fields[].extract the wildcard is silently ignored there and nothing is merged. See [writing mappings Response Mapping Reference](../writing mappings/SKILL.md response mapping reference transformation 10). Response mapping uses Transformation 1.0 syntax (extract/generate pairs), not the newer expression based transforms postResponseMap hook JavaScript processing after response mapping merges the lookup response back into the record. Configured on the flow's pageProcessors[] entry, but planned when building the lookup export Trace key each record's identity downstream: imports use it to match retries to their errors and as their default lock key, and it never de duplicates. By default it's the first of the top level fields id , internalId , id , internal id or identifier (exact names, in a few capitalizations) that the record has never a nested field. Set traceKeyTemplate when the identity is nested or composite ( {{join " " record.customerId record.orderId}} , without the record. prefix when a transform reshaped the record first). Keys are capped at 256 characters Export Execution Pipeline When a flow runs, each export executes this pipeline in strict order: 1. API request / query / file read fetches raw data from the external system. On a file export an optional preParse hook then gets the raw file text ( preParseData , with fileMeta ) and returns it for the parser; a throw there fails the run 2. Response parsing resourcePath extracts the record array from the response body or file (e.g., http.response.resourcePath for HTTP, file.json.resourcePath for JSON files, XPath for XML) 3. Grouping (optional) rows that share a key become one record, an array of those rows: groupByFields , or Group rows on a NetSuite saved search (see [Grouped records]( grouped records one record many rows)) 4. Transformation (optional) transform reshapes individual records after extraction (Transformation 2.0). On an HTTP export it also sees the response's headers, by lowercase name, so it can carry a header into the records: {{{headers.[x total count]}}} as an extract, or $.headers['x total count'] with inputContext: "envelope" (live verified in a preview and a test run, 2026 10 22). Use triple braces: double braces HTML escape the value 5. Output filter (optional) discards records that don't match filter expression rules 6. preSavePage hook (optional) JavaScript processing on the full page of records Key distinction: resourcePath tells the export WHERE to find records in the response. Transforms reshape WHAT each record looks like after extraction. When a user says "extract records from X" or "treat each X as a separate record", that's almost always a resourcePath change, not a transform. Use transforms when you need to flatten nested objects, rename fields, or restructure individual records. Three Categories of Export Not all exports work the same way. Before building, understand which category you need: Listeners Receive data pushed to Celigo from an external system. No polling, no scheduling the source system sends data when events happen. WebhookExport with type: "webhook" inbound HTTP listener (no connection required). URL, signature checks and responses: [Webhook listeners]( webhook listeners type url signature responses) AS2Export AS2 EDI file reception Distributed exports ( type: "distributed" ) real time listeners for NetSuite and Salesforce. NetSuite: saving the export deploys a Celigo user event script on the record type (the account needs the Celigo SuiteApp or bundle). Salesforce: saving creates nothing in Salesforce the org needs the Celigo managed package plus an Apex trigger on each object that calls it. Integration Apps create their triggers at installation; for a custom flow the customer deploys the trigger, starting from the code the export page shows under Required trigger. celigo connections ping distributed <connectionId checks that the package or SuiteApp responds; its success: false (still HTTP 200) also comes back for an offline connection, so read resBody before telling the user to install anything. Change data capture ( type: "stream" plus a cdc object) streams database changes as they happen: MongoDB change streams ( MongodbExport ), and SQL Server CDC or PostgreSQL logical replication ( RDBMSExport ). A CDC export has no query. The RDBMS form watches rdbms.tables ( database.schema.table on SQL Server, schema.table on PostgreSQL) or, on PostgreSQL only, cdc.publicationName . The save checks the export and the connection's database type, not the database: one whose own change capture isn't set up (PostgreSQL logical replication, SQL Server CDC on the database and each table, a MongoDB replica set) still saves, so when a CDC listener delivers nothing, check the database side first instead set one, not both. Every CDC export must resolve a snapshot mode ( cdc.snapshotMode , or a snapshot.mode entry in cdc.properties ), and a MongoDB one a capture mode too ( cdc.captureMode , or a capture.mode entry). See [cdc.yml](references/schemas/cdc.yml). Topic listeners ( adaptorType: "TopicExport" , type: "stream" ) consume what is published to the Event Streams topic in topic. topicId . No connectionId and no cdc object: with a connection, the save validates the export as that connection's adaptor and fails. Without type: "stream" the export saves but runs as a batch export and never starts on a publish; a PATCH can't add the type later, so update the full document with it, then disable and re enable the flow. A topic listener can't be a lookup. See [topic.yml](references/schemas/topic.yml). type: "stream" is accepted only on those three adaptors ( MongodbExport , RDBMSExport on SQL Server or PostgreSQL, TopicExport ); every other adaptor rejects the save with unsupported export type . A listener is switched by its flow. netsuite.distributed.disabled is maintained by the platform; pause a NetSuite listener by disabling its flow. A topic subscription or a NetSuite listener's deployment becomes active when the flow goes from disabled to enabled a flow created already enabled doesn't activate it. Create the flow disabled and enable it afterwards. After repairing a listener export, or when a NetSuite listener stops firing, disable and re enable the flow ( celigo flows set <flowId disabled=true , then disabled=false ). A webhook listener has no such step: it accepts requests as soon as a flow it feeds is enabled, a flow created enabled included, and refuses them with 403 while every such flow is disabled. When to use: The source system supports outbound webhooks, push notifications, or change data capture, or the records arrive on an Event Streams topic, and you want real time processing. File Transfers Read files from a remote location, then either parse them into records or transfer them as blobs. FTPExport / S3Export / FileSystemExport fetch files from FTP/SFTP, S3, or local filesystem HTTPExport with http.type: "file" read the files in a cloud storage folder (Google Drive, Box, Dropbox, Azure Blob Storage, Google Cloud Storage, Azure Data Lake Storage Gen2, Celigo Storage). On these file storage connectors it's folder mode: relativeURI is a folder path the connector resolves, not an API endpoint an API URL there saves cleanly and fails at runtime with "Directory is not found". A folder picked in the UI's folder browser is stored in file.directory ( pathMode: "directoryId" plus the provider's folder id ); on the drive style providers that id alone selects the folder and relativeURI isn't read, so retarget the export by changing file.directory.id (Google Cloud Storage keeps the bucket in storageRootId and the object prefix in id ). The connector handles auth; the file{} config handles parsing. To fetch one file per record from an API endpoint instead, use a blob lookup ( type: "blob" , isLookup: true ) without http.type: "file" , and response map its data.0.blobKey to blobKey , the field the upload import's blobKeyPath names. NetSuiteExport with netsuite.type: "file" fetch and parse files (CSV, JSON, XLSX, XML, EDI) from the NetSuite file cabinet Parsed mode ( file.output: "records" ) CSV, XML, JSON, XLSX, EDI files are parsed into individual records. Fixed width files, and files whose rows carry a type marker ( HDR / DTL ), parse through a file definition ( file.type: "filedefinition" ); see [building b2b](../building b2b/SKILL.md 5 create file definitions), since a few rule mistakes drop rows without an error. On a file provider export or lookup, each record's source file is also fileMeta exactly fileName , fileSize and lastModifiedTime (epoch milliseconds) read as {{fileMeta.fileName}} in Handlebars or $.fileMeta.<field in an envelope mode transform. Only file provider steps populate it at run time, even where an editor render resolves it Blob mode ( type: "blob" ) binary files transferred as is without parsing. Supported on HTTPExport, NetSuiteExport, SalesforceExport, FTPExport, and S3Export. Source files are deleted after a successful read. An FTP/SFTP or S3 export removes each file it processed from the folder or bucket unless file.skipDelete is true ; ftp.backupDirectoryPath (S3: s3.backupBucket ) moves them there instead, which also takes them out of the folder. Files that fail processing stay. When another system or flow reads the same files, set file.skipDelete: true and narrow the export to its own files with ftp.fileNameStartsWith / fileNameEndsWith (S3: s3.keyStartsWith / keyEndsWith ). A CSV export on SFTP that leaves the files for another reader ( hasHeaderRow and a comma delimiter are the defaults, shown for clarity): When to use: The source system drops files (CSV, EDI, XML, etc.)