Name: configuring-imports
Author: celigo

Configure Celigo imports -- the destination step that writes records to external systems. Use when creating imports, choosing the adaptor type, setting up field mappings, lookups, upsert logic, AI agent imports, or file-based imports.

<! TIER:1 Configuring Imports An import is the data destination in a Celigo integration. It takes records from an upstream step and writes them to an external system REST APIs, databases, ERPs, file servers, or AI models. Every import has one adaptor type and, apart from the connection less kinds in [Minimum Required Fields]( minimum required fields), is bound to exactly one connection. Imports handle six concerns: Field mapping transforming source fields into the destination system's expected format (including value resolution via static maps and lookup tables). Uses Mapper 2.0 ( mappings[] array) by default, including NetSuiteHTTPImport . SalesforceImport and WrapperImport support only Mapper 1.0 ( mapping.fields[] / mapping.lists[] ), and NetSuiteDistributedImport uses its own Mapper 1.0 dialect at netsuite da.mapping Operation logic create, update, upsert, delete, attach/detach Hooks JavaScript pre/post processing at various pipeline stages (preMap, postMap, postSubmit). File based imports that generate files from records also support postAggregate; an HTTP records import refuses it (422 invalid hook ) One to many fan out a child array already on the input record. Set oneToMany: true and pathToMany to that path (e.g., "lineItems" ) when one source record should create one import operation per child. After fan out the child is record ( {{record.sku}} ) and the parent (without the array) is record. PARENT ( {{record. PARENT.id}} ). Leave pathToMany blank when the arriving record is itself an array (grouped rows from a groupByFields export or a file group) each element then becomes the record, with no PARENT . A path on an array record processes zero children and still reports success; a blank path on an object record fails with oneToMany error . oneToMany doesn't batch by itself: templates see batch of records only when http.batchSize is above 1 (the default varies by API) or an RDBMS query runs per page Response mapping extract fields from the import's API response back into the record for downstream steps. Configured on the flow's pageProcessors[] entry, but planned when building the import. Extracts read the response envelope: id (the destination assigned record ID), statusCode , ignored , errors , headers , dataURI (the record's link, when the import sets dataURITemplate ), and json (the response body after any response transform, whose top level fields are also flattened onto the envelope). Use id for a created record's ID and json.fieldName for other response fields (e.g., json.output.1.content.0.text for OpenAI responses) json.id works only when the body itself has a top level id . Flow test runs never send imports, so id is empty there. Response mapping uses Transformation 1.0 syntax (extract/generate pairs), not the newer expression based transforms postResponseMap hook JavaScript processing after response mapping merges the response back into the record. Configured on the flow's pageProcessors[] entry, but planned when building the import. Use to transform or enrich the merged record before downstream steps Imports are used across flows, APIs, and tools. Import Execution Pipeline When records arrive at an import step, this pipeline executes in strict order: 1. Input filter (optional) records that don't match skip this import (see [Gotchas]( gotchas)). Configured as filter on the import resource itself an inputFilter key on an import is silently discarded. See [configuring filters](../configuring filters/SKILL.md import filter import filter) 2. preMap hook (optional) JavaScript processing before field mapping 3. Field mapping Mapper 2.0 or 1.0 maps source fields to destination fields, including lookups and hardcoded values 4. postMap hook (optional) JavaScript processing after field mapping, before submission 5. Submit to destination writes the mapped record to the external system 6. postSubmit hook (optional) JavaScript processing after the destination responds (access response data, log results, trigger side effects) 7. Response transform (optional, responseTransform ) reshapes the response body ( json ) after postSubmit. Its expression rules are Transformation 1.0 only ( expression.version: "1" ; "2" is refused with 422), and it keeps only the fields it maps, so a field it drops is gone for response mapping. Its execution log stage is responseTransformation 8. Response mapping (optional) carries data from the destination response back into the record for downstream steps. Configured on the flow's pageProcessors[] entry, not on the import itself 9. postResponseMap hook (optional) JavaScript processing after response mapping merges response data back into the record Key distinction: Response mapping lives on the flow's pageProcessors[] entry, not on the import resource. preMap, the mapping and postMap shape only the request this import sends: the next step receives the record as it arrived plus what response mapping and postResponseMap add, so a field added in steps 2 4 never reaches it. When building an import that needs to pass data downstream, plan the response mapping at flow design time. What an HTTP import's templates see. http.relativeURI and header values render against the record before mapping, and an undefined path fails the request: a field the record lacks in the URI or a {{lookup.<name }} whose lookup found nothing fails the record before anything is sent, with handlebars template parse error ( Failed to generate request url from template: … Details: "<field " not defined in the model. ). That is missing data, not a template bug, so a retry fails the same way; an update import that some records reach without the destination id needs a composite UPDATE/CREATE or ignoreMissing (Common Errors). A field holding "" is defined: it renders empty, and the request goes out without the id. A field holding null renders the text null instead (the request goes to …/null ), and a field under an object that is null fails the record like a missing one ( "<field " not defined in null ) see writing handlebars, gotcha 16. http.body renders against the mapped record, where an undefined path renders empty and the record still succeeds ( strictHandlebarEvaluation doesn't change that). Leave http.body unset to send the mapped record; when you template it, use the mapped (destination) field names, and double braces for string values in a JSON body (gotcha 11). Categories of Import Record Based Imports Submit structured records to APIs, databases, or ERPs. The vast majority of imports. NetSuiteDistributedImport high performance SuiteApp writes (add, update, addupdate, delete, attach, detach). The default for NetSuite when the Celigo SuiteApp or bundle is installed. netsuite da.restletVersion is fixed when the step is created, so author new steps on suiteapp2.0 ; a hook's SuiteScript version must match the step's NetSuiteHTTPImport NetSuite's REST APIs on a netsuite connection ( nsDomainType: "suitetalk" : POST /services/rest/record/v1/<type to insert, PATCH .../<type /{id} to update; "restlet" for a custom RESTlet). Use it when neither the SuiteApp nor the bundle is installed, or the user asks for REST HTTPImport REST/GraphQL APIs (POST, PUT, PATCH, DELETE). Supports connector assisted ( formType: "assistant" ) and GraphQL ( graph ql , assistant graphql ) modes; see [GraphQL imports]( graphql imports) SalesforceImport Salesforce writes through salesforce.api : soap (the default), rest , or compositerecord (Composite Graph, a tree of related records in one request); metadata deploys configuration, not records. salesforce.api has no Bulk value RDBMSImport SQL databases (Snowflake, PostgreSQL, MySQL, SQL Server, Oracle). rdbms.queryType is a one element array whose value fixes the companion field: per record , per page (renders against batch of records ) and first page (runs once, on the first page) take rdbms.query , an array of SQL strings; bulk insert takes rdbms.bulkInsert (pure INSERT, with no upsert or ignore existing logic); bulk load takes rdbms.bulkLoad (a staged load that becomes a MERGE with primaryKeys , or runs your own SQL with overrideMergeQuery ). Prefer bulk load where the database supports it, then bulk insert for pure INSERTs, then per page , and per record only when the logic can't be batched. The legacy uppercase INSERT (and the retired MERGE , COPY and BULK INSERT ) fail any save with 422 "Import query types … are deprecated"; UPDATE is legacy too, so don't use it. Other strings, such as lowercase insert , fail as an invalid enum value. A bulk load import's GET also returns a file block describing its staging file; the platform derives it on every read, so a save that changes it is accepted and discarded. The SQL itself: [writing sql](../writing sql/SKILL.md) MongodbImport , DynamodbImport , JDBCImport other databases GraphQL imports A GraphQL import is an HTTPImport with http.formType graph ql (or assistant graphql , the connector assisted GraphQL form). Each http.body[] entry is a string holding the {"query", "operationName", "variables"} envelope, one per operation on an update/create pair never an object, which is stored as is and doesn't render and the mutation's input comes from the record through Handlebars in variables . Endpoint. The import posts to the connection's base URI its form has no relative URI field and writes relativeURI: ["/"] on every save so the connection must supply the GraphQL endpoint: a graph ql connection, or a connector connection bound to the app's GraphQL API. On a connection bound to the app's REST API every record fails (HTTP 406 on Shopify); add a second connection bound to the GraphQL API, re point the GraphQL steps to it, and leave the shared REST connection unchanged. The form type is set on the step, so a plain http connection whose base URI is the endpoint works too. Variables. Under graph ql or assistant graphql , a variables value stored as a JSON string is sent as an object; under http it goes out as a string. An object takes expressions inside string values only: "variables": {"input": {"email": "{{record.email}}"}} . A string can also carry bare values, but its string values sit two JSON levels deep, so give each one jsonEncode in double braces: "variables": "{\"input\": {\"note\": \"{{jsonEncode record.note}}\", \"quantity\": {{record.qty}} } }" . An object or array goes in as {{jsonSerialize record.items}} , also in double braces. With {{record.note}} , or triple braces on either helper, a value holding a quote fails its record with 422 invalid json before anything is sent. Errors come back as HTTP 200. A GraphQL server usually answers a failed mutation with 200 and an errors array, and with no response settings the import counts the record as written. Set http.response.resourceIdPath to the id only a successful mutation returns ( ["data.customerCreate.customer.id"] ): any other answer then fails the record with missing id , top level errors and the mutation's own userErrors alike. For the server's message on top level errors, also set failPath: ["errors[0].message"] and errorPath: "errors[0].message" ; failPath: ["errors"] catches a response with exactly one error and lets two or more through. File Based Imports Write or upload files to remote storage. Require the file{} configuration block. Two modes: Record to file aggregates incoming records into a file (CSV, JSON, XML, XLSX). The file{} block defines the output format, and file.aggregation.type how a run's records become files: all (one file per run), page (one per page) or key (EDI file definitions only, with B2B Manager). Send the legacy file.skipAg