Name: configuring-lookup-caches
Author: celigo

Configure Celigo lookup cache resources -- in-memory key-value stores used for fast lookups, deduplication, cross-reference resolution, and state tracking during flow execution. Use when creating caches, loading data, referencing caches in import lookups, or managing cache lifecycle.

<! TIER:1 Configuring Lookup Caches A lookup cache is an in memory key value store managed by Celigo for fast data retrieval during integration processing. Unlike export based lookups that query external systems per record, lookup caches provide sub millisecond access to pre loaded reference data without consuming API calls. Lookup caches handle three concerns: Cache metadata name, description, and whether data is included in templates/cloning Data management loading, retrieving, updating, and purging key value entries via dedicated data endpoints Size governance each cache has a 50 MB limit; the per environment aggregate limit across all caches is 1 GB Lookup caches are consumed by import and export mappings through a lookup entry. The entry references a cache by lookupCacheId and optionally extracts a specific field from the cached object via the extract JSON path, and a mapping row uses it by name. On an HTTP import the entry goes in http.lookups ; on an export, in its Transformation 2.0 rules. This is configured on the import/export resource, not on the cache itself. A request template can also read a cache directly with {{getLookupCacheData}} ([Reading a Cache from a Template]( reading a cache from a template)). Used across flows, APIs, and tools. "Lookup" Which One Is Meant? "Lookup" is overloaded in Celigo, and a lookup cache is only one thing the word can mean. When a request just says "the lookup," confirm which one before acting they live in different places and are configured differently. A single entry in lookups[] resolves its value one of three ways: Cached the entry sets lookupCacheId to read a lookup cache (this skill): pre loaded reference data read in memory, no per record external call, as fresh as the last data load. Static map the entry carries an inline map object. No cache and no external call; the table travels with the import/export. Best for a handful of stable pairs that rarely change. Live / dynamic the entry issues a per record query against a connected system (an HTTP request, SOQL, a NetSuite search, SQL). Always fresh answers, at the cost of one external request per record. Quick test for which is meant: pre loaded data read in memory is a lookup cache ; an answer that must reflect the source system right now is a live/dynamic lookup; a small fixed table living on the step is a static map . The same word also names things outside this skill: a mid flow lookup step (an export with isLookup: true that fetches from an external system between flow steps) and transform lookups (tables inside an export transform's rules). "Lookup" alone is never enough listen for whether the subject is stored cache data, an inline rule on a step, a live query, or a separate flow step. Use Cases Lookup caches serve several distinct patterns: Cross Reference / ID Mapping The most common use case. Map external IDs to internal IDs across systems (e.g., Shopify product IDs to NetSuite internal IDs, external customer GUIDs to Salesforce Account IDs). Avoids repeated API lookups during import processing. Deduplication / Debouncing Track which records were already processed so a repeat is skipped. The cache can't do this on its own: a write is an upsert with no insert if absent, so two runs that both read "not there" both proceed. Build it as a [Deduplication Ledger]( deduplication ledger) and when the destination itself can answer "does this exist?", match there instead ([configuring imports Writing Exactly Once](../configuring imports/SKILL.md writing exactly once)). State Tracking Maintain timestamps, batch IDs, or processing markers across flow runs. Common patterns: "last sync time" caches and "open batch ID" holders. A cache is not a lock table it has no compare and set; serialize work per key with an import's idLockTemplate instead. The state store keeps such markers per export, import or integration too, for scripts that already use it or plans without lookup caches ([references/state store.md](references/state store.md)); a script reads and writes them with state.getForResource({ resourceType, id, key }) and state.putForResource({ resourceType, id, key, value }) from integrator api , and a value must be a non empty object or array. resourceType is the plural family ( exports , imports , integrations ), and getForResource throws State not found for key. until the key's first write, so read it in a try / catch that treats that error as the first run. Static Reference Data Store relatively stable reference tables product catalogs, category hierarchies, shipping overrides, zip code lookups. Set includeDataInTemplatesAndCloning: true when this data is part of the integration's configuration. Quick Reference Minimum Required Fields Every lookup cache needs at minimum: name . Optional but common: description , includeDataInTemplatesAndCloning . A lookup cache is an account level resource that lives in one environment. It has no writable integrationId ; one sent on create or update is silently dropped. To tie a cache to an integration, register it ( celigo integrations register lookup caches <integrationId <cacheId , or PUT /v1/integrations/{id}/lookupcaches/register with an array of cache ids), or create it already registered with POST /v1/integrations/{id}/lookupcaches ; no MCP tool registers one. Register a cache on every integration whose hooks or templates read it: a hook's integrator api calls reach only caches registered to the flow's own integration, and {{getLookupCacheData}} only caches with an alias there. Registration adds that aliases[] entry on the integration a slug of the cache name, lowercased with spaces and punctuation removed ( Product SKU Map becomes productskumap , with 2 , 3 , ... added on a collision). Give a cache an externalId when automation needs to find it again: GET /v1/lookupcaches?externalId=<value returns only the matching cache ( celigo platform get "v1/lookupcaches?externalId=<value " ). Data Operations All data operations use the cache ID. Data entries are key value pairs: keys are strings (1 255 characters, case sensitive, with no tab, newline, carriage return or NUL); values may be null, a number, a boolean, a string, or an object, and read back with the type they were written with. Store complex data as an object the consuming lookup's extract JSON path pulls fields from it. A top level array value is rejected (400 invalid value ), so wrap it ( {"items": [...]} ). Operation Method Endpoint Notes Upsert data POST /v1/lookupcaches/{id}/data Body: { "data": [{ "key": "k", "value": "v" }] } . MCP: upsert lookup cache data Get data POST /v1/lookupcaches/{id}/getData No body = one page of up to 1000 entries, unordered, with a nextPageURL while more remain (gotcha 5). With { "keys": [...] } (up to 1000) or { "startsWith": "prefix" } (paged the same way). MCP: list lookup cache data Delete keys DELETE /v1/lookupcaches/{id}/data Body: { "keys": ["k1", "k2"] } . MCP: delete lookup cache data with keys Purge all DELETE /v1/lookupcaches/{id}/data/purge Removes every entry, irreversibly. MCP: delete lookup cache data with all: true , which runs with no confirmation step "Update the cache" can mean its entries (these operations) or the resource's name, description and clone flag ( celigo lookup caches update / set , MCP upsert lookup cache ). Ask which when the request doesn't say. Schema Index All schemas are in [references/schemas/](references/schemas/): Base fields (create/update): [request.yml](references/schemas/request.yml) name, description, includeDataInTemplatesAndCloning, externalId Response shape: [response.yml](references/schemas/response.yml) includes size, sizeInMB, timestamps, and template or clone origin; no integration field (an integration lists its caches in registeredLookupCacheIds ) and no entry count ( size is bytes, sizeInMB rounds up in 5 MB steps), so answer "what's in it" from the data, never from the size Referencing Caches in Lookups Lookup caches are consumed through a lookup entry on the import or export that needs the data. See the writing mappings skill for full lookup configuration details. The key fields: name unique identifier for the lookup within the resource; a mapping row names it in lookupName lookupCacheId references the cache resource extract optional JSON path to pull a specific field from the cached value (e.g., $.details.price ); without it the whole value is returned default fallback value when the key is not found; a save refuses it beside allowFailures: false (422 Lookups with default set can not have allowFailures explicitly set to false. ) allowFailures false fails the record on a missing key ( lookupcache evaluation failed ); true uses the default ; left out, the record still succeeds, and with no default the field is silently missing. Set false explicitly when a missing key must stop the record On an HTTP import the entry goes in http.lookups : the save keeps that array and mirrors it to the top level lookups , and drops a lookups array sent only at the top level. Save it with a full PUT of the import GET it, add the entry and the row's lookupName , and send the whole document back because an import's PATCH accepts only debugUntil . The mapping row's extract is the key the cache is read with: An export has no lookups of its own. Its cache lookups live in its Transformation 2.0 rules, transform.expression.rulesTwoDotZero.lookups , beside the mappings that name them; those entries take a lookupCacheId (or a static map ) with extract , default and allowFailures , and no live request. Reading a Cache from a Template {{getLookupCacheData "<alias " <key "<pathToValue " "<default "}} (the last two optional) reads a cache from a step's Handlebars template, such as an HTTP request body, with no lookup configured on the step. It finds the cache by its alias on the flow's integration ([Minimum Required Fields]( minimum required fields)), matched case sensitively: a cache id or a differently cased alias fails the record with handlebars template parse error ( getLookupCacheData: alias "<value " not found in integration/flow aliases. ). A key that isn't in the cache renders the default, or an empty string when there's none, and the record goes on. The Handlebars processor has no integration to read from, so test a template that uses it in a flow test run. Related Skills [writing mappings Add lookups for value translation](../writing mappings/SKILL.md 6 add lookups for value translation) configuring lookups[] with lookupCacheId , extract , and allowFailures [writing handlebars Quick Reference](../writing handlebars/SKILL.md quick reference) Handlebars in mapping extracts and request templates [configuring imports How to Build an Import](../configuring imports/SKILL.md how to build an import) imports are the primary consumer of lookup caches [configuring exports Quick Reference](../configuring exports/SKILL.md quick reference) lookup exports ( isLookup: true ) as an alternative to caches for live data [building flows How to Build a Flow](../building flows/SKILL.md how to build a flow) wiring lookups into flow pipelines <! TIER:2 How to Build a Lookup Cache 1. Determine the use case What data needs to be cached? Cross reference IDs, static reference tables, deduplication keys, or processing state? This determines data structure, loading strategy, and lifecycle. 2. Check for existing caches Before creating a new cache, search for existing ones in the account: 3. Create the cache Create the cache metadata first. Data is loaded separately. Through MCP, upsert lookup cache creates th