CDF Data Modeling limits: concurrency (429s), pagination, batching, search vs filter, QueuedTaskRunner, and the cap on LLM calls over query results (5 default, 50 max, cached). Triggers: DMS limits, 429, pagination, nextCursor, cdfTaskRunner, instances.query, chat completions over query results.

CDF Data Modeling: Limits, Concurrency & Best Practices This is a reference skill. When writing or reviewing code that calls CDF Data Modeling APIs, apply the patterns below. This skill owns runtime reliability concerns: limits, concurrency, retries, throughput, and batching behavior. For traversal payload correctness and graph specific failure signatures, see dm graph traversal . DMS Limits Reference For the latest concurrency limits, resource limits, and property value limits, see the official documentation: https://docs.cognite.com/cdf/dm/dm reference/dm limits and restrictions Key things to be aware of: Instance apply , delete , and query operations each have their own concurrent request limits Exceeding these limits returns 429 Too Many Requests Transformations consume a large portion of the concurrency budget, leaving less for other clients instances.list has a max page size (use pagination for complete results) instances.query table expressions each have their own item limit instances.upsert accepts up to 1000 items per call in filters accept at most 1000 values per expression; larger sets must be split into batches Search vs Filter: When to Use Which instances.search — Free text search on text properties Use instances.search when you need fuzzy/text matching on string fields (names, descriptions, etc.). It supports an operator parameter: AND (default) — Narrow search. All terms must match. Use when the user provides a specific query. OR — Broad "shotgun" search. Any term can match. Use for exploratory/typeahead search where you want maximum recall. You can combine search with filter to further constrain results with exact match conditions: instances.list / instances.query with filter — Exact match filtering Use filter when you need precise, deterministic matching (equals, range, in, hasData, etc.). No fuzzy matching — values must match exactly. Decision Guide Need Use User typing in a search box instances.search with OR Find a specific item by name instances.search with AND Filter by status, date range, enums filter on list/query Text search + exact constraints instances.search + filter in filter value limit (1000) and batching CDF in filters support a maximum of 1000 values in a single filter expression. If you need to filter against more than 1000 IDs, split values into chunks and issue multiple requests, then merge results. QueuedTaskRunner (Semaphore) Always use the global cdfTaskRunner to wrap CDF API calls. It limits concurrent requests and prevents 429 errors and deadlocks. Source Code If the project does not already have a semaphore utility, create src/shared/utils/semaphore.ts with this implementation: Usage Pattern Always wrap CDF calls with cdfTaskRunner.schedule() : Deduplication with Keys Use the key option to cancel stale requests when the same query is triggered again (e.g., user changes filters quickly): Pagination DMS instances.list returns at most limit items and a nextCursor for the next page. DMS instances.query uses a cursors object keyed by table expression name. instances.list Pagination instances.query Pagination The query endpoint returns nextCursor as a Record<string, string (one cursor per table expression). Use it via the cursors parameter: Pagination + QueuedTaskRunner Combined Always wrap paginated fetches with the semaphore to avoid saturating the concurrency budget: Batching Write Operations When upserting many instances, chunk them to stay under the apply concurrency limit. Each instances.upsert call accepts up to 1000 items. Chunking Utility Batched Upsert with QueuedTaskRunner Batched Delete with QueuedTaskRunner Instance deletes have an even stricter concurrency limit. Use a separate, more restrictive task runner: Hard gate — LLM calls over query results Do not map chat completions over instances.list / query / search hits. Prefer one Atlas / EOS sidebar turn ( integrate fusion agent ). If per item completions are required: 5 per user action, ceiling 50 , cache by space:externalId:lastUpdatedTime , user initiated only. Common Pitfalls 1. Deadlocks from Nested Semaphore Calls If function A holds a semaphore slot and calls function B which also needs a slot, you can deadlock if all slots are occupied. Keep the semaphore at the outermost call level , or ensure inner calls don't go through the same semaphore. 2. Forgetting Pagination DMS returns at most limit items. If you don't paginate, you silently lose data. Always check nextCursor : 3. Unbounded Promise.all Without Semaphore Firing many parallel API calls will hit the 429 limit immediately: 4. Query Limit per Table Expression Each table expression in instances.query has its own limit . If your traversal might return more items than the limit in a single expression, you must paginate using the cursors parameter. 5. Oversized in Filters in filters are capped at 1000 values per expression. Passing more than 1000 values in a single in filter can fail or produce incomplete behavior depending on endpoint/version. Always chunk the values and run batched requests. Summary Checklist [ ] Wrap all CDF API calls with cdfTaskRunner.schedule() [ ] Paginate instances.list calls using cursor / nextCursor [ ] Paginate instances.query calls using cursors / nextCursor when data may exceed limits [ ] Chunk write operations to 1000 items per instances.upsert call [ ] Use a separate, stricter task runner for deletes [ ] Avoid nesting cdfTaskRunner.schedule() calls to prevent deadlocks [ ] Use Promise.all with semaphore wrapped functions, never with raw API calls [ ] Use instances.search for text matching, filter for exact match queries [ ] Split in filter values into batches of at most 1000 and merge responses [ ] LLM over query results capped (5 / max 50) and cached, or not present [ ] Refer to https://docs.cognite.com/cdf/dm/dm reference/dm limits and restrictions for current limits

dm-limits-and-best-practices