AE Data Development and Operations: Data warehouse management, flow orchestration, IDE queries, and data integration

ae dataops Cross skill collaboration When remaining work is outside this skill's scope, or a necessary prerequisite needs another capability, follow [the collaboration protocol](references/collaboration.md). Choose from the skills available in this run by capability, preserve verified context, and continue the remaining task. Reuse this protocol if already loaded. CRITICAL This skill is self contained. Use the Global AE CLI Rules below; do not require a separate shared skill for DataOps side tasks. Start with the user's intended outcome, then read only the matching reference below. These are files in this skill, not separately installed skills. Use the installed command's help for exact flags; if it differs from this reference, report the version mismatch instead of inventing aliases. Global AE CLI Rules Use ae cli and these references for DataOps tasks. Preserve an explicitly selected host and space across a workflow. Global parameters: Parameter Description format <json\ table Output format. Default is JSON. jq <expr jq over the command payload before wrapping; use .flowInstance.status , not .data.flowInstance.status . External jq over default CLI JSON includes .data . host <url Override the active AE host. Available on every command and may be placed after the subcommand, e.g. ae cli dataops ide +<command host <url . Output and errors: JSON results use an outer ok and data ; command errors use ok:false with error.type/message/hint on stderr and exit non zero. Inspect both streams. ok:true is not a business completion check. Legacy commands can return REJECTED , FAILED , or CHECK FAILED inside data . Check the specific result fields described by the reference before proceeding. Reading a failed task's status successfully is different from successfully executing that task. For asynchronous work, preserve the returned ID and query that same operation. Submission is not completion; do not submit again because it is still running. Use bounded polling with backoff; when the wait budget is exhausted, report the ID and current state rather than claiming success or restarting work. For an actual notice.host compat or explicit version compatibility warning, briefly report it and quote the supplied upgrade commands accurately. Unrelated stderr output is not a version warning. Preview semantics: Table field mutations and entity recycle/delete use server semantic dry run ; inspect the returned blockers and planned changes. Other current DataOps dry run implementations show a local request only. They do not validate server permissions, connections, SQL, or business state. For flow release differences use +preview release flow . Current DataOps commands do not implement the global validate server check. Do not treat its informational response as successful validation. Safety constraints: Read commands can execute directly after required IDs and references are verified. Ordinary write commands execute without yes ; use yes only for a high risk write command after explicit user confirmation. Reuse trusted IDs and context already provided; otherwise discover the exact resource using the matching reference. Refine searches or paginate when truncated; an incomplete list cannot prove absence. Ask only when the target or a consequential choice remains ambiguous. For existing resources, never invent IDs, fields, or parameter values. For creation, a new name is expected: use the requested name or propose one, check collisions, and do not mistake its absence for a blocker. Stop at the requested outcome: saving DEV does not authorize release, release does not authorize a manual run, and recycling does not authorize permanent deletion. Do not substitute a broader operation for an unsupported narrower request. Domains for DataOps: dataops repo , dataops datatable , dataops flow , dataops operations , dataops ide , dataops integration Core Concepts and Rules You must understand the following key concepts before use, otherwise errors are highly likely. Environment and Defaults Scenario Default Environment Description Table creation/field edits and workflow definitions DEV Release separately when requested dataops datatable +entity recycle / +recycle bin delete Existing DEV and PRODUCT mappings Entity lifecycle operations; not DEV only edits awaiting release Manual flow execution, operations and backfill PROD Inspect the exact flow/job/instance SQL query Determined by the SQL's discovered tables Do not choose DEV or PROD solely from a sample query Responsibility Boundaries Operation Correct Tool Prohibited Execute SELECT queries dataops ide — Create tables/views or mutate table fields dataops datatable dataops ide Recycle or permanently delete a table/view dataops datatable +entity recycle / +recycle bin delete dataops ide Flow Lifecycle Custom flow parameters: use +get flow params , +create flow param , +update flow param , and +delete flow param in dataops flow ; see [the flow reference](references/dataops flow create.md custom flow parameters). Writes configure DEV definitions and may stop DEV debug executions. Value, type, name, and deletion changes require release for PROD; a remark update without renaming also synchronizes an existing PROD parameter remark. Backfill lifecycle: Discover eligible PROD flow → Create or fully update DRAFT job → Run explicitly → Search / inspect plans → Stop or rerun the complete job; delete only after target inspection Scenario Routing User intent Command group Reference Completion boundary Discover or select a space dataops repo [Space discovery]( 1 space discovery) Resolve the intended spaceCode ; reuse trusted context Create tables/views or change a field type/comment dataops datatable [Table workflows](references/dataops table.md) Verify DEV state; publish only when requested Delete a table/view; inspect or empty one recycle bin entry dataops datatable [Entity lifecycle](references/dataops table.md entity lifecycle scope and safety) Ordinary deletion stops at recycling; permanent deletion needs separate authorization Create/configure a flow; add/update/delete tasks and parameters; schedule or release dataops flow [Flow configuration](references/dataops flow create.md) Read back the requested DEV or PROD definition; do not add a manual run Execute, inspect, or stop a run; troubleshoot failed work dataops flow , dataops operations [Execution and monitoring](references/dataops flow monitor.md) Match the exact instance; whole flow execution is not failed node retry Fill multiple historical business dates dataops operations [Backfill jobs](references/dataops backfill.md) Draft creation, job start, and completed plans are different outcomes Create/test a datasource or configure/run a sync dataops integration [Data integration](references/dataops integration.md) Verify saved config or the exact run, according to the request Schedule a sync and publish it without running now dataops integration , dataops flow [Data integration](references/dataops integration.md), then [Flow configuration](references/dataops flow create.md) Confirm the PROD node and schedule; no manual execution Browse schemas/tables or answer a data question dataops datatable , dataops ide [Query workflow](references/dataops query.md) Read the actual result and answer the question, not merely return a download task ID 1. Space Discovery dataops repo exposes only one read command. Use it to discover a valid spaceCode before calling DataOps commands that require one. It returns createTime , spaceCode , and spaceDisplayName . If the user already provided a trusted spaceCode , reuse it. If spaceCode is unknown, run +list spaces first. If exactly one space is returned, use its spaceCode . If multiple spaces are returned and the user intent does not identify one, ask the user which space to use. Do not guess. 2. Data Table and View Management Detailed workflow, command flags, examples, and parameter notes live in [ references/dataops table.md ](references/dataops table.md). Key constraints: Start with dataops datatable +dict search tables for visible DataOps catalog discovery. Use dataops ide +search tables only for raw engine metadata, and dataops ide +ide list tables only for known catalog/schema browsing. Create tables/views with dataops datatable , not dataops ide ; creation is DEV only. Use +publish entity when publication is requested. Add, modify, or delete one ordinary table field at a time in DEV; preview first, never mutate partition fields, and publish PROD separately. An ordinary "delete table/view" request means recycle only with dataops datatable +entity recycle . Permanent deletion is a separate workflow using +recycle bin list and +recycle bin delete ; read the table reference before either workflow. Both writes affect DEV and PRODUCT and require an exact entity ID plus name, semantic dry run , and explicit confirmation before yes . Never automatically delete an old same name recycled entity to bypass RECYCLE NAME CONFLICT . After confirmed actual recycling, the Agent may ask whether to permanently delete the entity, warning that this is irreversible and may delete internal table data. Stop until a new explicit confirmation. This is an Agent reply, not a CLI prompt or JSON field. A preview, FAILED , PARTIAL , or unverified state must not trigger this offer; reconcile the same entity ID first. DDL follows Trino syntax; current space view DDL should keep the literal ${env} placeholder. 3. Flow Orchestration Flow references cover creation and configuration , execution and monitoring , and backfill jobs . Lifecycle: DEV configuration → Requested release → Requested execution or troubleshooting. Stop at the user's boundary. Detailed creation/configuration commands live in [ references/dataops flow create.md ](references/dataops flow create.md). Detailed execution, monitoring, operation instance, task log, and stop commands live in [ references/dataops flow monitor.md ](references/dataops flow monitor.md). Persistent multi date backfill jobs live in [ references/dataops backfill.md ](references/dataops backfill.md). Key constraints: Create and update tasks in DEV, preview/release before PROD execution. Use +get flow overview for custom flowParams and each task's dag.tasks[].taskParams ; DEV is the default, PROD is supported, and both require dwWorkflowEdit . Definitions preserve expression text and code parsed references whose source/value fields are null or omitted. Instance details require dwOMInstanceView ; inspect historical values in flowInstance.instanceParamMap or task log.params with includeLog true . Treat +delete task as high risk: verify the target with +get flow overview , preview with dry run , and use yes only after explicit user confirmation. Deletion affects DEV; release the flow to apply it to PROD. +execute flow always runs PROD; it returns executeId for early stop. Prefer flowInstanceId from operations search for stable inspection and troubleshooting. A backfill job is persistent and batches multiple base dates; do not emulate it by looping +execute flow . Create a draft with +create backfill job and run it as a separate step. +rerun backfill job reruns the complete job, not only failed plans. +update backfill job replaces a DRAFT job's complete configuration; inspect the job first and do not treat it as a partial patch. Treat +delete backfill job as high risk and preview it with dry run before confirmation. Reference flow or workspace parameters in task

ae-dataops