Name: databricks-jobs
Author: databricks

Develop and deploy Lakeflow Jobs on Databricks via DABs, Python SDK, or the CLI. Use when creating data engineering jobs with notebooks, Python wheels, SQL, dbt, or pipelines. Invoke BEFORE starting implementation.

Lakeflow Jobs Development FIRST : Use the parent databricks core skill for CLI basics, authentication, profile selection, and data exploration commands. Compute : jobs run on serverless by default. Before you choose compute or write any cluster setting, use the databricks serverless skill; it decides serverless or classic and the performance mode. Lakeflow Jobs orchestrate data workflows with multi task DAGs, flexible triggers, and comprehensive monitoring. Jobs support diverse task types and can be managed via Declarative Automation Bundles (DABs), Python SDK, or CLI. Reference Files Use Case Reference File Configure task types (notebook, Python, SQL, dbt, pipeline, JAR, run job, for each) [references/task types.md](references/task types.md) Set up triggers and schedules (cron, periodic, file arrival, table update, continuous) [references/triggers schedules.md](references/triggers schedules.md) Configure notifications, health rules, retries, timeouts, queues [references/notifications monitoring.md](references/notifications monitoring.md) Complete worked examples (ETL, warehouse refresh, event driven, ML training, multi env, streaming, cross job) [references/examples.md](references/examples.md) Scaffolding a New Job Project Use databricks bundle init with a config file to scaffold non interactively. This creates a project in the <project name / directory: project name : letters, numbers, underscores only bundle init also creates an AGENTS.md in the project. Customize that file for the project instead of creating new agent instruction files. Project Structure Quick Start Declarative Automation Bundles (DABs) — recommended Python SDK CLI Core Concepts Multi Task Workflows Jobs support DAG based task dependencies: run if conditions: ALL SUCCESS (default) — run when all dependencies succeed ALL DONE — run when all dependencies complete (success or failure) AT LEAST ONE SUCCESS — run when at least one dependency succeeds NONE FAILED — run when no dependencies failed ALL FAILED — run when all dependencies failed AT LEAST ONE FAILED — run when at least one dependency failed Task Types Summary Task Type Use Case Reference notebook task Run notebooks [references/task types.md notebook task](references/task types.md notebook task) spark python task Run Python scripts [references/task types.md spark python task](references/task types.md spark python task) python wheel task Run Python wheels [references/task types.md python wheel task](references/task types.md python wheel task) sql task Run SQL queries/files/dashboards/alerts [references/task types.md sql task](references/task types.md sql task) dbt task Run dbt projects [references/task types.md dbt task](references/task types.md dbt task) pipeline task Trigger SDP (formerly DLT) pipelines [references/task types.md pipeline task](references/task types.md pipeline task) spark jar task Run Spark JARs [references/task types.md spark jar task](references/task types.md spark jar task) run job task Trigger other jobs [references/task types.md run job task](references/task types.md run job task) for each task Loop over inputs [references/task types.md for each task](references/task types.md for each task) Trigger Types Summary Trigger Type Use Case Reference schedule Cron based scheduling [references/triggers schedules.md cron schedule](references/triggers schedules.md cron schedule) trigger.periodic Interval based [references/triggers schedules.md periodic trigger](references/triggers schedules.md periodic trigger) trigger.file arrival File arrival events [references/triggers schedules.md file arrival trigger](references/triggers schedules.md file arrival trigger) trigger.table update Unity Catalog table change events [references/triggers schedules.md table update trigger](references/triggers schedules.md table update trigger) continuous Always running jobs [references/triggers schedules.md continuous jobs](references/triggers schedules.md continuous jobs) Compute Configuration Serverless is the default. Leave out cluster settings ( job clusters , new cluster , job cluster key , existing cluster id ); a notebook, Python script, wheel or JAR task without them runs on serverless. SQL tasks, and the SQL of dbt tasks, run on a SQL warehouse ( warehouse id , see [Task Types](references/task types.md)); a pipeline task runs on the pipeline's own compute ( serverless: true in the pipeline), and the job's performance target sets its mode. If creating or running the job fails because serverless is not available in the workspace, put the whole job on classic and say so (job clusters for the tasks that run on job compute). For everything else about compute, use the databricks serverless skill : the performance mode ( performance target ), libraries ( environments ), existing code that may not run on serverless, and the few workloads that need a classic cluster. If that skill is not installed, run databricks aitools install . Either way, set performance target on every job: STANDARD to optimize cost, PERFORMANCE OPTIMIZED when the job is time sensitive, interactive, or other workloads depend on it. Job Parameters Parameters defined at job level are passed to ALL tasks (no need to repeat per task): Access in notebooks: Pass to specific tasks: Common Operations Python SDK CLI Declarative Automation Bundle Operations Permissions (DABs) Permission levels: CAN VIEW — view job and run history CAN MANAGE RUN — view, trigger, and cancel runs CAN MANAGE — full control including edit and delete Unit Testing Run unit tests locally: Development Workflow 1. Validate : databricks bundle validate profile <profile 2. Deploy : databricks bundle deploy t dev profile <profile 3. Run : databricks bundle run <job name t dev profile <profile 4. Check run status : databricks jobs get run run id <id profile <profile Common Issues Issue Solution Serverless job startup slow performance target: PERFORMANCE OPTIMIZED ; see databricks serverless Task dependencies not working Verify task key references match exactly in depends on Schedule not triggering Check pause status: UNPAUSED and valid timezone File arrival not detecting Ensure path has proper permissions and uses cloud storage URL Table update trigger missing events Verify Unity Catalog table and proper grants Parameter not accessible Use dbutils.widgets.get() in notebooks admins group error Cannot modify admins permissions on jobs Serverless task fails See the Common Issues of databricks serverless (libraries, RDD code, DBFS mounts) Related Skills databricks dabs — DABs configuration patterns shared by jobs and pipelines databricks pipelines — SDP (formerly DLT) pipelines triggered by pipeline task databricks serverless — compute decisions: serverless, performance mode, classic fallback Documentation [Lakeflow Jobs](https://docs.databricks.com/jobs) [Task types](https://docs.databricks.com/jobs/configure task) [Declarative Automation Bundles](https://docs.databricks.com/dev tools/bundles/) [Jobs API Reference](https://docs.databricks.com/api/jobs/v2/job) [Bundle Examples Repository](https://github.com/databricks/bundle examples)