Name: databricks-serverless-migration
Author: databricks

Migrate Databricks workloads from classic compute to serverless compute. Use when migrating notebooks, jobs, pipelines, or Scala JARs (`spark_jar_task`) from classic clusters to serverless, checking if existing code is serverless-compatible, or writing new serverless-compatible code. Provides concre

Serverless Compute Migration FIRST : Use the parent databricks core skill for CLI basics, authentication, and profile selection. Analyze existing Databricks code for serverless compute compatibility and guide migration from classic clusters. The skill follows a 4 step migration lifecycle: Ingest the workload → Analyze for compatibility → Test via A/B comparison → Validate and iterate. When to Use This Skill Migrating notebooks, jobs, or pipelines from classic compute to serverless Checking if existing code is serverless compatible Writing new code that targets serverless compute Troubleshooting serverless specific errors after migration Choosing between Performance Optimized and Standard mode Where to Run This Skill Runs in any agentskills.io client: on a laptop (Claude Code, Cursor, ...; the default, install via databricks aitools install ), or inside a Databricks workspace via Genie Code Agent mode (per user or workspace wide .assistant/skills/ folder). See [Install in Databricks Genie Code](references/install in databricks genie code.md) for the three install methods and the serverless compute caveat (the Databricks CLI isn't pre installed on serverless, so some deploy steps need adjustment). If you finish a migration analysis for a user who's currently running you from a laptop client, mention the Genie Code option once at the end — many users prefer iterating on migrations inside the workspace where the workload lives. Understanding Migration Blockers Migration blockers fall into three categories. Focus your effort on category 2 — that's where this skill helps most. Category Description Action 1. Feature expanding Databricks is actively expanding support (e.g., SparkML, custom JDBC) Use the workaround now and revisit later 2. Code/config change needed Your code uses patterns that need updating for serverless (e.g., RDDs, DBFS, streaming triggers) This skill helps here — it detects these patterns and provides fixes 3. Classic only Workload requires capabilities not available on serverless (e.g., root OS access, R language) Keep on classic compute Decision Tree: Is My Workload Ready? Migration Workflow Step 1: Ingest — Gather Workload Context Confirm the migration target is serverless compute. This skill is purpose built for classic → serverless migrations. The checks, fixes, and workflow all target the serverless compute architecture (Spark Connect, Environments, NCCs). If the user wants to upgrade between classic DBR versions instead, this skill does not apply — classic DBR upgrades have a different compatibility surface and should follow the standard DBR upgrade guide. Collect the full picture of what needs to migrate to serverless: Read the user's notebook/script files Identify the classic cluster configuration (instance type, DBR version, Spark configs, init scripts, libraries) Note the networking setup (VPC peering, instance profiles, mounts) Understand the workload type: batch job, streaming, interactive notebook, pipeline Determine the target: the output is always a serverless compute configuration, not a classic cluster with a newer DBR Step 1.5: Handle Multi Notebook Workloads If the workload spans 3+ user notebooks (exclude resources/ setup notebooks) or any notebook exceeds ~5KB of source, process notebooks one at a time to avoid context overflow (autocompact thrashing): enumerate first (including upstream sources declared in bundle config.py , H4), then per notebook read → run the Step 2 checklist → record a structured summary and drop the source, then synthesize the plan from the summaries. Trigger the Failure Reporting Protocol once per workload, not per notebook. Don't merge all notebooks into one file or act on all of them in one mega turn. Full procedure (summary fields, scratch dir layout, report aggregation): [Multi Notebook Workloads](references/multi notebook workloads.md). Step 2: Analyze — Scan for Serverless Readiness Read notebooks before running them — do not rely on failed job runs to discover issues. A pre run scan surfaces incompatibilities faster than iterating on error traces, and many serverless failures (hardcoded catalog references, init scripts, missing dependencies) are easy to spot statically but expensive to debug after a failed run. Before creating or running any test job: 1. Read every notebook and source file referenced by the job 2. Scan for all hardcoded catalog/schema references (e.g., spark.table("main.schema.table") , spark.sql("... FROM main...") , catalog = "main" ) 3. Check for dependency patterns: init scripts, local wheel files, custom install functions, %pip install lines 4. Locate any requirements.txt or equivalent and resolve the full dependency set 5. Flag OS level installs ( apt install , yum install ) for conversion or escalation Scan the code for patterns that are incompatible with the serverless compute architecture. These checks are serverless specific — most of these patterns work fine on classic compute regardless of DBR version. For each issue found, report: Category : Which of the 3 blocker categories it falls into Severity : Blocker (must fix for serverless) / Warning (should fix) / Info (awareness) Pattern : What was detected and where Fix : Specific remediation targeting serverless compute Cell magic boundary lint (A1), high impact: in any cell with a MAGIC %<word line, every non blank line must start with MAGIC . A plain Python comment above MAGIC %run in the same cell makes %run fall back to IPython ( File './…' not found ); put migration notes in their own MAGIC %md cell instead. Pattern catalog : load [Analyze Checklist](references/analyze checklist.md) for every Step 2 scan. It lists each check with severity and fix: A APIs (RDD, caching, MLflow/AutoML A2–A5, M1–M4), B data access (B1–B2), C streaming, D config (D1), E libraries (E1–E2), F networking, G sizing, H job level config (H1–H4). Required Output: Serverless Environment Specification The migration output MUST include a Serverless Environment specification alongside migrated code. Generate this by: 1. Scanning all import statements and %pip install lines to detect required packages 2. Extracting init script pip install commands from the job configuration 3. Producing a JSON block suitable for the Jobs API environments field: Important : ML runtime is NOT available on serverless and ML libraries (mlflow, scikit learn, hyperopt, xgboost, tensorflow, torch, etc.) are NOT pre installed, so list them explicitly in the environment spec dependencies . Step 3: Test — Two Branch Strategy Use separate branches for testing and production to keep test only workarounds out of the code that ships. The test branch is a safe sandbox for experimentation; the production branch contains only changes that production actually needs. Branch by branch comparison (names, base branch, PR, catalog, what each may contain): [Testing Strategy](references/testing strategy.md two branch comparison). Test branch ( serverless test {job name} {timestamp} ): Temporary, no PR needed. 1. Create a branch from your current working branch 2. Set up test data: create sampled copies of upstream tables in a test catalog using job lineage (see [Testing Strategy § Test Data Setup](references/testing strategy.md test data setup)) 3. Parameterize the catalog so the notebook works with both test and production data (see [Quick Fixes § Parameterize catalogs](references/quick fixes.md parameterize catalogs for testing)) 4. Apply all compatibility fixes discovered in Step 2 5. Create a serverless test job and run it 6. If it fails, get the error output, debug, fix, and retry 7. Document which changes are test workarounds vs. real compatibility fixes Production branch ( serverless prod {job name} ): PR required, created from master. 1. Create a new branch from master (NOT from the test branch) 2. Apply ONLY the real compatibility fixes — no test workarounds 3. Apply job config changes (see [Quick Fixes and Debugging]( quick fixes and debugging)) 4. Commit and create a PR Decision Tree: Should This Change Go to Production? Change type Production? Reason Remove incompatible Spark configs Yes Serverless compatibility fix Update library versions Yes Serverless compatibility fix Replace DBFS paths with UC Volumes Yes Serverless compatibility fix Remove init scripts, add Environments Yes Serverless compatibility fix Fix hardcoded cluster settings Yes Serverless compatibility fix Catalog override to test catalog No Test workaround only Empty DataFrame handling for missing test data No Test workaround only Date range limiting for faster tests No Test workaround only Simple test : Would production fail without this change on serverless? If yes → include. If no → test branch only. A/B Comparison After both branches are ready, compare classic vs serverless outputs (row count, schema, exceptAll row diff; snippet and the temporary bridge config rule in [Testing Strategy § A/B Comparison](references/testing strategy.md ab comparison)). Test against sampled copies of upstream tables in a test catalog, never production data ([Test Data Setup](references/testing strategy.md test data setup)). Step 4: Validate — Confirm and Monitor Once the A/B comparison passes: 1. Merge the production branch PR 2. Switch the production job to serverless compute 3. Monitor cost via system tables ( system.billing.usage ) and budget policies 4. Remove any temporary bridge configurations 5. Set up budget alerts for cost visibility Migration Deliverables Surface all six at the end of a successful run: test branch name/URL, production branch name/URL, test job ID and run URL, classic vs serverless comparison, serverless environment spec, and change summary (production vs test only, with reasons). What each is and why it matters: [Testing Strategy § Migration Deliverables](references/testing strategy.md migration deliverables). If any deliverable is missing, the migration is incomplete — do not mark it as done. Stopping Conditions Do not attempt workarounds for these — surface them to the user and stop: Permission failures on source tables, the test catalog, or the workspace Category 3 blockers (R code, custom Spark data source JARs, features that require classic compute) SQL warehouse or test catalog not available Repeated failures (typically 5+) with no new information in the error trace — generate a failure report instead (see Failure Reporting Protocol) Quick Fixes and Debugging [Quick Fixes](references/quick fixes.md): before/after snippets for DBFS → UC Volumes, RDD → DataFrame, streaming triggers (+ OOM caps for Delta, Kafka, Auto Loader), caching removal, runtime serverless detection ( IS SERVERLESS ), and catalog parameterization for tests. [Debugging Errors](references/debugging errors.md): error message → fix table for failed serverless runs. Always get the actual error with w.jobs.get run output(run id=...) before guessing. Transform job config from classic to serverless Remove job clusters / new cluster , add environments with serverless spec, replace job cluster key with environment key , remove init scripts . Full rules (init scripts → dependencies , Maven/JAR libraries, spark conf triage) and before/after JSON: [Configuration Guide](references/configuration guide.md job definition migration transformation rules). Environment version mapping (match to the DBR version the workload was on): Classic DBR Serverless spec.client Python 13.x, 14.x "1" 3.10 15.x "2" 3.11 16.x+ "3" 3.12 Performance Mode Selection Performance Optimized (default in UI and API; <50 s startup; higher cost; notebooks, jobs, SDP) for interactive, d