Migrate Databricks workloads from classic compute to serverless compute. Use when migrating notebooks, jobs, pipelines, or Scala JARs (`spark_jar_task`) from classic clusters to serverless, checking if existing code is serverless-compatible, or writing new serverless-compatible code. Provides concre

Serverless Compute Migration FIRST : Use the parent databricks core skill for CLI basics, authentication, and profile selection. Analyze existing Databricks code for serverless compute compatibility and guide migration from classic clusters. The skill follows a 4 step migration lifecycle: Ingest the workload → Analyze for compatibility → Test via A/B comparison → Validate and iterate. When to Use This Skill Migrating notebooks, jobs, or pipelines from classic compute to serverless Checking if existing code is serverless compatible Writing new code that targets serverless compute Troubleshooting serverless specific errors after migration Choosing between Performance Optimized and Standard mode Where to Run This Skill This skill is published as an Agent Skill (agentskills.io) and runs in any compatible client: Claude Code, Cursor, or any agentskills.io client on your laptop — the default. Install via databricks aitools install or follow the per client docs. Inside a Databricks workspace via Genie Code Agent mode — drop the skill into /Workspace/Users/<you /.assistant/skills/databricks serverless migration/ (per user) or /Workspace/.assistant/skills/databricks serverless migration/ (workspace wide, admin only). See [Install in Databricks Genie Code](references/install in databricks genie code.md) for the three install methods and the important serverless compute caveat (the Databricks CLI isn't pre installed on serverless, so some deploy steps need adjustment). If you finish a migration analysis for a user who's currently running you from a laptop client, mention the Genie Code option once at the end — many users prefer iterating on migrations inside the workspace where the workload lives. Understanding Migration Blockers Migration blockers fall into three categories. Focus your effort on category 2 — that's where this skill helps most. Category Description Action 1. Feature expanding Databricks is actively expanding support (e.g., SparkML, custom JDBC) Use the workaround now and revisit later 2. Code/config change needed Your code uses patterns that need updating for serverless (e.g., RDDs, DBFS, streaming triggers) This skill helps here — it detects these patterns and provides fixes 3. Classic only Workload requires capabilities not available on serverless (e.g., root OS access, R language) Keep on classic compute Decision Tree: Is My Workload Ready? Migration Workflow Step 1: Ingest — Gather Workload Context Confirm the migration target is serverless compute. This skill is purpose built for classic → serverless migrations. The checks, fixes, and workflow all target the serverless compute architecture (Spark Connect, Environments, NCCs). If the user wants to upgrade between classic DBR versions instead, this skill does not apply — classic DBR upgrades have a different compatibility surface and should follow the standard DBR upgrade guide. Collect the full picture of what needs to migrate to serverless: Read the user's notebook/script files Identify the classic cluster configuration (instance type, DBR version, Spark configs, init scripts, libraries) Note the networking setup (VPC peering, instance profiles, mounts) Understand the workload type: batch job, streaming, interactive notebook, pipeline Determine the target: the output is always a serverless compute configuration, not a classic cluster with a newer DBR Step 1.5: Handle Multi Notebook Workloads If the workload spans more than one user notebook (exclude resources/ setup notebooks from the count), process them one at a time rather than all at once. The agent's context window is finite, and trying to hold the full source of an 8 notebook job in active context while doing analysis, fixes, and migration triggers autocompact thrashing (Claude Code) or equivalent context overflow failures in other clients. Procedure : 1. Enumerate first. List every user notebook with its path. Do not read the bodies in full yet beyond a few lines for orientation. Check for bundle config.py (H4). If the source tree contains a bundle config.py , parse it and follow any upstream source declarations (git URLs, S3 paths, init scripts , git source , or custom upstream repo declarations). Clone or fetch each referenced repo into a sibling location and include its notebooks in the enumeration. List "external sources" as a first class artifact category in the migration plan. Without this step, multi source demos (e.g., dbt on databricks references the upstream dbt databricks c360 repo) get mis enumerated and the real task notebooks are missed. 2. For each notebook, in order : a. Read its full source. b. Run the Step 2 Analyze checklist scoped to this notebook only. c. Record a structured summary in your response or to a scratch file, then drop the raw source from active working memory. The summary must include: notebook path (relative) detected patterns[] (pattern IDs from this skill's catalog) blockers[] (Category 3 patterns this notebook hits, if any) migration steps[] (concrete fixes ordered) unmigratable (bool — true if any blocker has no workaround) d. If you have a writable scratch directory ( ~/.databricks migration skill/scratch/<run id /findings/ ), persist the summary as <notebook basename .json . This frees the source from context safely. If not, keep the summary compact in conversation history. 3. Synthesize. After all notebooks have a summary, produce the unified migration plan from the summaries alone. Apply fixes notebook by notebook, never re reading the original source unless required to resolve an ambiguity in a summary. 4. Failure Reporting. Trigger the Failure Reporting Protocol exactly once per workload, not once per notebook. The detected patterns array in the report aggregates across all notebooks; notebook characteristics uses summed line counts and the union of language/streaming/ML flags. Threshold guidance : apply this procedure when the user notebook count is ≥ 3, or when individual notebooks exceed ~5KB of source. For 1–2 small notebooks the single pass workflow in Steps 2–4 is fine. Anti pattern : do NOT attempt a "merge all notebooks into one big file" or "read all notebooks, then act in one mega turn" strategy. Both defeat the purpose by reintroducing the original context pressure problem. Step 2: Analyze — Scan for Serverless Readiness Read notebooks before running them — do not rely on failed job runs to discover issues. A pre run scan surfaces incompatibilities faster than iterating on error traces, and many serverless failures (hardcoded catalog references, init scripts, missing dependencies) are easy to spot statically but expensive to debug after a failed run. Before creating or running any test job: 1. Read every notebook and source file referenced by the job 2. Scan for all hardcoded catalog/schema references (e.g., spark.table("main.schema.table") , spark.sql("... FROM main...") , catalog = "main" ) 3. Check for dependency patterns: init scripts, local wheel files, custom install functions, %pip install lines 4. Locate any requirements.txt or equivalent and resolve the full dependency set 5. Flag OS level installs ( apt install , yum install ) for conversion or escalation Scan the code for patterns that are incompatible with the serverless compute architecture. These checks are serverless specific — most of these patterns work fine on classic compute regardless of DBR version. For each issue found, report: Category : Which of the 3 blocker categories it falls into Severity : Blocker (must fix for serverless) / Warning (should fix) / Info (awareness) Pattern : What was detected and where Fix : Specific remediation targeting serverless compute Post rewrite lint: Cell magic boundary check (A1) HIGH IMPACT. Before declaring a migrated notebook ready, run this lint pass on every output cell. Caused 3/7 demos to fail in the dbdemos E2E sweep (hls readmission, fsi fraud, retail c360). Detect : any cell that contains a MAGIC %<word line (e.g., MAGIC %run , MAGIC %sql , MAGIC %md , MAGIC %pip , MAGIC %fs ). Within that cell, every non blank line must either start with MAGIC or be a blank line. If a plain Python comment (or any other Python code) precedes the MAGIC %... directive in the same cell, the cell is corrupted: Databricks parses it as Python and %run falls back to IPython line magic, producing errors like File './00 global setup v2' not found . Fix : never prepend plain Python comments above a MAGIC %... line within the same cell. Two valid options: 1. Preferred : put migration notes in a separate MAGIC %md cell above (its own COMMAND block). 2. Acceptable : drop the migration note entirely and rely on git/file history. Example before (corrupted; %run fails with File not found ): Example after (clean; %run fires as cell magic): Category A: Unsupported APIs Pattern Severity Fix sc.parallelize(data) Blocker spark.createDataFrame([(x,) for x in data], ["value"]) rdd.map(fn) Blocker df.select(F.col("value") 2) or df.withColumn(...) rdd.filter(fn) Blocker df.filter(F.col("value") 3) rdd.reduce(fn) Blocker df.agg(F.sum("col")).collect()[0][0] rdd.flatMap(fn) Blocker df.select(F.explode(F.split(col, " "))) rdd.groupByKey() Blocker df.groupBy("key").agg(F.collect list("value")) rdd.mapPartitions(fn) Blocker df.groupBy(F.spark partition id()).applyInPandas(fn, schema) sc.textFile(path) Blocker spark.read.text(path) sc.wholeTextFiles(path) Blocker spark.read.format("binaryFile").load(path) sc.broadcast(data) Blocker from pyspark.sql.functions import broadcast; df.join(broadcast(lookup df), key) sc.accumulator(init) Blocker df.agg(F.sum("col")) or df.count() spark.sparkContext Blocker Use spark (SparkSession) directly SparkContext.getOrCreate() Blocker Not supported — raises RuntimeError: Only remote Spark sessions using Databricks Connect are supported . Replace with spark.createDataFrame() or spark.range() for data setup. sqlContext.sql(query) Blocker spark.sql(query) sc.hadoopConfiguration.set(...) Blocker Use UC external locations — no credential configs needed df.cache() / df.persist() Warning Remove caching calls. For expensive intermediate results, materialize to a Delta table. Native support coming soon. df.checkpoint() Warning Write to Delta table instead spark.catalog.cacheTable(t) / CACHE TABLE Warning Remove — not needed on serverless %scala cells in notebook Blocker Port to PySpark/SQL or compile as JAR for job tasks %r cells in notebook Blocker No serverless equivalent — keep on classic or port to PySpark Hive variable syntax ${var} Warning Use DECLARE VARIABLE / SET VARIABLE (SQL) or Python f strings CREATE GLOBAL TEMPORARY VIEW Blocker Use CREATE OR REPLACE TEMPORARY VIEW — global temp database doesn't exist on serverless global temp. prefix in queries Warning Remove prefix — session scoped temp views are accessible without qualifier Builtin max(..., key=) / min(..., key=) / sorted(..., key=) with from pyspark.sql.functions import (A2) Blocker pyspark.sql.functions.max shadows the builtin and rejects key= (raises TypeError: max() got an unexpected keyword argument 'key' ). Use sort+index: xs.sort(key=...); top = xs[0] . See [MLflow on UC](references/mlflow uc patterns.md). from databricks import automl / automl.classify() / automl.regress() / automl.forecast() (A3) Blocker AutoML not available on serverless and the DBDemos.create mockup automl run fallback

databricks-serverless-migration