Use when scraping JavaScript-heavy pages or SPAs, crawling multiple URLs concurrently, extracting structured data with reusable CSS/JSON schemas, or building automated web data pipelines. Wraps the Crawl4AI library (`crwl` CLI and Python SDK) with schema-generation patterns for LLM-free extraction.

Crawl4AI Verified against crawl4ai [ VERSION ](VERSION). PEP 723 pins in scripts/ .py and tests/ .py floor at that version. Overview Crawl4AI wraps a headless browser (Playwright) plus a markdown aware content pipeline. Use it when defuddle/curl can't reach the content — JavaScript rendered pages, login gated content, infinite scroll, multi URL concurrency, repeatable schema based extraction. This skill exposes both interfaces of the underlying library: CLI ( crwl ) — quick, scriptable commands: [CLI Guide](references/cli guide.md) Python SDK — full programmatic control: [SDK Guide](references/sdk guide.md) Invoked with a URL argument When the user runs /crawl4ai <url with a single URL and no further qualifier, treat it as the JS heavy fetch case and default to: wait until=networkidle waits for the network to be quiet for ~500ms post load — the right default when the user hasn't named a specific element on a JS rendered page. (Avoid wait for=css:body : <body exists at t=0 on every HTML response, so it's satisfied before JS renders content.) Then return the markdown to the agent context. Adjust to wait for=css:<selector if the user named a specific element. Skip the default and route to the relevant section below for any task that names extraction, batch / multi URL, login / session, screenshot / PDF, or URL discovery — those each have their own pipeline. If the URL is clearly static (a docs page, a blog post), route the user to /fetch web instead per the "When NOT to use" section below. When NOT to use this skill Static HTML pages (most documentation sites, blog posts, news articles, tweets) — use /fetch web or defuddle directly. Static extraction is ~0ms cold start; crawl4ai pays a ~2s browser startup tax. Local file conversion ( .pdf , .docx , .pptx , .epub ) — use /markdown convert . One URL agent context reads (the agent just needs to read this page) — use /fetch web and let it route to defuddle . Mutating UI flows (form fills, multi step clicks, login + navigation) — /browse (gstack's persistent headless Chromium) is built for that. When stuck For unknown crwl/SDK flags, scrape failures, or extraction edge cases the references don't cover, see [references/escalation.md](references/escalation.md) for the lookup order (qmd solutions → upstream docs → GitHub issues → ask the user) and worked examples. Quick Start Installation CLI (Recommended) Python SDK For SDK configuration details: [SDK Guide Configuration](references/sdk guide.md configuration). Core Concepts Configuration Layers Both CLI and SDK use the same underlying configuration: Concept CLI SDK Browser settings B browser.yml or b "param=value" BrowserConfig(...) Crawl settings C crawler.yml or c "param=value" CrawlerRunConfig(...) Extraction e extract.yml s schema.json extraction strategy=... Content filter f filter.yml markdown generator=... Key Parameters Browser Configuration: headless : Run with/without GUI viewport width/height : Browser dimensions user agent : Custom user agent proxy config : Proxy settings Crawler Configuration: page timeout : Max page load time (ms) wait for : CSS selector or JS condition to wait for cache mode : bypass, enabled, disabled js code : JavaScript to execute css selector : Focus on specific element For complete parameters: [CLI Config](references/cli guide.md configuration) [SDK Config](references/sdk guide.md configuration) Output Content Every crawl returns: markdown Clean, formatted markdown html Raw HTML links Internal and external links discovered media Images, videos, audio found extracted content Structured data (if extraction configured) Markdown Generation (Primary Use Case) Crawl4AI excels at generating clean, well formatted markdown. CLI Filter templates: [ templates/filter bm25.yml ](templates/filter bm25.yml) (relevance scored against a query), [ templates/filter pruning.yml ](templates/filter pruning.yml) (no query, prunes low quality blocks). Python SDK For filter selection and config field reference, see [Content Filters](references/content filters.md). Data Extraction 1. Schema Based CSS Extraction (Most Efficient) No LLM required at extract time — fast, deterministic, cost free. One time LLM cost to derive the schema, then reuse indefinitely. The bundled scripts split the pipeline by responsibility: Or via the CLI with the YAML strategy template + the saved schema: Schema skeleton: [ templates/css schema.json ](templates/css schema.json). Strategy YAML: [ templates/extract css.yml ](templates/extract css.yml). 2. LLM Based Extraction For one off / irregular content where a CSS schema is too brittle: Or via the CLI with the strategy template: Strategy YAML: [ templates/extract llm.yml ](templates/extract llm.yml). Pays an LLM call per URL — for repeat extraction, prefer the schema pipeline above. For extraction strategy reference: [Extraction Strategies](references/complete sdk reference.md extraction strategies). Advanced Patterns Dynamic Content (JavaScript Heavy Sites) Crawler config template: [ templates/crawler.yml ](templates/crawler.yml). Multi URL Processing The two scripts split on responsibility: batch crawl.py returns markdown per URL; batch extract.py returns schema extracted JSON per URL. Python equivalent uses arun many() : For batch processing reference: [arun many() Reference](references/complete sdk reference.md arunmany reference). URL Discovery Before Crawl When the URL list comes from a sitemap / domain rather than a known list, do discovery first, then feed the result into batch crawl.py / batch extract.py . See [URL Discovery](references/url discovery.md) for the full surface; quick shape: AsyncUrlSeeder is best when you want BM25 scored filtering against a query; DomainMapper is best when you want maximum coverage of one domain. Session & Authentication Fill the login template, then reuse the session id on subsequent crawls: Login template: [ templates/login crawler.yml ](templates/login crawler.yml) (fill in the field id selectors and the post login wait condition before use). For session management reference: [Advanced Features](references/complete sdk reference.md advanced features). Anti Detection & Proxies Browser config template: [ templates/browser.yml ](templates/browser.yml) (uncomment proxy config and init scripts as needed). For pre page load script injection (fingerprint patches that must fire before any site script), populate init scripts: rather than js code: (which fires after the page loads). proxy config works with both the browser strategy and the non browser HTTPCrawlerStrategy — the latter is the cheap path for static fetches behind a corporate proxy. Full surface (CDP attachment, undetected mode, init script patterns): [Anti Detection](references/anti detection.md). Rendering Cached HTML ( raw: / file:// ) If the agent already has HTML in hand (e.g., from defuddle or a previous crawl) and only needs a screenshot, PDF, or MHTML render, skip the network fetch and pass the HTML directly. base url controls relative link resolution: Common Use Cases Eight worked end to end flows (docs page, JS heavy SPA, e commerce product extraction, news aggregation, topic bound domain crawl, login required content, render existing HTML, Q&A) live in [Recipes](references/recipes.md). Pick the recipe closest to the task at hand and adapt. Resources Provided Scripts Script Responsibility scripts/basic crawler.py <url One URL → markdown + screenshot scripts/batch crawl.py <urls.txt Many URLs → markdown files scripts/batch extract.py <urls.txt <schema.json Many URLs + schema → JSON scripts/generate schema.py <url "<instruction " Derive a reusable CSS schema (one time LLM call) scripts/extract with schema.py <url <schema.json Apply a saved schema (no LLM) scripts/extract with llm.py <url "<instruction " Per request LLM extraction (expensive; one off only) Templates YAML and JSON skeletons users copy and fill. All sit at the skill root under templates/ : Template Used for templates/browser.yml BrowserConfig (headless, proxy, user agent, init scripts) templates/crawler.yml CrawlerRunConfig (cache, wait, timeout, JS) templates/extract css.yml JsonCssExtractionStrategy declaration templates/extract llm.yml LLMExtractionStrategy declaration templates/filter bm25.yml BM25 content filter (relevance scored) templates/filter pruning.yml Pruning content filter (quality based, no query) templates/login crawler.yml Session establishing login flow templates/css schema.json CSS schema skeleton Reference Documentation Document Purpose [CLI Guide](references/cli guide.md) Command line interface reference [SDK Guide](references/sdk guide.md) Python SDK quick reference [Recipes](references/recipes.md) Eight worked end to end flows [URL Discovery](references/url discovery.md) AsyncUrlSeeder , SeedingConfig , DomainMapper [Content Filters](references/content filters.md) BM25 vs Pruning vs LLMContentFilter — when to use which [Anti Detection](references/anti detection.md) init scripts , proxy config , undetected mode, CDP attachment [Troubleshooting](references/troubleshooting.md) Symptoms, causes, fixes; what to try before escalating [Complete SDK Reference](references/complete sdk reference.md) Full API documentation (5900+ lines) [Escalation](references/escalation.md) Lookup order, iron rule, halt vs continue, worked examples Best Practices 1. Start with CLI for quick tasks, SDK for automation 2. Use schema based extraction 10 100x more efficient than LLM 3. Enable caching during development bypass cache only when needed 4. Set appropriate timeouts 30s normal, 60s+ for JS heavy sites 5. Use content filters for cleaner, focused markdown 6. Respect rate limits Add delays between requests Troubleshooting For symptom → cause → fix tables (JS not loading, bot detection, empty extracted content, session not persisting, slow crawl, schema generation nonsense, post upgrade regressions), see [Troubleshooting](references/troubleshooting.md). For unknown surface the references don't cover, follow [Escalation](references/escalation.md). For comprehensive API documentation, see [Complete SDK Reference](references/complete sdk reference.md). License Dual licensed under [MIT](LICENSE MIT) OR [Apache 2.0](LICENSE APACHE) at your option (SP