Web scraping plus social data: YouTube, TikTok, Instagram, LinkedIn, Reddit, Threads, plus robust web-page fallback extraction. Use when extracting public posts, transcripts, or pages JS-heavy enough to block plain fetch (e.g. download YouTube transcript, scrape TikTok comments, listing pages behind

Preferred entry: call exports.py (don't hand roll requests) Ready made helpers live in skills/web crawler/exports.py . Prefer them over writing your own proxied get / proxied post calls — they already inject the proxy credentials, so there is no API key to find (don't read $SCRAPECREATORS API KEY / $FIRECRAWL API KEY , don't check .env , don't ask the user). Transcript route — ONE entry for any media link Design rule: captions / subtitles / published transcripts are METADATA; the media file is PAYLOAD. Exhaust every metadata provider first; touch the media only when the user explicitly agrees. Same rule claude video, spoken.md and yt dlp's skip download workflow follow. Nothing here is site specific: provider (cost order) what it is identity / validity check media info(url) yt dlp extract info(download=False) — 1800+ sites, returns title, duration, upload date, channel, caption track URLs. Zero media bytes. track fetched must be 2xx with a real body RSS <podcast:transcript Podcasting 2.0 standard, any podcast feed item matched by duration ±5 % + date ±3 d; transcript URL must be 2xx, 500 chars, not an error page show's own YouTube upload channel = the YouTube URL the show declares in its RSS (else search hit whose channel name equals the show name minus stopwords) candidates = duration ±5 % AND date ±3 d; identity = title similarity ≥ 0.6 OR show notes overlap (≥ 6 shared distinctive tokens, Jaccard ≥ 0.15). A candidate that fails identity is returned in candidate with found=False — never delivered as the episode scrape markdown(url) any other page (publisher, Snipd, blog) — browser rendered, survives WAF 403 1000 chars found=False ⇒ report kind , title , tried , note (and candidate if any) and ask the user before any download. YouTube is recognised by URL: the captions API runs even when yt dlp metadata fails, and a page scrape is never reported as a transcript for a media URL. yt dlp skip download / J / list subs are metadata calls and pass the bash gate; yt dlp <url , curl … .mp3 , ffmpeg, whisper are held for confirmation. Title only, no link → web search("<show <title transcript") then get transcript(<page url ) . Incident 2026 09 17: two 403s on transcript hosts led an agent to a 49 MB mp3 + ffmpeg + 4 whisper calls + an incomplete summary. Same link through get transcript : RSS had no transcript → RSS declares youtube.com/@a16z → channel listing → one upload within 1.2 % duration, same day → captions, 55k chars, ~30 s, zero audio. Site audit — "check all links" / "this button doesn't work" You are the auditor. exports.py gives you small, site agnostic tools that observe and return raw data ; they never decide what is broken. There is no keyword list, no per site config and no verdict function — every site is different, so you look, form a hypothesis, test it, and judge from evidence. Tool Returns fetch(url, method="GET") raw status , final url , headers, body, error check urls(urls, method="HEAD") fetch for many URLs in parallel, no bodies extract refs(html, base url) every URL like reference + where it came from ( a[href] , link[rel=…] , inline script , attr[data …] , meta[refresh] …) parse sitemap(xml) <loc entries of a sitemap / sitemap index browse(url, actions=[…], viewport=…, mobile=…) real Chromium: rendered links (incl. shadow DOM), visible controls, JS exceptions, console errors, ≥400 responses, failed requests, and for each action you asked for: URL before/after, navigations with status, new windows with their own status, downloads, DOM changed?, new links Method (details and worked checks: reference/site audit method.md ; full worked example: reference/site audit case study.md ): 1. Map the site from several sources, then reconcile. Home page HTML via extract refs , robots.txt → sitemap(s) via parse sitemap , rendered DOM via browse , and any URL the user named. Read the inline script and attr[...] hits yourself — route tables and JS navigation live there. A page the user mentions that none of your sources found is a finding about your map, not about the user. 2. Learn what "normal" and "error" look like on THIS site. Fetch one real page and one surely missing URL (random path). Compare status, title, size, text. That baseline — not a word list — tells you how this site shows a 404 (real 404, 200 soft 404, redirect to home…). 3. Check every page and every reference. check urls , re try HEAD failures with GET, re check transient errors once. Keep raw results; group them yourself (same site pages / assets / third party). 4. For what users actually click, click it. Use browse(...).controls and rendered links to choose targets, then browse(url, actions=[{"click": sel}]) at desktop and mobile width. Judge from the action record: did the URL change, did a new window open and what was its status, did a download finish, did the DOM change, did a JS error fire, was the element covered ( element.topmost at center )? 5. Explain anything unexpected before reporting it. 401/403/429 or bot walls = "could not verify", not broken and not fine. A click with no visible effect may be correct (already on that page) — say what you observed. 6. Report with evidence. Pages found and how; what is broken (URL, where it is referenced, status/observation, viewport); what could not be verified and why; what you did not cover. Never write "all links OK" for areas you did not check. Safety: only click targets you chose for a reason. Don't click anything form associated or whose text/context implies submit, buy, delete, send or subscribe unless the user explicitly asked. browse aborts non GET requests for the whole browser session by default — page load, actions and any new windows ( blocked writes , each with its phase ); keep it that way unless the user authorizes writes. If Playwright is unavailable browse returns {"error": …} — report clicks as not verified; never infer them from status codes. Quick trigger rules (read this first) Use this skill immediately when any of these conditions is true: web fetch returns HTTP 401/403/429/5xx Response is an anti bot challenge page (for example Cloudflare "Attention Required", "Just a moment", or challenge/captcha pages) The page is JS heavy and the first fetch misses required detail fields (for example publish time, author, listing code, updated time, price breakdown) Search results do not contain the requested field and the value must be extracted from the target page itself Fallback rule: If ordinary fetch is blocked or incomplete, switch to Firecrawl fallback in this skill before asking the user for screenshots/manual text. Error signatures action Signature Action web fetch HTTP 401/403/429/5xx Call Firecrawl POST /v2/scrape once with formats:["markdown","links"] + onlyMainContent:true Cloudflare/challenge page text in body Same Firecrawl call as above Markdown still misses key fields Retry once with formats:["rawHtml"] Firecrawl returns 403 / empty for a social media URL Check the intent routing tables below for a ScrapeCreators platform specific endpoint for this domain (e.g. sc get('/v1/instagram/post', url=...) , sc get('/v2/tiktok/video', url=...) ). If one exists, use it — these have dedicated extraction that bypasses anti bot. If no dedicated endpoint exists, fall through to archive fallback or ask the user. Firecrawl itself returns 403 / empty (hard paywall: NYT, WSJ, Economist, FT, Bloomberg) Call archive fallback(url) — recovers full text from a web archive snapshot Need structured data from a China app (抖音/小红书/微博/B站/京东/淘宝/1688/闲鱼/得物 etc.) Call apify run() — Apify Store has purpose built actors for these platforms that Firecrawl/ScrapeCreators don't cover Instagram story URL ( instagram.com/stories/<user /<id ) ScrapeCreators has no story endpoint and Firecrawl hits the login wall — go straight to apify run("data slayer~instagram stories scraper", ...) . See "Instagram stories" below Paywall / Firecrawl blocked fallback chain (use archive fallback ) When Firecrawl can't get the page either (it returns 403, or markdown comes back empty) the site is behind a hard paywall or aggressive WAF. Do NOT keep retrying Firecrawl. Recover the article from a web archive snapshot instead: How it works (and why this order): archive.today first ( archive.ph / archive.is mirrors). User triggered, real browser captures; historically preserves full text behind paywalls. Best bet for NYT/WSJ/Economist. We scrape its /newest/ snapshot via Firecrawl (archive.today has its own Cloudflare, so scrape it through Firecrawl, never web fetch it directly). Wayback Machine second ( archive.org ). Automated crawler that honors robots.txt and paywalls, so it often has NO full text for hard paywalls — but it's a good fallback for ordinary 403/Cloudflare pages that aren't paywalled. Limitation: archives only return text someone already saved . If res["markdown"] == "" , no snapshot exists — stop, tell the user, and try the outlet's official API/RSS or a different source. Do not fabricate the article. China app structured data (use apify run ) When you need structured data from a China app — Douyin video search, Xiaohongshu notes, Weibo posts, Bilibili videos, JD/Taobao product prices, 1688 wholesale listings, Xianyu second hand, Dewu sneakers, etc. — Firecrawl and ScrapeCreators don't cover these platforms. Use the Apify Store fallback instead. Apify is a serverless scraper marketplace with hundreds of community maintained actors that run real browsers + proxy pools against Chinese platforms. Auth: No user supplied key needed. sc proxy injects the platform Apify token automatically. The Authorization: Bearer header can be any fake value — the proxy replaces it with the real token. Do NOT read $APIFY TOKEN from env, do NOT check .env , do NOT ask the user for an Apify key. When to use Apify (vs Firecrawl/ScrapeCreators): ✅ China apps: 抖音, 小红书, 微博, B站, 京东, 淘宝, 1688, 闲鱼, 携程, 知乎, 豆瓣, 雪球, 优酷 (得物/快手/爱奇艺: no priced actor — search the Store) ✅ Southeast Asia e commerce: Shopee, Lazada, Temu ✅ Instagram stories (ScrapeCreators has no story endpoint) — see "Instagram stories" below ❌ Other Western social media (TikTok/Instagram posts & reels/YouTube/X/Reddit) → use ScrapeCreators first (cheaper) ❌ Generic web page scraping → use Firecrawl first (cheaper) ❌ Hard paywall articles → use archive fallback (Apify doesn't help here) How to pick an actor: Use the table below first — these actors are priced in sc proxy and were verified live on 2026 10 08 (30 day successful runs in brackets). For a platform not listed, search the Store through the proxy ( proxied get("https://api.apify.com/v2/store?search=<platform &limit=10") ), prefer high 30 day run counts, and expect the default $0.007/result price. Platform Actor ID Input key 抖音 search zen studio~douyin search scraper {"keywords": [...], "maxResultsPerQuery": N} 抖音 creator profile zen studio~douyin profile scraper (22k) (see actor input schema) 抖音 video/audio download apple yang~douyin video audio downloader (5k) (see actor input schema) 抖音 transcripts apple yang~douyin transcripts scraper (2.6k) (see actor input schema) 小红书 search zen studio~rednote search scraper {"keywords": [...], "maxResults": N} 小红书 note detail sian.agency~xiaohongshu rednote scraper {"operation": "noteDetail", "noteId": "...", "xsecToken": "..."} 小红书 data API socialdatax~socialdatax xhs data api (64k) (see actor input schema) 微博 hot search gentle cloud~weibo hot search scraper {"mode":

web-crawler