Web scraping plus social data: YouTube, TikTok, Instagram, LinkedIn, Reddit, Threads, plus robust web-page fallback extraction. Use when extracting public posts, transcripts, or pages JS-heavy enough to block plain fetch (e.g. download YouTube transcript, scrape TikTok comments, listing pages behind

Preferred entry: call exports.py (don't hand roll requests) Ready made helpers live in skills/web crawler/exports.py . Prefer them over writing your own proxied get / proxied post calls — they already inject the proxy credentials, so there is no API key to find (don't read $SCRAPECREATORS API KEY / $FIRECRAWL API KEY , don't check .env , don't ask the user). Named wrappers exist for the high frequency actions (YouTube/TikTok transcript & video, IG/Twitter/Reddit posts, profiles, Google/Reddit search). For any other ScrapeCreators endpoint use sc get(path, params) — it auto strips leading @ / from handles/hashtags. The intent routing tables below still tell you which endpoint to pass. Pass caller id="chat:<thread " (or job: / preview: ) for cost tracking. Quick trigger rules (read this first) Use this skill immediately when any of these conditions is true: web fetch returns HTTP 401/403/429/5xx Response is an anti bot challenge page (for example Cloudflare "Attention Required", "Just a moment", or challenge/captcha pages) The page is JS heavy and the first fetch misses required detail fields (for example publish time, author, listing code, updated time, price breakdown) Search results do not contain the requested field and the value must be extracted from the target page itself Fallback rule: If ordinary fetch is blocked or incomplete, switch to Firecrawl fallback in this skill before asking the user for screenshots/manual text. Error signatures action Signature Action web fetch HTTP 401/403/429/5xx Call Firecrawl POST /v2/scrape once with formats:["markdown","links"] + onlyMainContent:true Cloudflare/challenge page text in body Same Firecrawl call as above Markdown still misses key fields Retry once with formats:["rawHtml"] Firecrawl returns 403 / empty for a social media URL Check the intent routing tables below for a ScrapeCreators platform specific endpoint for this domain (e.g. sc get('/v1/instagram/post', url=...) , sc get('/v2/tiktok/video', url=...) ). If one exists, use it — these have dedicated extraction that bypasses anti bot. If no dedicated endpoint exists, fall through to archive fallback or ask the user. Firecrawl itself returns 403 / empty (hard paywall: NYT, WSJ, Economist, FT, Bloomberg) Call archive fallback(url) — recovers full text from a web archive snapshot Need structured data from a China app (抖音/小红书/微博/B站/京东/淘宝/1688/闲鱼/得物 etc.) Call apify run() — Apify Store has purpose built actors for these platforms that Firecrawl/ScrapeCreators don't cover Paywall / Firecrawl blocked fallback chain (use archive fallback ) When Firecrawl can't get the page either (it returns 403, or markdown comes back empty) the site is behind a hard paywall or aggressive WAF. Do NOT keep retrying Firecrawl. Recover the article from a web archive snapshot instead: How it works (and why this order): archive.today first ( archive.ph / archive.is mirrors). User triggered, real browser captures; historically preserves full text behind paywalls. Best bet for NYT/WSJ/Economist. We scrape its /newest/ snapshot via Firecrawl (archive.today has its own Cloudflare, so scrape it through Firecrawl, never web fetch it directly). Wayback Machine second ( archive.org ). Automated crawler that honors robots.txt and paywalls, so it often has NO full text for hard paywalls — but it's a good fallback for ordinary 403/Cloudflare pages that aren't paywalled. Limitation: archives only return text someone already saved . If res["markdown"] == "" , no snapshot exists — stop, tell the user, and try the outlet's official API/RSS or a different source. Do not fabricate the article. China app structured data (use apify run ) When you need structured data from a China app — Douyin video search, Xiaohongshu notes, Weibo posts, Bilibili videos, JD/Taobao product prices, 1688 wholesale listings, Xianyu second hand, Dewu sneakers, etc. — Firecrawl and ScrapeCreators don't cover these platforms. Use the Apify Store fallback instead. Apify is a serverless scraper marketplace with hundreds of community maintained actors that run real browsers + proxy pools against Chinese platforms. Auth: No user supplied key needed. sc proxy injects the platform Apify token automatically. The Authorization: Bearer header can be any fake value — the proxy replaces it with the real token. Do NOT read $APIFY TOKEN from env, do NOT check .env , do NOT ask the user for an Apify key. When to use Apify (vs Firecrawl/ScrapeCreators): ✅ China apps: 抖音, 小红书, 微博, B站, 京东, 淘宝, 1688, 闲鱼, 得物, 携程, 知乎, 豆瓣, 雪球, 快手, 爱奇艺, 优酷 ✅ Southeast Asia e commerce: Shopee, Lazada, Temu ❌ Western social media (TikTok/Instagram/YouTube/X/Reddit) → use ScrapeCreators first (cheaper) ❌ Generic web page scraping → use Firecrawl first (cheaper) ❌ Hard paywall articles → use archive fallback (Apify doesn't help here) How to pick an actor: The reliable actors catalog is at output/apify china reliable.json (sorted by 30 day success count). Pick the top actor for the target platform. A few common ones: Platform Actor ID Input key 抖音 search zen studio~douyin search scraper {"keywords": [...], "maxResultsPerQuery": N} 小红书 search zen studio~rednote search scraper {"keywords": [...], "maxResults": N} 小红书 note detail sian.agency~xiaohongshu rednote scraper {"operation": "noteDetail", "noteId": "...", "xsecToken": "..."} 微博 hot search gentle cloud~weibo hot search scraper {"mode": "hot band", "includeScores": true} 微博 posts zhorex~weibo scraper (see actor input schema) B站 videos zhorex~bilibili scraper (see actor input schema) 京东 search zen studio~jd com search scraper {"keyword": "...", "maxProducts": N} 京东 products sian.agency~jd com product scraper {"operation": "productSearch", "keyword": "...", "maxPages": 1} 淘宝 products sian.agency~taobao tmall product scraper {"operation": "keywordSearch", "keyword": "...", "maxPages": 1} 1688 wholesale zen studio~1688 wholesale scraper (see actor input schema) 闲鱼 search zen studio~goofish xianyu search scraper (see actor input schema) TikTok clockworks~tiktok scraper {"hashtags": [...]} or {"profiles": [...]} Two step Xiaohongshu workflow (search → note detail): The search scraper ( zen studio~rednote search scraper ) returns only a truncated desc (~60 chars). For the full post body, run a second call with sian.agency~xiaohongshu rednote scraper in noteDetail mode, passing the id and xsec token from the search result row. This is the only reliable way to get full note text for price/lodge/itinerary details. Usage: Billing & cost control — read this before every Apify call: Apify uses pay per event billing: (actor start + result count × per result + add ons) × 2 credits. The dominant factor is result count × per result price , and each actor prices differently ($0.003–$0.007/result). Unknown actors default to the highest tier ($0.007). Full pricing table and estimation examples: reference/apify pricing.md . You MUST estimate cost before calling: 1. Look up the actor's per result price (table in reference/apify pricing.md ; unlisted = $0.007). 2. Estimate result count from input params ( maxResults , maxProducts , etc.). If no limit is set, assume 100+. 3. Calculate: result count × per result × 2 = estimated credits. 4. If estimate 5 credits, tell the user the cost before proceeding. Hard spending cap — proxy enforced default + per call override: The proxy automatically injects maxTotalChargeUsd=$2.5 (≈ 5 credits) on every actor run that doesn't already specify one. Apify terminates the run when the budget is hit and returns whatever results were collected so far — no overcharging, no runaway costs. apify run() also passes max charge usd (same default $2.5), which takes precedence over the proxy default. This means: Default (no extra param): capped at ≈ 5 credits per call. Safe for routine searches, profile lookups, small batch scraping. Need more data? Raise the cap explicitly: max charge usd=10 (≈ 20 credits). Use when the user explicitly wants a large dataset and you've already told them the estimated cost. First time test: lower it: max charge usd=0.5 (≈ 1 credit), limit results to 5–10, verify output quality before scaling up. Disable cap entirely: max charge usd=None . Never do this unless the user explicitly asks for an uncapped run after being warned of the potential cost. Small batch test first: When using an actor for the first time, set max charge usd=0.5 (≈ 1 credit) and limit results to 5–10. Verify output quality before scaling up. Error handling: 400 run failed → bad input (wrong field name). Fetch the actor's input schema page and fix. 401 → proxy misconfigured (should not happen). Report to user. Empty result [] → actor ran but found nothing. Try different keywords or another actor. Timeout → increase timeout param (default 180s). Some actors are slow. Cost discipline: Apify actors are more expensive than Firecrawl/ScrapeCreators. Only use Apify when the cheaper options can't get the data (China apps, structured e commerce fields). For a single web page, always try Firecrawl first. What each service is for ScrapeCreators — Social media data extraction (27+ platforms) Use for any request involving social media profiles, posts, videos, comments, transcripts, search, ads, trending content, or engagement metrics. Covers TikTok, Instagram, YouTube, LinkedIn, Facebook, Twitter/X, Reddit, Threads, Bluesky, Pinterest, Snapchat, Twitch, Kick, Truth Social, TikTok Shop, Google search, and link in bio services (Linktree, Komi, Pillar, Linkbio, Linkme, Amazon Shop). Base URL: https://api.scrapecreators.com Auth: No user supplied key needed. sc proxy injects platform credentials automatically — just send the request. The x api key header can be any value or omitted entirely. Do NOT bail out or ask the user for a key if $SCRAPECREATORS API KEY looks unset; that env var is intentionally not required. Method: All endpoints use GET requests with query params. Responses are JSON. Firecrawl — Fallback web page scraper Only a fallback crawler for one web page when ordinary fetching fails. Use POST /v2/scrape with a single url and focused formats like markdown , html , rawHtml , links , summary , or constrained json / question / highlights extraction. Auth: No user supplied key needed. sc proxy injects the Firecrawl credential automatically when you call through core.http client.proxied post — just send the request. Do NOT read $FIRECRAWL API KEY from env, do NOT check .env for it, and do NOT ask the user for a Firecrawl key if it looks unset; that env var is intentionally not required. The same proxy injection model as ScrapeCreators applies here. Do not use Firecrawl crawl/map/search/agent/browser endpoints. Do not request screenshots, audio, branding, images, or browser actions unless the proxy policy is expanded later. ScrapeCreators — Intent routing Map user intent to the right endpoint. Endpoint paths use the pattern /v1/platform/action . Important: After selecting an endpoint from the tables below, fetch its OpenAPI spec at https://docs.scrapecreators.com/{path}/openapi.json for full parameter details, types, and example response before making the actual API call. For example: https://docs.scrapecreators.com/v1/tiktok/profile/openapi.json Profiles / User Info Platform Endpoint Primary Param Example TikTok /v1/tiktok/profile handle stoolpresidente Instagram /v1/instagram/profile handle jane YouTube /v1/youtube/channel handle, channelId, or url ThePatMcAfee

web-crawler