This skill activates for web scraping and Actor development. It proactively discovers APIs via traffic interception, recommends optimal strategy (traffic interception/sitemap/API/DOM scraping/hybrid), and implements iteratively. For production, it guides TypeScript Actor creation via Apify CLI.

Web Scraping with Intelligent Strategy Selection When This Skill Activates Activate automatically when user requests: "Scrape [website]" "Extract data from [site]" "Get product information from [URL]" "Find all links/pages on [site]" "I'm getting blocked" or "Getting 403 errors" (loads strategies/anti blocking.md ) "Make this an Apify Actor" (loads apify/ subdirectory) "Productionize this scraper" Input Parsing Determine reconnaissance depth from user request: User Says Mode Phases Run "quick recon", "just check", "what framework" Quick Phase 0 only "scrape X", "extract data from X" (default) Standard Phases 0 3 + 5, Phase 4 only if protection signals detected "full recon", "deep scan", "production scraping" Full All phases (0 5) including protection testing Default is Standard mode. Escalate to Full if protection signals appear during any phase. Adaptive Reconnaissance Workflow This skill uses an adaptive phased workflow with quality gates. Each gate asks "Do I have enough?" — continue only when the answer is no. See : strategies/framework signatures.md for framework detection tables referenced throughout. Phase 0: QUICK ASSESSMENT (curl, no browser) Gather maximum intelligence with minimum cost — a single HTTP request. Step 0a: Fetch raw HTML and headers Step 0b: Check response headers Match headers against strategies/framework signatures.md → Response Header Signatures table Note Server , X Powered By , X Shopify Stage , Set Cookie (protection markers) Check HTTP status code (200 = accessible, 403 = protected, 3xx = redirects) Step 0c: Check Known Major Sites table Match domain against strategies/framework signatures.md → Known Major Sites If matched: use the specified data strategy, skip generic pattern scanning Step 0d: Detect framework from HTML Search raw HTML for signatures in strategies/framework signatures.md → HTML Signatures table Look for NEXT DATA , NUXT , ld+json , /wp content/ , data reactroot Step 0e: Search for target data points For each data point the user wants: search raw HTML for that content Track which data points are found vs missing Check for sitemaps: curl s https://[site]/robots.txt grep i Sitemap Step 0f: Note protection signals 403/503 status, Cloudflare challenge HTML, CAPTCHA elements, cf ray header Record for Phase 4 decision See : strategies/cheerio vs browser test.md for the Cheerio viability assessment QUALITY GATE A : All target data points found in raw HTML + no protection signals? → YES: Skip to Phase 3 (Validate Findings). No browser needed. → NO: Continue to Phase 1. Phase 1: BROWSER RECONNAISSANCE (only if Phase 0 needs it) Launch browser only for data points missing from raw HTML or when JavaScript rendering is required. Step 1a: Initialize browser session proxy start() → Start traffic interception proxy interceptor chrome launch(url, stealthMode: true) → Launch Chrome with anti detection interceptor chrome devtools attach(target id) → Attach DevTools bridge interceptor chrome devtools screenshot() → Capture visual state Step 1b: Capture traffic and rendered DOM proxy list traffic() → Review all traffic from page load proxy search traffic(query: "application/json") → Find JSON responses interceptor chrome devtools list network(resource types: ["xhr", "fetch"]) → XHR/fetch calls interceptor chrome devtools snapshot() → Accessibility tree (rendered DOM) Step 1c: Search rendered DOM for missing data points For each data point NOT found in Phase 0: search rendered DOM Use framework specific search strategy from strategies/framework signatures.md → Framework → Search Strategy table Only search patterns relevant to the detected framework Step 1d: Inspect discovered endpoints proxy get exchange(exchange id) → Full request/response for promising endpoints Document: method, headers, auth, response structure, pagination QUALITY GATE B : All target data points now covered (raw HTML + rendered DOM + traffic)? → YES: Skip to Phase 3 (Validate Findings). No deep scan needed. → NO: Continue to Phase 2 for missing data points only. Phase 2: DEEP SCAN (only for missing data points) Targeted investigation for data points not yet found. Only search for what's missing. Step 2a: Test interactions for missing data proxy clear traffic() before each action → Isolate API calls humanizer click(target id, selector) → Trigger dynamic content loads humanizer scroll(target id, direction, amount) → Trigger lazy loading / infinite scroll humanizer idle(target id, duration ms) → Wait for delayed content After each action: proxy list traffic() → Check for new API calls Step 2b: Sniff APIs (framework aware) Search only patterns relevant to detected framework: Next.js → proxy list traffic(url filter: "/ next/data/") WordPress → proxy list traffic(url filter: "/wp json/") GraphQL → proxy search traffic(query: "graphql") Generic → proxy list traffic(url filter: "/api/") + proxy search traffic(query: "application/json") Skip patterns that don't apply to the detected framework Step 2c: Test pagination and filtering Only if pagination data is a missing data point or needed for coverage assessment proxy clear traffic() → click next page → proxy list traffic(url filter: "page=") Document pagination type (URL based, API offset, cursor, infinite scroll) QUALITY GATE C : Enough data points covered for a useful report? → YES: Go to Phase 3. → NO: Document gaps, go to Phase 3 anyway (report will note missing data in self critique). Phase 3: VALIDATE FINDINGS Every claimed extraction method must be verified. A data point is not "found" until the extraction path is specified and tested. See : strategies/cheerio vs browser test.md for validation methodology Step 3a: Validate CSS selectors For each Cheerio/selector based method: confirm the selector matches actual HTML Test against raw HTML (curl output) or rendered DOM (snapshot) Confirm selector extracts the correct value, not a different element Step 3b: Validate JSON paths For each JSON extraction (e.g., NEXT DATA , API response): confirm the path resolves Parse the JSON, follow the path, verify it returns the expected data type and value Step 3c: Validate API endpoints For each discovered API: replay the request (curl or proxy get exchange ) Confirm: response status 200, expected data structure, correct values Test pagination if claimed (at least page 1 and page 2) Step 3d: Downgrade or re investigate failures If a selector doesn't match: try alternative selectors, or downgrade to PARTIAL confidence If an API returns 403: note protection requirement, flag for Phase 4 If a JSON path is wrong: re examine the JSON structure, correct the path Phase 4: PROTECTION TESTING (conditional) See : strategies/proxy escalation.md for complete skip/run decision logic Skip Phase 4 when ALL true : No protection signals detected in Phases 0 2 All data points have validated extraction methods User didn't request "full recon" Run Phase 4 when ANY true : 403/challenge page observed during any phase Known high protection domain High volume or production intent User explicitly requested it If running : Step 4a: Test raw HTTP access 200 → Cheerio viable, no browser needed for accessible endpoints 403/503 → Escalate to stealth browser Step 4b: Test with stealth browser (if needed) Already running from Phase 1 — check if pages loaded without challenges interceptor chrome devtools list cookies(domain filter: "cloudflare") → Protection cookies interceptor chrome devtools list storage keys(storage type: "local") → Fingerprint markers proxy get tls fingerprints() → TLS fingerprint analysis Step 4c: Test with upstream proxy (if needed) proxy set upstream("http://user:pass@proxy provider:port") Re test blocked endpoints through proxy Document minimum access level for each data point Step 4d: Document protection profile What protections exist, what worked to bypass them, what production scrapers will need Phase 5: REPORT + SELF CRITIQUE Generate the intelligence report, then critically review it for gaps. See : reference/report schema.md for complete report format Step 5a: Generate report Follow reference/report schema.md schema (Sections 1 6) Include Validated? status for every strategy (YES / PARTIAL / NO) Include all discovered endpoints with full specs Step 5b: Self critique Write Section 7 (Self Critique) per reference/report schema.md : Gaps : Data points not found — why, and what would find them Skipped steps : Which phases skipped, with quality gate reasoning Unvalidated claims : Anything marked PARTIAL or NO Assumptions : Things not verified (e.g., "consistent layout across categories") Staleness risk : Geo dependent prices, A/B layouts, session specific content Recommendations : Targeted next steps (not "re run everything") Step 5c: Fix gaps with targeted re investigation If self critique reveals fixable gaps: go back to the specific phase/step, not a full re run Example: "Price selector untested" → run one curl + parse, don't re launch browser Update report with results Step 5d: Record session (if browser was used) proxy session start(name) → proxy session stop(session id) → proxy export har(session id, path) HAR file captures all traffic for replay. See strategies/session workflows.md IMPLEMENTATION (after reconnaissance) After reconnaissance report is accepted, implement scraper iteratively. Core Pattern : 1. Implement recommended approach (minimal code) 2. Test with small batch (5 10 items) 3. Validate data quality 4. Scale to full dataset or fallback 5. Handle blocking if encountered 6. Add robustness (error handling, retries, logging) See : workflows/implementation.md for complete implementation patterns and code examples PRODUCTIONIZATION (on request) Convert scraper to production ready Apify Actor. Activation triggers : "Make this an Apify Actor", "Productionize this", "Deploy to Apify" Core Pattern : 1. Confirm TypeScript preference (STRONGLY RECOMMENDED) 2. Initialize with apify create command (CRITICAL) 3. Port scraping logic to Actor format 4. Test locally and deploy Note : During development, proxy mcp provides reconnaissance and traffic analysis. For production Actors, use Crawlee crawlers (CheerioCrawler/PlaywrightCrawler) on Apify infrastructure. See : workflows/productionization.md for complete workflow and apify/ for Actor development guides Quick Reference Task Pattern/Command Documentation Reconnaissance Adaptive Phases 0 5 workflows/reconnaissance.md Framework detection Header + HTML signature matching strategies/framework signatures.md Cheerio vs Browser Three way test + early exit strategies/cheerio vs browser test.md Traffic analysis proxy list traffic() + proxy get exchange() strategies/traffic interception.md Protection testing Conditional escalation strategies/proxy escalation.md Report format Sections 1 7 with self critique reference/report schema.md Find sitemaps RobotsFile.find(url) strategies/sitemap discovery.md Filter sitemap URLs RequestList + regex reference/regex patterns.md Discover APIs Traffic capture (automatic) strategies/api discovery.md DOM scraping DevTools bridge + humanizer strategies/dom scraping.md HTTP scraping CheerioCrawler strategies/cheerio scraping.md Hybrid approach Sitemap + API strategies/hybrid approaches.md Handle blocking Stealth mode + upstream proxies strategies/anti blocking.md Session recording pro

web-scraping