Extract structured data from multiple web pages using Playwright with built-in ethical crawling practices including rate limiting, robots.txt compliance, and error monitoring. Use when asked to "scrape data from", "extract information from pages", "collect data from site", "crawl multiple pages", or

Playwright Web Scraper Extract structured data from multiple web pages with respectful, ethical crawling practices. When to Use This Skill Use when extracting structured data from websites with "scrape data from", "extract information from pages", "collect data from site", or "crawl multiple pages". Do NOT use for testing workflows (use playwright e2e testing ), monitoring errors (use playwright console monitor ), or analyzing network (use playwright network analyzer ). Always respect robots.txt and rate limits. Quick Start Scrape product listings from an e commerce site: Table of Contents 1. Core Workflow 2. Rate Limiting Strategy 3. URL Validation 4. Data Extraction 5. Error Handling 6. Processing Results 7. Supporting Files 8. Expected Outcomes Core Workflow Step 1: Prepare URL List Create a text file with URLs to scrape (one per line): Validate URLs and check robots.txt compliance: Step 2: Initialize Scraping Session Navigate to the site and take a snapshot to understand structure: Identify CSS selectors for data extraction using the snapshot. Step 3: Implement Rate Limited Crawling Use random delays between requests (1 3 seconds minimum): Step 4: Extract Structured Data Use browser evaluate to extract data with JavaScript: See references/extraction patterns.md for comprehensive extraction patterns. Step 5: Handle Errors and Rate Limits Monitor for rate limiting indicators: Step 6: Aggregate and Store Results Save results to JSON file: Process and convert to desired format: Rate Limiting Strategy Minimum Delays Always add delays between requests: Standard sites : 1 3 seconds (random) High traffic sites : 3 5 seconds Small sites : 5 10 seconds After errors : Exponential backoff (5s, 10s, 20s, 40s) Implementation Adaptive Rate Limiting Adjust delays based on response: Response Code Action 200 OK Continue with normal delay (1 3s) 429 Too Many Requests Increase delay to 10s, retry 503 Service Unavailable Wait 60s, then retry 403 Forbidden Stop scraping this domain See references/ethical scraping.md for detailed rate limiting strategies. URL Validation Use validate urls.py before scraping to ensure compliance: Output includes : URL format validation Domain grouping robots.txt compliance check Summary statistics Data Extraction Basic Pattern Pagination Pattern See references/extraction patterns.md for: Advanced selectors Data cleaning patterns Table extraction JSON LD extraction Shadow DOM access Error Handling Network Errors Content Validation Monitoring Indicators Check for blocking/errors: Processing Results View Statistics Output: Convert Formats Combine Statistics with Conversion Supporting Files Scripts scripts/validate urls.py Validate URL lists, check robots.txt compliance, group by domain scripts/process results.py Convert scraped JSON to CSV/JSON/Markdown, view statistics References references/ethical scraping.md Comprehensive guide to rate limiting, robots.txt, error handling, and monitoring references/extraction patterns.md JavaScript patterns for data extraction, selectors, pagination, tables Expected Outcomes Successful Scraping With Error Handling Rate Limit Detection Expected Benefits Metric Before After Setup time 30 45 min 5 10 min Rate limit errors Common Rare robots.txt violations Possible Prevented Data format conversion Manual Automated Error detection Manual review Automated monitoring Success Metrics Success rate 95% (pages successfully scraped) Rate limit errors < 5% of requests Valid data rate 90% (complete records) Scraping speed 6 12 requests/minute (polite crawling) Requirements Tools Playwright MCP browser tools Python 3.8+ (for scripts) Standard library only (no external dependencies for scripts) Knowledge Basic CSS selectors JavaScript for data extraction Understanding of HTTP status codes Awareness of web scraping ethics Red Flags to Avoid ❌ Scraping without checking robots.txt ❌ No delays between requests (hammering servers) ❌ Ignoring 429/503 response codes ❌ Scraping personal/private information ❌ Not monitoring console for blocking messages ❌ Scraping sites that explicitly prohibit it (check ToS) ❌ Using scraped data in violation of copyright ❌ Not handling pagination correctly (missing data) ❌ Hardcoding selectors without fallbacks ❌ Not validating extracted data structure Notes Default to polite crawling : 1 3 second delays minimum, adjust based on site response Always check robots.txt first : Use validate urls.py before scraping Monitor console and network : Watch for rate limit warnings and adjust delays Start small : Test with 5 10 URLs before scaling to hundreds Save progress : Write results incrementally in case of interruption Respect ToS : Some sites prohibit scraping in their terms of service Use descriptive user agents : Identify your bot clearly Handle errors gracefully : Log failures for manual review, don't crash

playwright-web-scraper