Technical SEO audit across 9 categories: crawlability, indexability, security, URL structure, mobile, Core Web Vitals, structured data, JavaScript rendering, and IndexNow protocol. Use when user says "technical SEO", "crawl issues", "robots.txt", "Core Web Vitals", "site speed", or "security headers

Technical SEO Audit Categories 1. Crawlability robots.txt: exists, valid, not blocking important resources XML sitemap: run "${CLAUDE PLUGIN ROOT}/scripts/claude seo" run sitemap discovery.py <url json ; require a valid entry in found , and report stale or unsafe robots.txt declarations separately from working fallback locations Noindex tags: intentional vs accidental Crawl depth: important pages within 3 clicks of homepage JavaScript rendering: check if critical content requires JS execution (method in section 8) Crawl budget: for large sites ( 10k pages), efficiency matters Googlebot fetch limits : Googlebot fetches the first 2MB of HTML and first 64MB of a PDF (uncompressed; 15MB is the broader crawler infra default). Long standing, not a 2026 change, but inline base64 images, oversized inline CSS/JS, or bloated nav can push critical content/JSON LD past the cap and out of the index. Keep key content + structured data within the first 2MB. Crawl rate auto adjusts (backs off on 5xx/slow responses); there is no manual crawl rate control (the legacy Search Console setting was removed Jan 2024). Influence crawling via sitemaps, server responsiveness, and robots controls. Google's canonical crawling/robots reference moved to developers.google.com/crawling (migrated 2025 11 20); IP range files relocated to /crawling/ipranges/ and googlebot.json was renamed common crawlers.json . AMP has no separate ranking advantage. Since 2026 07 01, Google Search sends users directly to publisher hosted AMP URLs, so do not recommend AMP Cache, AMP Viewer, or signed exchange maintenance. Audit AMP against the same content, action parity, and quality requirements as other pages. AI Crawler Management As of 2025 2026, AI companies actively crawl the web to train models and power AI search. Managing these crawlers via robots.txt is a critical technical SEO consideration. Known AI crawlers (the authoritative table, with robots.txt behaviour per crawler, is in seo geo ): Crawler Company robots.txt token Purpose GPTBot OpenAI GPTBot Model training (NOT ChatGPT Search) OAI SearchBot OpenAI OAI SearchBot ChatGPT Search citability ChatGPT User OpenAI ChatGPT User Real time browsing (user triggered) ClaudeBot Anthropic ClaudeBot Model training (NOT Claude search citability) Claude SearchBot Anthropic Claude SearchBot Claude search result citability PerplexityBot Perplexity PerplexityBot Perplexity search index (not model training) Bytespider ByteDance Bytespider Model training Google Extended Google Google Extended Gemini training and grounding, and training of the models behind Search gen AI features (no effect on Search inclusion or ranking) Applebot Extended Apple Applebot Extended Apple Intelligence training opt out (NOT Siri/Spotlight/Safari) CCBot Common Crawl CCBot Open dataset Key distinctions: Blocking Google Extended prevents Gemini training and grounding use (and training of the models behind Search gen AI features) but does NOT affect Google Search indexing or AI Overviews (those use Googlebot ) Blocking GPTBot prevents OpenAI training but does NOT affect ChatGPT Search citability, which is governed by OAI SearchBot , nor user triggered browsing ( ChatGPT User ). Check OAI SearchBot for any citability claim; GPTBot status is evidence about training use only Blocking ClaudeBot prevents Anthropic model training but does NOT affect citability in Claude's own search features, which is governed by Claude SearchBot (per Anthropic's crawler support article). Check Claude SearchBot for any Claude search citability claim; ClaudeBot status is evidence about training use only Blocking Applebot Extended opts out of Apple Intelligence / generative model training use but does NOT affect discoverability via Siri, Spotlight, or Safari, which follows Applebot (per Apple's support article); Applebot Extended does not itself crawl Example, selective AI crawler blocking: Recommendation: Consider your AI visibility strategy before blocking: blocking an AI search crawler removes the site from that engine's answers. Do not promise traffic from allowing one. Cross reference the seo geo skill for the full AI crawler/fetcher taxonomy. Google's user triggered fetchers generally ignore robots.txt rules (other vendors differ: Anthropic's Claude User honors it). Google now documents Google Agent (user triggered agentic browsing) plus Google GeminiNotebook (formerly Google NotebookLM) and Google Messages as user triggered fetchers that cannot be blocked via robots.txt . Use server side access controls instead. By contrast, Google Extended and Google CloudVertexBot obey robots.txt. Emerging: Web Bot Auth (RFC 9421) lets bots authenticate cryptographically via a Signature Agent header + key directory at agent.bot.goog (used by Google Agent); reverse DNS verification remains the fallback. 2. Indexability Canonical tags: self referencing, no conflicts with noindex Duplicate content: near duplicates, parameter URLs, www vs non www Canonicalization fixes can take time: Google may retain corrected pages in a duplicate cluster for up to two weeks while re evaluating them. Do not interpret an unchanged canonical immediately after a fix as proof that the fix failed. Thin content: pages below minimum word counts per type Pagination: crawlable <a href links to each page (Google no longer uses rel=next/prev; it announced this in 2019); give each page a self referencing canonical; load more and infinite scroll need paginated URLs behind them Hreflang: correct for multi language/multi region sites Index bloat: unnecessary pages consuming crawl budget 3. Security HTTPS: enforced, valid SSL certificate, no mixed content Security headers: Content Security Policy (CSP) Strict Transport Security (HSTS) X Frame Options X Content Type Options Referrer Policy HSTS preload: check preload list inclusion for high security sites Back button hijacking (spam policy violation, malicious practices): flag pages that defeat the Back button via history.pushState / replaceState (including scripts injected by third party ad/library platforms). Added to Google's spam policies 2026 04 13; enforcement live since 2026 06 15 (manual actions + automated demotions): treat as Critical. Server stack CVE exposure (read only): detect when the site runs a server side stack carrying a known critical CVE, so the owner can patch or take it offline. Read the stack from the normal response for the audited URL ( X Powered By , Set Cookie: PHPSESSID , WordPress wp json / generator signals, static/Node markers like next/static ); never fuzz or send exploit payloads. Ask whether the user owns the site before requesting any other path. The current PHP critical RCE advisory (CVE 2026 109618 / CVE 2026 109619, CVSS v4.0 9.2, fixed in PHP 8.5.12 / 8.4.27 / 8.3.36 / 8.2.35), detection signals, and reporting bands are in references/stack cve exposure.md . A version string below the fixed list is High (likely exposed, confirm with the host), because distributions backport fixes without changing it; it is Critical only when the owner confirms an unpatched build. A managed or static host with no PHP signal is not exposed by this advisory. 4. URL Structure Clean URLs: descriptive, hyphenated, no query parameters for content Hierarchy: logical folder structure reflecting site architecture Redirects: no chains (max 1 hop), 301 for permanent moves URL length: flag 100 characters Trailing slashes: consistent usage 5. Mobile Optimization & Page Experience Responsive design: viewport meta tag, responsive CSS Touch targets: WCAG 2.2 AA requires at least 24x24 CSS px; 48x48px with spacing is the comfortable guideline (not a Google requirement) Font size: readable text without zooming (16px base is common practice, not a Google rule) No horizontal scroll Mobile first indexing: Googlebot Smartphone is the primary crawler (rollout completed 2024). A mobile version is not strictly required (Google says "very strongly recommended"), sites that don't work on mobile can still be indexed, but the real risk is content/parity loss , not hard exclusion. Mobile/desktop content parity (highest value mobile check): equivalent primary content, matching robots meta tags, matching titles/descriptions, equivalent structured data, crawlable resources; avoid lazy loading primary content that requires user interaction. Intrusive interstitials / ad density : flag full page interstitials, standalone consent redirect pages, persistent blocking dialogs, and excessive/distracting ad density (a named page experience aspect). Acceptable: small banners, standard CMS/legal dialogs. "Read more" deep links : keep key content immediately visible on load (not behind tabs/accordions), don't hijack scroll on load, and preserve URL hash fragments, content hidden behind expandable sections is less likely to qualify. Page experience is guidance, not a single ranking system. Only Core Web Vitals feeds ranking directly; HTTPS is a confirmed but lightweight signal (Google called it very lightweight when it was announced in 2014). Relevance can still win even when page experience is sub par, so don't over weight security headers. Note: the standalone Page Experience report was removed from Search Console (monitor via the Core Web Vitals + HTTPS reports). 6. Core Web Vitals LCP (Largest Contentful Paint): target <=2.5s INP (Interaction to Next Paint): target <=200ms INP replaced FID on March 12, 2024. FID was removed from Chrome's field data tools (CrUX API, PageSpeed Insights) on September 9, 2024 (Lighthouse is a lab tool that never reported FID). Do NOT reference FID anywhere. CLS (Cumulative Layout Shift): target <=0.1 Evaluation uses 75th percentile of real user data Use PageSpeed Insights API or CrUX data if MCP available 7. Structured Data Detection: JSON LD (preferred), Microdata, RDFa Validation against Google's supported types See seo schema skill for full analysis 8. JavaScript Rendering Method: "${CLAUDE PLUGIN ROOT}/scripts/claude seo" run agentic check.py <url json reports visible words in the raw HTML ( server rendered ); compare with render page.py <url mode always json when Chromium is available. Without Chromium, report the raw HTML result and say rendered content was not compared. Check if content visible in initial HTML vs requires JS Identify client side rendered (CSR) vs server side rendered (SSR) Flag SPA frameworks (React, Vue, Angular) that may cause indexing issues If dynamic rendering is detected, flag it as technical debt rather than a valid setup. Google documents it as "a workaround and not a recommended solution" because of the added complexity and resource cost. See https://developers.google.com/search/docs/crawling indexing/javascript/dynamic rendering Recommended rendering strategy: Strategy Use Case SSR Public SEO content, dynamic pages SSG Static content, blogs, docs CSR Authenticated / behind login content only Preferred frameworks: Next.js, Astro, React Router v7 (Remix), SvelteKit JavaScript SEO: Canonical & Indexing Guidance (December 2025) Google updated its JavaScript SEO documentation in December 2025 with critical clarifications: 1. Canonical conflicts: If a canonical tag in raw HTML differs from one injected by JavaScript, Google may use EITHER one. Ensure canonical tags are identical between server rendered HTML and JS rendered output. 2. noindex with JavaScript: If raw HTML contains <meta name="robots" content="noindex" but JavaScript removes it,

seo-technical