Investigates network, streaming, and protocol failures with falsification-first layered experiments. Use for ECONNRESET, HTTP/2 RST_STREAM, SSE or long-polling stalls, fixed-time drops, CDN/proxy/CGNAT timeouts, client-side proxy/TUN misrouting, CNAME-rule surprises, certificate errors, generic thro

Debugging Network Issues Evidence driven investigation methodology for incidents where the obvious cause is probably wrong. Built from a real 5 hour production case (see [references/case sse rst 130s.md](references/case sse rst 130s.md)) where assumption stacking wasted hours that a 10 minute layered experiment would have resolved. Apply this skill when the user reports a network/streaming/protocol symptom and the investigator feels tempted to diagnose from one log line or one circumstantial data point. The skill's job is to slow that reflex down. Triage first — is this a known domain? Before applying the general methodology below, check whether the symptom points at a stack that already has a dedicated skill in this repo. Those carry the domain specific symptom→cause→fix tables this skill deliberately stays general about — start there, and come back here for methodology if the root cause turns out to be elsewhere. If the symptom is… Start with macOS Tailscale ⨯ proxy/VPN conflict (Shadowrocket / Clash / Surge): tailscale ping works but SSH/curl/git fails, Connection closed by 198.18.x.x , TUN DNS hijack, ~60s getaddrinfo resolver stall tunnel doctor Cloudflare config: ERR TOO MANY REDIRECTS , SSL mode mismatch, DNS / proxy status issues behind the orange cloud cloudflare troubleshooting Windows App / AVD / W365 RDP connection quality: WebSocket instead of UDP Shortpath, high RTT, STUN/TURN interference windows remote desktop connection doctor Client side proxy / VPN / TUN misrouting: one specific site fails with ERR CONNECTION CLOSED or SSL ERROR SYSCALL , other sites work, DNS returns fake/TUN IPs, and adding a PROXY rule did not help this skill — read [references/case proxy tun cname override.md](references/case proxy tun cname override.md) first TLS certificate verification errors ( UNKNOWN CERTIFICATE VERIFICATION ERROR , a cert for the wrong site) or mid handshake EOFs on every DIRECT routed/domestic domain at once, while proxied domains work — and any proxy health watchdog still reports green tunnel doctor (TUN DIRECT split brain step) The proxy is reachable and the OS/Tailscale route is healthy, but bulk transfers are slow and changing the active exit/node changes the measured rate tunnel doctor — proxy node / exit / chain capacity branch Nothing errors — it is just slow. Every request returns 200, latency and health checks look fine, but transfers crawl, bulk jobs overrun their estimates, or files arrive truncated this skill — go straight to [Step 0.7]( step 07 throughput collapse when nothing errors and everything is slow); the error driven steps have nothing to bite on If none match — or you tried a domain skill and the evidence points elsewhere — continue below. The methodology generalizes to any multi layer system. Note for this skill specifically : If the symptom is a Cloudflare 524/522 on a large POST body (e.g., /<openrouter path with Content Length 1 MB), the failure is often upload time to origin exceeding Cloudflare's origin read timeout , not backend slowness. Use the upload vs processing checklist below before assuming a backend stall. Core principles 1. Evidence over assumption If you cannot point to a concrete artifact — log line, pcap frame, probe output, metric sample — you are guessing, not diagnosing. Before stating "X is the cause", require yourself to name the direct evidence. If it does not exist yet, add instrumentation (see [references/instrumentation patterns.md](references/instrumentation patterns.md)) or capture it (see [references/packet capture recipes.md](references/packet capture recipes.md)) before continuing. 2. Falsification over confirmation N independent sources "confirming" a hypothesis does not make it true. One falsifying observation rules it out. Before acting on a hypothesis, answer: "What observation would make me abandon this hypothesis?" If the answer is "nothing" or "I cannot think of one", the hypothesis is unfalsifiable and must not drive the investigation. If the answer is concrete, go look for that observation before committing to action. 3. Layered isolation Multi hop systems (client → CDN → LB → reverse proxy → app → upstream) concentrate bugs at the seams between layers. When a symptom could plausibly come from several layers, do not reason about which layer; test . The canonical technique: run the same logical request through three or more paths that differ by exactly one hop, then compare where the symptom appears. This resolves in minutes what stacking hypotheses cannot resolve in hours. See [references/layered isolation experiment.md](references/layered isolation experiment.md). The same technique isolates capacity, not just correctness. When the symptom is a rate rather than an error, vary the stack instead of the hop : measure the same direction over two channels that share the network path but share no application code. It is the cheapest way to stop tuning an application that was never the bottleneck. Resist compressing the reading into "agreement means the path" — that holds only when both channels are slow ; two fast channels mean the probe failed to reproduce the symptom and prove nothing at all. See Step 0.7 for how to read each of the four outcomes. 4. Counter review before committing Before committing to a root cause or shipping a fix, have independent reviewers challenge the conclusion — not confirm it. Agents are good at surfacing risks a single investigator did not think of; they are bad at weighing them. Apply the four question filter (see [references/counter review pattern.md](references/counter review pattern.md)) to every finding before it shapes action. 5. A green health check certifies only the path it probes Monitors watch the paths their authors thought to probe. Multi plane systems — a TUN proxy with a DIRECT plane and a proxied plane, a service with a data plane and a control plane, a stack with a backend API and a web login page — fail one plane at a time, and a watchdog that probes only the other plane stays green through the entire outage. In the incident behind this principle, a proxy health daemon probed one overseas endpoint through the proxy every 5 minutes and logged "healthy" for 2+ hours while the direct plane was completely down. Before accepting "the monitor says it's healthy" as evidence, ask: which exact path does that check exercise? Its green counts as evidence for that path alone. Enumerate the planes the system actually forwards or serves, and probe the failing one directly — the check that would have caught the outage is usually one curl away. A check also certifies only the quantity it is large enough to measure. Path coverage is one axis; scale is the other, and it is the one that hides throughput collapse. A liveness probe returns a few hundred bytes, so its timing is dominated by handshake and round trip — in the case study it answered in 40 ms all day, unchanged, across a 100× swing in the link's actual capacity, and it was not lying: it genuinely did what it measures. Same for ping /RTT, which times a small packet's round trip and says nothing about capacity. The practical rule: a probe measures the link only once transfer time dominates its total time. So size it by the answer you need, not by convenience — if a probe returns in well under a second, essentially all of that was setup and you have measured setup. Sidestep the sizing question entirely by budgeting time instead of bytes (stream for N seconds, divide what arrived by N), which is what Step 0.7's commands do and why they stay cheap on a link that is already crawling. Extend the question to: which path, and at what scale? 6. Exhaust the instruments before asking the user — escalate only what only they can provide A user's spoken observation ("the Wi Fi drops for a few seconds", "some apps report connection errors") is a lead , not a question to hand back. Treat each open sub question it contains as presumed observable, and search for the instrument before asking: monitor/probe ledgers (where a health daemon exists, its log holds the failure timetable of the paths it probes — in both directions: a green ledger never falsifies the user's observation (Principle 5), and a red one says nothing about paths it never probed), system logs (kernel link/disassociation events settle "did Wi Fi physically drop"; per process error bursts settle "which apps failed" only for apps that log there), and control channels on the same network (a second host over a different medium — a wired box with no proxy, a hotspot comparison, even the router's own status page; when no control channel exists, say so explicitly rather than treating the question as answered). When the instrument search comes back empty for a sub question, the move is Step 4 — stand up instrumentation and wait for recurrence — not "ask the user to recall". Only three categories legitimately reach the user mid investigation: 1. Decision rights — production changes, restarts, spending, anything past your authorization boundary. 2. Physical actions and first person observations no instrument captured — unplugging a box, reading a label password, whether the Wi Fi icon itself disappeared on a device you have no telemetry for. 3. Credentials or access you do not have — a management password that exists only on a device label. Observed 2026 09 10: a home network investigation closed its report with four questions for the user. "Which apps reported errors" — its diagnostic content, a per plane failure timetable — was already in the health daemon's ledger (the app name list itself lives only on the user's screen, and adds no layer information the timetable lacks); "did Wi Fi actually drop" was settled by twelve hours of kernel logs with zero disassociation events; both went to the user anyway. The fourth — "is the retired router still broadcasting" — is the honest counter example: its ARP absence proved nothing (Trap 14: a probe certifies only the L2 domain it ran on), the platform redacted the Wi Fi scan, so that one correctly reached the user as a five second physical check. For every question you hand the user, write one sentence in the report naming why no instrument can answer it and which of the three categories it falls in. If you cannot produce that sentence, the question is not exhausted — go back to the instruments; do not ask. Workflow Copy this checklist into the investigation notes and check items off: Step 0: Scope A tight scope is the difference between a 20 minute investigation and a 5 hour one. Before looking at anything, extract: Exact error string (copy paste, not paraphrase). socket closed is not the same as ECONNRESET is not the same as HTTP/2 RST STREAM INTERNAL ERROR (err 2) . Exact timestamps (ISO 8601 with timezone, not "yesterday evening") Reproducibility (every time / intermittent / only specific users) Who is affected, who is not (differential observations narrow the search) What changed recently (deploys, config, upstream dependencies, client versions) Distinguish symptom from diagnosis. "Slow" is not a symptom. "Request took 130.898s then returned HTTP/2 INTERNAL ERROR" is. But do not let that sentence route a real incident into the bin. It demands quantification, not an error code — and a whole failure family quantifies as a rate with no error code anywhere:

debugging-network-issues