Goal-driven E2E testing where a browser agent (Playwright MCP / computer-use) reads a natural-language goal and explores the app via the accessibility tree to assert outcomes — no pre-written script. Covers when intent-driven beats scripted, making agent runs deterministic (pinned model, temperature

<objective Use a browser agent to explore a scoped natural language goal and collect evidence from the application. Keep stable, high frequency critical paths scripted. A browser agent can adapt its navigation when a page changes, but its exploration remains variable. A project owned oracle must independently establish the result before CI accepts it. Leave a bounded run, a trace, a typed evidence report, and a path to a durable Playwright regression test. </objective Quick Route Situation Go to Stand up a goal driven run from scratch Discovery + references/setup.md Decide agentic vs scripted for a given flow Fit: Intent Driven vs Scripted Agent passes one run, fails the next Repeatability "How does it click without screenshots?" Interaction Model Runs are slow / burning tokens Cost and Latency Agent reports false success Success Assertion (the Oracle) Flow is stable — make it permanent Graduation → references/graduation and ci.md Block a merge on the goal CI Gating → references/graduation and ci.md Canvas / no accessibility tree Canvas Fallback → references/graduation and ci.md Discovery Questions First, check .agents/qa project context.md in the project root and skip anything it already answers (stack, environments, seed/reset tooling, model access). 1. Which flow, and how often does its UI change? Fast changing/experimental UI favors intent driven; a stable critical path (login) favors scripted. This decides the whole approach. 2. Is there a seeded fixture and a way to reset state? Repeatable comparisons require controlled data and a per run reset. If neither exists, that is step zero. 3. Can you deep link past auth to a seeded entry point? Re driving login every run is the biggest avoidable cost; a seeded entry URL scopes the goal and cuts steps. 4. What is the unambiguous success oracle? Specific account text, a /dashboard URL, an order number — plus a forbidden state. "No error" is not an oracle. 5. Does the target render to canvas / WebGL? No accessibility tree means snapshot first won't work; plan the vision fallback or instrument the canvas with ARIA. 6. Which model and budget? Choose an available model version and enforce step, time and token budgets up front. Record the installed adapter and actual model returned by the service. Core Principles 1. Choose intent driven exploration where it helps. The agent reads a goal and chooses controls from the current accessibility tree. Compare its observed cost, latency and repeatability with scripted checks before using it for a critical path. Role based scripted locators also survive many layout and CSS changes. 2. Control variance and measure repeatability. Pin supported model settings, reset the fixture, bound steps and wall time, and evaluate a separate oracle. Temperature zero and a pinned model reduce some variability; they do not guarantee identical navigation or verdicts. Run repeated healthy and broken controls before enabling a CI gate. 3. The oracle lives outside the agent. Never let the LLM self grade "looks good." Success is a checkable assertion against the final browser snapshot — specific expected text, a URL, AND a forbidden state negative check — evaluated by your harness, not the model. 4. Accessibility tree first, pixels last. browser snapshot returns roles, refs, and accessible names in a machine readable form. Screenshots, pixel coordinates, vision, and OCR are a scoped last resort for canvas only, never the default. 5. Graduation is the goal, not perpetual agent runs. Once a flow is stable, promote the run to a durable scripted tests/ .spec.ts with role based locators. An agent that has been green for two weeks should become a fast, free regression test — keep the agent for exploration, not for guarding a settled path. Fit: Intent Driven vs Scripted The decision is per flow, not per project. Run risk based testing first if you need the risk map; this table is the routing rule once you have it. Flow characteristic Use Why Stable, high frequency critical path (login, payment) Scripted + pinned ( playwright automation ) Runs every PR with a maintained sequence and explicit assertions. Fast changing / experimental UI (a dashboard that churns weekly, a redesign in flight) Agentic / intent driven Selectors would break constantly; a goal survives layout churn. Hard to locate flow you can't reliably select Agentic The agent finds the control by role/name instead of you reverse engineering a selector. Exploratory smoke / "does the happy path still work at all" Agentic One NL goal covers a lot of ground without a maintained script. Anything in CI that must never falsely pass Scripted, OR agentic with a hard oracle Non determinism is a false pass risk you must actively cap. The rule, stated plainly: keep stable critical paths scripted and pinned; point intent driven agents at fast changing UI and exploratory smoke. Do not move everything to the agent. Measure cost, latency and repeatability per flow; not every test should be agentic. The Interaction Model (accessibility tree first) Playwright MCP is not computer use with screenshots and pixel coordinates. It is accessibility tree first: 1. browser navigate to the seeded entry URL. 2. browser snapshot returns the accessibility tree — each interactive element as a role , a current ref , and its accessible name (from ARIA/labels). References and snapshot size depend on the live page; refresh after a navigation or DOM change. 3. The agent picks an element by ref and calls browser click or browser type . 4. browser wait for waits on text appearing/disappearing — never a fixed sleep. 5. Re browser snapshot after the DOM changes; assert against that tree. Use semantic snapshots for ordinary controls because they expose roles and accessible names directly. Measure token use and timing on the actual page. Use screenshots when visual information matters; do not infer correctness from a screenshot alone. browser take screenshot is for human evidence only, never as the assertion input. See references/setup.md for the MCP registration, the full tool table, and the goal prompt. Repeatability: making a run trustworthy in CI A run that passes once and fails the next with no app change is not yet a test. The fix is never "just retry" or bumping temperature for "smarter" exploration — that adds variance. Pin the variables instead: Lever Setting Model Pinned model id (e.g. claude haiku 4 5 20251001 ), never latest Sampling Lowest supported sampling variance ; record the setting, without claiming determinism Data Seeded fixture + reset/seed the database before every run Scope Bounded step budget ( maxSteps ), e.g. 18 — exceeding it FAILS, never auto retries Oracle Independent checks against fresh application observations Evidence Assert on the accessibility tree , never a screenshot diff Avoid: temperature: 0.7 or 1 for exploration, retry until pass loops, waitForTimeout sleeps, and screenshot based assertions. Each one hides flakiness rather than removing it. Adapter configuration examples are in references/setup.md ; these fields need an actual host adapter to enforce them. Success Assertion: the Oracle (where agents fail silently) This is the sharpest failure mode: the agent reports success while stuck on the login page, because "page loaded / no error / looks good" was accepted as success and the LLM was allowed to self grade. Force an explicit oracle the harness checks — never the agent. For the goal "sign in as an existing user and confirm the dashboard shows the right account name" : The positive checks (specific account name + /dashboard URL) prove where it landed; the negative check (must NOT be on the login page) is what kills the false pass. "No error," "didn't crash," "screenshot looks correct," and "trust the agent" are not success criteria. Cost and Latency Measure agent latency and cost against the scripted alternative. Model round trips can dominate a small browser flow. Control unnecessary work with explicit budgets: Step budget — keep maxSteps low and enforced; fewer round trips, less drift. Model selection — use measured task accuracy, latency and cost to choose an available model. Pin the model version where the provider supports it. Prompt caching — cache the static system prompt, tool schemas, and goal; they repeat every run. Scope via a seeded entry point — one narrow goal per run, deep linked past login instead of re driving it each time. Snapshot size — scope snapshots to the relevant region where supported. Measure payload size; large DOMs can make text snapshots expensive too. Backwards moves to reject: "use the biggest model for every step," "raise the step limit," "screenshot every step," and running with no budget at all. See references/setup.md . Graduation and CI Gating Promote a stabilized goal into a durable scripted test, and gate merges on the verdict. Both are detailed in references/graduation and ci.md ; the essentials: Graduate with Playwright Test Agents (planner / generator / healer, shipped in Playwright v1.56.0 ). npx playwright init agents loop=claude . The planner writes a Markdown test plan to specs/<flow .md ; the generator turns it into tests/<flow .spec.ts with role based locators ( getByRole , getByLabel , getByText ) verified against the live DOM; the healer repairs broken locators. This is the promotion path — not "keep running it as an agent," not recorded clicks, not page.locator('xpath=...') , not data testid only. Gate CI with scripts/agent result gate.mjs from a full QA Skills checkout. It runs a reviewed oracle command and requires typed, nonempty evidence for the exact run, target and build. failed and blocked both exit nonzero. The host agent adapter must separately enforce step/time budgets; a model produced {"passed": true} cannot pass this gate. Canvas with no accessibility tree: prefer instrumenting the canvas with ARIA; as a scoped last resort enable caps=vision to unlock browser mouse click xy for that flow only. browser snapshot will not work on a raw canvas, but don't make coordinates the default and don't abandon agentic testing. Migrating a brittle script to a goal (honest tradeoffs) Converting an 80 line script that re types login and walks 6 hardcoded steps into a single NL goal with an explicit success assertion is a real win for a churning flow — but state the downsides honestly: Non determinism / false pass risk — the run could pass falsely; that's why the hard oracle and the negative check are non negotiable. Cost/latency — measure both; enforce a step budget , wall clock limit and seeded entry point. Not every test should be agentic — keep stable paths scripted, and plan to graduate this one back to a scripted test once it stabilizes. Reject the over promise: it is not "strictly better with no downsides," do not "migrate everything," and never drop the assertions to make it pass. Anti Patterns 1. Reflexively writing a scripted Playwright test "Browser test" pattern matches to codegen, so the default is page.goto / page.locator / await expect(page...) / hunting data testid . That misses the entire point. A goal driven agent reads NL intent and explores via browser snapshot — no pre written selectors. 2. "Use the agent for everything" Agent exploration has variable cost, latency and behavior. Stable critical paths (login) stay scripted and pinned; intent driven wins on fast changing UI. Never "always use the agent" or "agents replace all scripted tests."

agentic-browser-testing