Scope coverage, build a testing strategy, test the feature with agent-browser, capture screenshots/video (always mandatory), upload to S3, and create a structured test report (never skip)

Task: Test Feature & Generate Demo Report Analyze the current branch's changes, scope the coverage level with the user, build an explicit testing strategy, start the app's dev server, test the feature in a real browser using agent browser , capture screenshots and video ( always — no exceptions ), upload everything to S3, and generate a structured test report ( always — never skip ). Phase 0: Environment Setup 1. Get artifact context IDs Use the active task context when it is present in the prompt. Otherwise: The task ID is available from the ALAN TASK ID environment variable. The conversation ID is available from the ALAN SESSION ID environment variable. Resolve teamId with Alan MCP context/tools before creating the report. When calling create test report , always pass taskId , conversationId , and teamId explicitly. Do not substitute one ID for another. 2. Verify agent browser is available agent browser is a CLI tool by Vercel Labs ( npm: agent browser ) — it is NOT a skill or MCP tool. It provides headless browser automation via bash commands (open, click, fill, screenshot, record). In sandboxes it is pre installed by runtime bootstrap.ts . If missing , install it: If installation fails (e.g. no network, no npm), STOP and tell the user: "agent browser is not available and could not be installed. Install it manually: npm install g agent browser && agent browser install " Do NOT proceed to Phase 2 without a working agent browser — all browser testing depends on it. 3. Create workspace All captures (screenshots, videos) MUST be written to /tmp/alan captures/ . CRITICAL — file path rules: ONLY write capture files to /tmp/alan captures/ . Never anywhere else. NEVER write to .claude/ , project directories, reports/ , or any path inside the repo. .claude/ is a sensitive system directory — writing to it will be blocked and will abort the test run. If you are tempted to create a reports/ or screenshots/ folder anywhere other than /tmp/ , stop and use /tmp/alan captures/ instead. Phase 0.5: Coverage Scoping & Context Gathering (MANDATORY — do before anything else) Never skip this phase. You must understand what the user wants before writing a single test step. 1. Ask about coverage level Use AskUserQuestion to ask the user: Wait for the response before continuing. 2. Ask targeted follow up questions Based on the feature (which you may not know yet — do a quick git diff stat first to get a hint), use AskUserQuestion to ask 1–3 targeted follow up questions. Examples: "Is there an authenticated state I should test? If yes, what test credentials should I use?" "Are there any known edge cases or previous bugs related to this feature?" "What's the definition of 'working correctly' for this feature — what should I see?" "Are there any specific non happy paths you're concerned about (e.g. invalid input, network errors, empty states)?" "Any pages or user roles I should specifically include or exclude?" Do not skip questions if context is unclear. Ask. A well scoped test is 10× more valuable than a blind one. 3. Confirm the plan After gathering answers, summarize back to the user (no tool call needed — just a short message): Coverage level chosen Specific areas / flows to focus on Any known risks or edge cases to target Rough count of test scenarios expected Only proceed to Phase 1 after this confirmation. Phase 1: Discover What to Test 1. Analyze the branch Run these commands to understand what changed: Read the actual diff to understand the feature or bug fix. Identify: What part of the app is affected (which pages, components, API routes) What the expected behavior change is What URL path to navigate to for testing 2. Discover and start the dev server Look at the project to figure out how to run it: 1. Read package.json — check scripts.dev , scripts.start , scripts.serve 2. Check for docker compose.yml / docker compose.yaml / compose.yml 3. Check for Makefile (look for dev or serve targets) 4. Check for Procfile , .env , Pipfile , requirements.txt , Gemfile 5. Check for framework specific files: next.config. , vite.config. , nuxt.config. , angular.json , manage.py , config/routes.rb Start the dev server in the background. Common patterns: 3. Wait for the server Poll until the server is responding: Check common ports if unclear: 3000, 5173, 8080, 4200, 8000, 4000, 3001, 8888. 4. Determine the app URL Parse dev server stdout/stderr for "Local:" or "ready on" messages with URLs Check .env or .env.local for PORT or VITE PORT or similar Test common ports with curl sf http://localhost:PORT /dev/null If you cannot determine it, ask the user via AskUserQuestion 5. Summarize before proceeding Tell the user: What you found in the diff (feature/fix summary) What URL you will test What test scenarios you plan to cover Phase 1.5: Build Testing Strategy & Create Todos (MANDATORY) Before opening the browser, you must have an explicit plan. Do not improvise test steps on the fly. 1. Draft the testing strategy Based on: The coverage level the user chose in Phase 0.5 The diff analysis from Phase 1 The context gathered from the user Write out the full test plan as a structured list. For each scenario, note: Scenario name — short label (e.g. "Happy path: create item") Type — happy path / non happy path / edge case / error state / visual / navigation Steps — what to do Expected outcome — what "pass" looks like Screenshot needed — yes/no Coverage requirements by level: light : happy path only (1–3 scenarios) standard : happy path + 2–4 non happy paths + 1–2 error states comprehensive : happy path + all non happy paths + all error states + boundary values + empty states + responsiveness + navigation Non happy path examples to consider: Invalid / missing required inputs Submitting with no data / empty state Duplicate entries (if applicable) Permission denied / unauthorized access Network error / API failure simulation (if testable) Rapid repeated actions (double click, spam submit) Long input strings / special characters Back button / browser navigation mid flow 2. Create todos for each scenario Use TodoWrite to create a todo entry for each test scenario so progress is tracked. Each todo should be the scenario name. 3. Present the strategy Output the full testing strategy as a numbered list before proceeding. The user should be able to see exactly what will be tested before Phase 2 starts. Phase 2: Test with agent browser agent browser is a CLI tool — invoke it via bash, not as an MCP tool or skill. Docs: https://github.com/vercel labs/agent browser Key commands: open , snapshot , click , fill , screenshot , record , wait , find , close 1. Open the app 2. Start video recording — MANDATORY Always start a video recording before any interaction. No exceptions. If video recording fails for technical reasons, note the failure but continue — screenshots are still required. 3. Take initial screenshot — MANDATORY Always capture the initial page state before any interaction. 4. Execute each scenario from the testing strategy Work through every scenario defined in Phase 1.5. For each scenario: Mark the corresponding todo as in progress Use agent browser snapshot i to discover interactive elements Use agent browser click @eN , agent browser fill @eN "text" , etc. to interact Use agent browser wait load networkidle or agent browser wait 1500 between actions Take a screenshot after every significant state change — never go more than 2 meaningful actions without a screenshot: Name screenshots descriptively: step 02 form filled.png , step 03 submit clicked.png , step 04 success state.png Re snapshot after navigation or DOM changes (refs go stale) If elements are hard to find by ref, use semantic locators: Mark each todo as complete or failed based on outcome Mandatory coverage checklist (execute ALL that apply to the coverage level chosen): Happy path (always required): [ ] Main feature flow works end to end as expected [ ] Success state / confirmation is visible [ ] Data is persisted / reflected correctly after action Non happy paths (required for standard + comprehensive): [ ] Empty / missing required inputs — form validation fires, error messages shown [ ] Invalid input values — correct rejection, no crash [ ] Boundary values — minimum and maximum accepted values [ ] Duplicate / conflicting data (if applicable) [ ] Unauthorized / permission denied state (if applicable) [ ] Empty list / zero state view (if applicable) [ ] Rapid repeated actions (double click submit, spam button) [ ] Long strings / special characters in text inputs Error states (required for standard + comprehensive): [ ] API / network failure behavior (if simulatable) [ ] Partial failure — what happens if only part of the action succeeds [ ] Graceful degradation — app does not crash, user sees a meaningful message Visual & navigation (required for comprehensive): [ ] Layout is correct, no overflow or broken UI [ ] Responsive on narrower viewport (resize if possible) [ ] Links and navigation work; back button does not break state [ ] Loading states / spinners shown while async work is in progress 5. Stop recording and clean up Keep the recording under 2 minutes. If the feature requires more exploration, split into multiple recordings (e.g. happy path.webm , error states.webm ). 6. Fix WebM duration metadata (MANDATORY if ffmpeg is available) WebM files recorded by agent browser often have duration = Infinity in the container header — the video player then shows 0:00. Fix every .webm file by remuxing it through ffmpeg, which reads the entire file, computes the real duration, and writes it into the output header: If ffmpeg is not available, skip this step and continue — the video will still play, it just won't show the correct duration in the player. Phase 3: Upload Captures to S3 For each captured file, use the mcp alan get upload url MCP tool to get a presigned S3 URL, then curl PUT the file directly to S3. Per file upload flow 1. Get the file size (needed by the MCP tool): 2. Call the MCP tool to get a presigned upload URL: Returns: { "uploadUrl": "https://...", "s3Key": "sandbox captures/..." } 3. Upload the file to S3 using the presigned URL: 4. Save the s3Key — you will pass it to create test report in Phase 4. Repeat for each screenshot and video file. Common MIME types: Screenshots: image/png Video recordings: video/webm If uploads fail : Continue to Phase 4 anyway — omit the screenshotUrl , videoUrl , and screenshotUrls fields. The structured report is still valuable without media. Phase 4: Create Structured Test Report (MANDATORY — NEVER SKIP) CRITICAL: You MUST call mcp alan create test report to complete this skill. This is non negotiable. Do NOT output the report as markdown text Do NOT summarize findings in chat only Do NOT skip this phase for any reason — not because of time, not because uploads failed, not because testing was partial The report MUST be persisted via this MCP tool call so the UI renders it as an interactive artifact Skipping this step means the entire test session is wasted and untracked If uploads failed in Phase 3, create the report anyway — omit the media URLs but include all steps, issues, and summary text. Call the mcp alan create test report MCP tool with structured data from your testing. Gather all the data from the previous phases and call the tool: Field guidelines status : "pass" if all steps passed, "fail" if an

alan-test-feature