Vision-driven desktop automation using Midscene. Control your local desktop (macOS, Windows, Linux) or a remote Windows desktop over RDP with natural language commands. Operates entirely from screenshots — no DOM or accessibility labels required. Can interact with all visible elements on screen rega

Desktop Computer Automation CRITICAL RULES — VIOLATIONS WILL BREAK THE WORKFLOW: 1. Never run midscene commands in the background. Each command must run synchronously so you can read its output (especially screenshots) before deciding the next action. Background execution breaks the screenshot analyze act loop. 2. Run only one midscene command at a time. Wait for the previous command to finish, read the screenshot, then decide the next action. Never chain multiple commands together. 3. Allow enough time for each command to complete. Midscene commands involve AI inference and screen interaction, which can take longer than typical shell commands. A typical command needs about 1 minute; complex act commands may need even longer. 4. Always report task results before finishing. After completing the automation task, you MUST proactively summarize the results to the user — including key data found, actions completed, screenshots taken, and any relevant findings. Never silently end after the last automation step; the user expects a complete response in a single interaction. 5. Only minimize windows, never close them unless explicitly asked. When you need to dismiss or get a window out of the way, minimize it instead of closing it. Do not close any app or window unless the user explicitly asks you to do so. Control your desktop (macOS, Windows, Linux) using npx y @midscene/computer@1 . Each CLI command maps directly to an MCP tool — you (the AI agent) act as the brain, deciding which actions to take based on screenshots. What act Can Do Inside a single act call on desktop, Midscene can move the mouse, click, double click, right click, drag items, type or clear text, scroll, press single keys or keyboard shortcuts, and work through multi step interactions on whatever is visible on the selected display. Prerequisites Midscene requires models with strong visual grounding capabilities. The following environment variables must be configured — either as system environment variables or in a .env file in the current working directory (Midscene loads .env automatically): Example: Gemini (Gemini 3 Flash) Example: Qwen 3.5 Example: Doubao Seed 2.0 Lite Commonly used models: Doubao Seed 2.0 Lite, Qwen 3.5, Zhipu GLM 4.6V, Gemini 3 Pro, Gemini 3 Flash. If the model is not configured, ask the user to set it up. See [Model Configuration](https://midscenejs.com/model common config) for supported providers. Commands Connect to Desktop Connect via RDP Use RDP mode to drive a remote Windows desktop instead of the local machine. Providing host switches connect to RDP and routes every subsequent command ( act , tap , take screenshot , assert , disconnect ) through the RDP helper binary bundled with @midscene/computer . The local mouse/keyboard is not touched. Minimum example: All RDP options for connect (RDP mode is activated when host is set; the other flags are optional): host <fqdn or ip — RDP host. Required to enter RDP mode. port <number — RDP port (default 3389 ). username <user — RDP user account. password <secret — RDP password. Prefer reading from an environment variable, secrets manager, or interactive prompt; never paste it into a shared transcript. domain <domain — Active Directory / NTLM domain. security protocol <auto tls nla rdp — Security protocol negotiation. Defaults to auto . ignore certificate — Skip TLS certificate validation. Use only for trusted dev hosts with self signed certs. admin session — Attach to the admin/console session (equivalent to mstsc /admin ). desktop width <px and desktop height <px — Request a specific remote desktop resolution. The actual size is whatever the RDP server negotiates back; e.g. requesting 1024x768 against a host that pins 1280x720 will land on 1280x720 . Confirm the negotiated size with listdisplays host ... username ... password ... after connect. Notes specific to RDP mode: displayId and headless are ignored in RDP mode. A connected RDP session always exposes a single virtual display whose size is whatever the server negotiated. Two display listing commands exist and behave differently — pick the right one: list displays (with underscore, platform tool) — enumerates local physical displays only. Does not accept RDP flags. Useless after an RDP connect. listdisplays (no underscore, action tool) — accepts the same RDP flags as connect / take screenshot /etc. In RDP mode it returns the negotiated virtual display, e.g. [{ "id": "...", "name": "RDP 10.70.86.26:3389 (1280x720)", "primary": true }] . Use this to verify the actual resolution. The RDP transport uses a native helper binary shipped inside @midscene/computer . If you see RDP helper binary not found errors, the optional bin/<platform /rdp helper was stripped from your install — reinstall the package or unpack a fresh tarball. Treat RDP credentials as secrets: do not commit .env files containing password to the repo; prefer export RDP PASSWORD=... in the current shell and reference it as password "$RDP PASSWORD" . Latency expectations : every CLI invocation is a fresh node process, so each command re establishes the RDP session. Budget roughly: connect / take screenshot / keyboardpress / scroll : ~5 s (node startup + RDP TLS+NLA handshake + first frame). act / assert / tap locate : ~5 s + AI inference + any planned sub actions; expect 8–20 s end to end for typical interactions. The RDP handshake itself is ~700 ms; the rest is unavoidable cold start cost in the CLI shape. Connect failure diagnostics : when connect fails, the first line of stderr is the actionable error (e.g. connect failed: Failed to connect to RDP server: ERRCONNECT LOGON FAILURE: Logon failed. ). The subsequent stack trace is diagnostic noise — read the first line, then check credentials/network. Common ERRCONNECT causes: LOGON FAILURE — bad username/password/domain. CONNECT TRANSPORT FAILED — host unreachable or RDP port blocked. Verify with nc zv <host 3389 . TLS CONNECT FAILED — TLS handshake rejected. Try ignore certificate for self signed dev hosts, or pin security protocol nla . After connect host ... succeeds, the rest of the workflow ( act , tap locate , assert , take screenshot , listdisplays , report tool , disconnect ) is identical to local mode — just remember to pass the same host / username / password / ignore certificate flags to every subsequent command, since each CLI invocation is stateless and reconnects. List Displays Take Screenshot After taking a screenshot, read the saved image file to understand the current screen state before deciding the next action. Perform Action Use act to interact with the computer and get the result. It autonomously handles all UI interactions internally — clicking, typing, scrolling, waiting, and navigating — so you should give it complex, high level tasks as a whole rather than breaking them into small steps. Describe what you want to do and the desired effect in natural language: Assert Current Screen State Use assert to verify that the current screen satisfies a natural language condition. It does not perform UI actions; it checks the visible screen state and passes only when the assertion is true. Use this for validation, QA checks, and final state verification after act . By default a failed assertion throws an AI generated reason. Pass message to throw a custom error message instead, which is useful for surfacing the intended outcome in QA and CI logs. When the assertion needs to compare against a reference image (icon, logo, screenshot), pass image for the URL/path and image name for its display name. Each image may be an http(s) link, a data: URI, or a local file path. Repeat both flags in matching order when you need to attach more than one image. Add convertHttpImage2Base64 true when the model cannot reach the URL directly. Requires @midscene/computer@1.9.0+ . Record and Assert Transient UI Use a recording when the state to verify may disappear before a current screen assertion runs, such as a toast, loading banner, animation, or transition: Pass local display or RDP target flags and output to record start , then wait for Recording. Press Ctrl+C to stop and save. Keep the recorder as a foreground process in its dedicated terminal; never add shell & . Perform the interaction manually or from a second terminal, send Ctrl+C to the recorder, and wait until it prints the saved path before asserting. Repeat target flags on actions and the final assert . Optional capture flags are interval ms , max frames , and watchdog ms ; max frames caps sampled frames, and the manifest may contain one additional final representative frame. The default watchdog finalizes and saves the recording after five minutes, while watchdog ms 0 disables that safety limit. The output is a JSON manifest plus an adjacent <name .frames image directory, not an encoded video or archive. The manifest contains relative JPEG/PNG paths and no base64 image bodies. Keep or move the JSON file and image directory together, and pass the JSON path to assert record . Use ordinary assert without record when only the current screen matters. Use a Reference Image for Precise Targeting When the user provides a screenshot, icon, logo, or reference image and wants an exact visual match, prefer tap locate instead of a generic act prompt . Pass locate as JSON. The prompt describes the target, images supplies named reference images, and convertHttpImage2Base64: true is useful when the image URL may not be directly accessible to the model. The same locate JSON shape also works for other commands that accept a locate parameter. Disconnect Consume Report Files The generated HTML report is recommended for human reading first. It includes step by step execution details and replay videos for each operation, which makes it much easier to understand what happened and troubleshoot problems. If another skill or tool needs to consume the report, first convert it with report tool from the same platform CLI package. Prefer Markdown for LLM based workflows. Use JSON when the report needs to be processed programmatically. Workflow Pattern Since CLI commands are stateless between invocations, follow this pattern: 1. Connect to establish a session 2. Health check — observe the output of the connect command. If connect already performed a health check (screenshot and mouse movement test), no additional check is needed. If connect did not perform a health check, do one manually: take a screenshot and verify it succeeds, then move the mouse to a random position ( act prompt "move the mouse to a random position" ) and verify it succeeds. If either step fails, stop and troubleshoot before continuing. Only proceed to the next steps after both checks pass without errors. 3. Launch the target app and take screenshot to see the current state, make sure the app is launched and visible on the screen. 4. Execute action using act to perform the desired action or target driven instructions. Use assert for the resulting screen state, or keep record start output ... running in a dedicated terminal during transient state workflows, stop it with Ctrl+C, and then use assert record . 5. Disconnect when done 6. Report results — summarize what was accomplished, present key findings and data extracted during the task, and list any generated files (screenshots, logs, etc.) with their paths Best Practices 1. Always run a health check first : After connecting, observe the output of the connect command. If connect already performed a health check (screenshot and mouse movement test), no additional check is needed. If it did not, d

computer-automation