Desktop Control Skill This skill provides comprehensive desktop automation capabilities through PyAutoGUI, allowing AI agents to control the mouse, keyboard, take screenshots, and interact with the desktop environment. How to Use This Skill As an AI agent, you can invoke desktop automation commands using the uvx desktop agent CLI. Command Structure All commands follow this pattern: Categories: mouse Mouse control keyboard Keyboard input screen Screenshots and screen analysis message User dialogs app Application control (open, focus, list windows) Available Commands 🖱️ Mouse Control ( mouse ) Control cursor movement and clicks. Examples: ⌨️ Keyboard Control ( keyboard ) Type text and execute keyboard shortcuts. Examples: Common Key Names: Modifiers: ctrl , shift , alt , win Special: enter , tab , esc , space , backspace , delete Function: f1 through f12 Arrows: up , down , left , right 🖼️ Screen & Screenshots ( screen ) Capture screenshots and analyze screen content. Supports targeting specific windows. Examples: 💬 Message Dialogs ( message ) Display user interaction dialogs. Examples: 📱 Application Control ( app ) Control applications across Windows, macOS, and Linux. Examples: Common Automation Workflows Workflow 1: Open Application and Type Workflow 2: Screenshot + Analysis Workflow 3: Form Filling Workflow 4: Copy/Paste Operations Safety Considerations When using this skill, AI agents should: 1. Verify coordinates : Use screen size and on screen before clicking 2. Add delays : Insert appropriate delays between commands for UI responsiveness 3. Validate images : Ensure image files exist before using locate commands 4. Handle failures : Commands may fail if windows change or elements move 5. User safety : Always confirm destructive actions with user via message confirm Troubleshooting PyAutoGUI Fail Safe PyAutoGUI has a fail safe: moving mouse to screen corner aborts operations. This is a safety feature. Image not found When using screen locate , ensure: Image file exists and path is correct Adjust confidence (try 0.7 0.9) Image matches exact screen appearance (resolution, colors) Getting Help Integration Tips for AI Agents 1. Always check screen size first when working with absolute coordinates 2. Use relative positioning when possible (e.g., get current position, calculate offset) 3. Combine commands for complex workflows 4. Validate before executing (e.g., check if image exists on screen) 5. Provide user feedback using message dialogs for important operations 6. Handle errors gracefully commands may fail if UI state changes Performance Notes Mouse movements with duration are animated and take time Image location ( locate ) can be slow on large screens use regions when possible Keyboard commands are generally fast (< 100ms) Screenshots depend on screen resolution and region size Output Format All commands output structured JSON by default, ideal for programmatic use by AI agents: Response Schema All JSON responses follow this schema: Error Response Schema Error Codes Code Description success Command succeeded invalid argument Invalid command arguments coordinates out of bounds Coordinates outside screen image not found Image file not found or not on screen window not found Target window not found ocr failed OCR operation failed application not found Application not found permission denied Permission denied platform not supported Platform not supported timeout Operation timed out unknown error Unknown error Mouse move: Screen size: Locate image: List windows: Error example: Effective Usage Guide for AI Agents This section teaches AI agents how to use this skill effectively with optimal command sequences and best practices. 🎯 Core Strategy: Observe First, Then Act Always understand the current state before performing actions. This avoids clicking wrong coordinates or typing in the wrong window. Recommended Initial Sequence: 📋 Recommended Command Sequences by Task Open and Interact with Application Find and Click UI Element (Image Based) Find and Click UI Element (Text Based with OCR) Fill a Form with Multiple Fields Take Targeted Screenshots for Analysis Safe Drag and Drop 🔄 Error Recovery Patterns When Window Not Found When Image Not Found When Click Seems to Miss ⚡ Performance Optimization Minimize Screenshots Batch Keyboard Input Use Hotkeys Over Mouse When Possible 🛡️ Defensive Programming Patterns Always Verify Critical Actions Use JSON Mode for Reliable Parsing Validate Before Multi Step Operations 🎮 Platform Specific Considerations Windows macOS Linux 📊 Decision Tree: Choosing the Right Command Integration Tips for AI Agents 1. Always check screen size first when working with absolute coordinates 2. Use relative positioning when possible (e.g., get current position, calculate offset) 3. Combine commands for complex workflows 4. Validate before executing (e.g., check if image exists on screen) 5. Provide user feedback using message dialogs for important operations 6. Handle errors gracefully commands may fail if UI state changes