mirror of
https://github.com/actionbook/actionbook.git
synced 2026-09-19 03:28:06 +08:00
AlexZ
a35ba49827
- P2-1: `app attach [port]` → `app attach <target>` (required argument) - P2-2: `--port` → `--cdp` (correct global flag for CDP port) - P2-3: Remove invalid custom-path launch fallback; show manual start + attach workflow instead Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
447 lines
11 KiB
Plaintext
447 lines
11 KiB
Plaintext
---
|
|
title: "Electron App Automation"
|
|
description: "Automate desktop applications with actionbook app"
|
|
---
|
|
|
|
# Electron App Automation
|
|
|
|
The `actionbook app` command enables AI agents to automate any Electron-based desktop application using Chrome DevTools Protocol. This includes popular apps like VS Code, Slack, Discord, Figma, Notion, and Spotify.
|
|
|
|
<Note>
|
|
All browser automation features work with desktop apps — Shadow DOM, iframes, hotkeys, tab management, and more.
|
|
</Note>
|
|
|
|
## Why Desktop App Automation?
|
|
|
|
Many workflows involve desktop applications:
|
|
|
|
- **Development**: Automate VS Code to run tests, format code, or navigate files
|
|
- **Communication**: Send Slack messages, manage Discord channels
|
|
- **Design**: Export Figma designs, take screenshots of designs
|
|
- **Productivity**: Create Notion pages, organize notes
|
|
- **Music**: Control Spotify playback, manage playlists
|
|
|
|
Traditional approaches require:
|
|
- Platform-specific automation (AppleScript, PowerShell, xdotool)
|
|
- Screen coordinate-based clicking (brittle, display-dependent)
|
|
- Accessibility APIs (limited, complex)
|
|
|
|
**Actionbook uses Chrome DevTools Protocol** — the same protocol that powers browser automation. Since Electron apps are built on Chromium, they expose the same CDP interface.
|
|
|
|
## How It Works
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ Actionbook App │
|
|
│ ┌────────────┐ ┌────────────┐ │
|
|
│ │ discover │ → │ launch │ (with CDP) │
|
|
│ │ apps │ │ app with │ │
|
|
│ │ │ │ --remote │ │
|
|
│ │ │ │ -debugging│ │
|
|
│ │ │ │ -port │ │
|
|
│ └────────────┘ └─────┬──────┘ │
|
|
│ │ │
|
|
│ ▼ │
|
|
│ ┌────────────────┐ │
|
|
│ │ CDP WebSocket │ │
|
|
│ │ Connection │ │
|
|
│ └───────┬────────┘ │
|
|
└──────────────────────────────┼──────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────┐
|
|
│ Electron App │
|
|
│ (VS Code, │
|
|
│ Slack, etc.) │
|
|
└─────────────────┘
|
|
```
|
|
|
|
**Key Steps:**
|
|
|
|
1. **Auto-Discovery**: Scans common install locations for Electron apps
|
|
2. **CDP Launch**: Starts app with `--remote-debugging-port=9222`
|
|
3. **Session Tracking**: Stores app path for restart preservation
|
|
4. **Full Feature Parity**: All browser automation features work
|
|
|
|
## Quick Start
|
|
|
|
### 1. Launch an App
|
|
|
|
```bash
|
|
# Auto-discover and launch VS Code
|
|
actionbook app launch "Visual Studio Code"
|
|
|
|
# Launch Slack
|
|
actionbook app launch Slack
|
|
|
|
# Launch Discord
|
|
actionbook app launch Discord
|
|
```
|
|
|
|
The CLI will:
|
|
- Find the app's installation path
|
|
- Launch it with CDP enabled on port 9222
|
|
- Store the session for future commands
|
|
|
|
### 2. Run Automation Commands
|
|
|
|
Once launched, all browser commands work with the `app` prefix:
|
|
|
|
```bash
|
|
# Click an element
|
|
actionbook app click "button[aria-label='New File']"
|
|
|
|
# Type text
|
|
actionbook app type "Hello World" "div[contenteditable='true']"
|
|
|
|
# Send keyboard shortcuts
|
|
actionbook app hotkey "Cmd+Shift+P" # macOS
|
|
actionbook app hotkey "Ctrl+Shift+P" # Windows/Linux
|
|
|
|
# Take a screenshot
|
|
actionbook app screenshot vscode-window.png
|
|
|
|
# Get page snapshot
|
|
actionbook app snapshot --format compact
|
|
```
|
|
|
|
### 3. Close the App
|
|
|
|
```bash
|
|
actionbook app close
|
|
```
|
|
|
|
## Example Workflows
|
|
|
|
### VS Code: Open File and Format
|
|
|
|
```bash
|
|
# Launch VS Code
|
|
actionbook app launch "Visual Studio Code"
|
|
|
|
# Open command palette
|
|
actionbook app hotkey "Cmd+Shift+P"
|
|
|
|
# Type and select command
|
|
actionbook app type "Format Document"
|
|
actionbook app hotkey "Enter"
|
|
|
|
# Wait a moment
|
|
sleep 1
|
|
|
|
# Save
|
|
actionbook app hotkey "Cmd+S"
|
|
```
|
|
|
|
### Slack: Send Message to Channel
|
|
|
|
```bash
|
|
# Launch Slack
|
|
actionbook app launch Slack
|
|
|
|
# Wait for app to load
|
|
sleep 2
|
|
|
|
# Click on a channel
|
|
actionbook app click "button[data-qa='channel_sidebar_name_general']"
|
|
|
|
# Type message in the message box
|
|
actionbook app type "Hello team! 👋" "div[role='textbox']"
|
|
|
|
# Send message
|
|
actionbook app hotkey "Enter"
|
|
```
|
|
|
|
### Figma: Export Design
|
|
|
|
```bash
|
|
# Attach to already-running Figma (if started with --remote-debugging-port=9222)
|
|
actionbook app attach 9222
|
|
|
|
# Click on a frame
|
|
actionbook app click "[data-test-id='canvas-frame']"
|
|
|
|
# Open export menu
|
|
actionbook app hotkey "Cmd+Shift+E"
|
|
|
|
# Take screenshot of export panel
|
|
actionbook app screenshot figma-export-panel.png
|
|
```
|
|
|
|
### Notion: Create New Page
|
|
|
|
```bash
|
|
# Launch Notion
|
|
actionbook app launch Notion
|
|
|
|
# Wait for load
|
|
sleep 2
|
|
|
|
# Click "New Page" button
|
|
actionbook app click "div[role='button']:has-text('New page')"
|
|
|
|
# Wait for page to appear
|
|
sleep 1
|
|
|
|
# Type page title
|
|
actionbook app type "Weekly Meeting Notes"
|
|
|
|
# Add some content
|
|
actionbook app hotkey "Enter"
|
|
actionbook app type "## Agenda"
|
|
actionbook app hotkey "Enter"
|
|
actionbook app type "- Topic 1"
|
|
```
|
|
|
|
## Advanced Features
|
|
|
|
### Shadow DOM Support
|
|
|
|
Many Electron apps use Web Components with Shadow DOM:
|
|
|
|
```bash
|
|
# Click a button inside a shadow root
|
|
actionbook app click "my-component::shadow-root > button.submit"
|
|
|
|
# Type in a shadow DOM input
|
|
actionbook app type "text" "custom-input::shadow-root > input"
|
|
```
|
|
|
|
### IFrame Context Switching
|
|
|
|
Some apps embed iframes:
|
|
|
|
```bash
|
|
# Switch to iframe
|
|
actionbook app switch-frame "iframe#content"
|
|
|
|
# Interact with iframe content
|
|
actionbook app click "button.action"
|
|
|
|
# Switch back to main frame
|
|
actionbook app switch-frame default
|
|
```
|
|
|
|
### Multi-Modifier Hotkeys
|
|
|
|
```bash
|
|
# Control+Shift+P (Windows/Linux)
|
|
actionbook app hotkey "Control+Shift+P"
|
|
|
|
# Cmd+Shift+P (macOS)
|
|
actionbook app hotkey "Meta+Shift+P"
|
|
|
|
# Control+Alt+Delete (Windows)
|
|
actionbook app hotkey "Control+Alt+Delete"
|
|
```
|
|
|
|
### Tab Management
|
|
|
|
Some apps like VS Code have multiple internal tabs:
|
|
|
|
```bash
|
|
# List all pages/tabs
|
|
actionbook app tab list
|
|
|
|
# Switch to a specific tab
|
|
actionbook app tab switch <PAGE_ID>
|
|
|
|
# Create new tab (if supported by app)
|
|
actionbook app tab new
|
|
```
|
|
|
|
### Scroll with Wait
|
|
|
|
For apps with lazy-loaded content:
|
|
|
|
```bash
|
|
# Scroll down and wait for scrollend event
|
|
actionbook app scroll down 500 --wait
|
|
|
|
# Scroll to bottom
|
|
actionbook app scroll bottom --wait
|
|
```
|
|
|
|
## Attaching to Running Apps
|
|
|
|
If an Electron app is already running with CDP enabled, you can attach to it:
|
|
|
|
```bash
|
|
# Attach to port 9222
|
|
actionbook app attach 9222
|
|
|
|
# Check connection status
|
|
actionbook app status
|
|
|
|
# Now run commands
|
|
actionbook app snapshot
|
|
```
|
|
|
|
**Starting Apps Manually with CDP:**
|
|
|
|
```bash
|
|
# VS Code
|
|
code --remote-debugging-port=9222
|
|
|
|
# Slack (macOS example)
|
|
open -a Slack --args --remote-debugging-port=9222
|
|
|
|
# Any Electron app
|
|
/path/to/app --remote-debugging-port=9222
|
|
```
|
|
|
|
## App Discovery
|
|
|
|
List all discoverable Electron apps on your system:
|
|
|
|
```bash
|
|
actionbook app list
|
|
```
|
|
|
|
**Output Example:**
|
|
|
|
```json
|
|
[
|
|
{
|
|
"name": "Visual Studio Code",
|
|
"path": "/Applications/Visual Studio Code.app/Contents/MacOS/Electron",
|
|
"platform": "macos"
|
|
},
|
|
{
|
|
"name": "Slack",
|
|
"path": "/Applications/Slack.app/Contents/MacOS/Slack",
|
|
"platform": "macos"
|
|
},
|
|
{
|
|
"name": "Discord",
|
|
"path": "/Applications/Discord.app/Contents/MacOS/Discord",
|
|
"platform": "macos"
|
|
}
|
|
]
|
|
```
|
|
|
|
## Restart Preservation
|
|
|
|
The `restart` command preserves the session:
|
|
|
|
```bash
|
|
# Launch app
|
|
actionbook app launch "Visual Studio Code"
|
|
|
|
# Do some work...
|
|
actionbook app click "button"
|
|
|
|
# Restart (keeps session, re-launches same app)
|
|
actionbook app restart
|
|
```
|
|
|
|
This is useful for:
|
|
- Recovering from app crashes
|
|
- Testing app restart behavior
|
|
- Refreshing app state
|
|
|
|
## Platform-Specific Notes
|
|
|
|
### macOS
|
|
|
|
Apps are typically in `/Applications/*.app/Contents/MacOS/`:
|
|
|
|
```bash
|
|
# Standard locations
|
|
/Applications/Visual Studio Code.app/Contents/MacOS/Electron
|
|
/Applications/Slack.app/Contents/MacOS/Slack
|
|
```
|
|
|
|
### Linux
|
|
|
|
Apps are in various locations:
|
|
|
|
```bash
|
|
# Snap apps
|
|
/snap/code/current/usr/share/code/code
|
|
/snap/slack/current/usr/lib/slack/slack
|
|
|
|
# System-wide apps
|
|
/usr/share/code/code
|
|
/usr/bin/slack
|
|
```
|
|
|
|
### Windows
|
|
|
|
Apps are typically in `Program Files`:
|
|
|
|
```bash
|
|
# Common paths
|
|
C:\Program Files\Microsoft VS Code\Code.exe
|
|
C:\Program Files\Slack\slack.exe
|
|
C:\Users\<username>\AppData\Local\Discord\app-*\Discord.exe
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### App Not Found
|
|
|
|
```bash
|
|
actionbook app list
|
|
# Check if app is in the list
|
|
# If not listed, start the app manually with CDP enabled,
|
|
# then attach by port:
|
|
/path/to/app --remote-debugging-port=9222
|
|
actionbook app attach 9222
|
|
```
|
|
|
|
### Port Already in Use
|
|
|
|
```bash
|
|
# Use a different CDP port via the global --cdp flag
|
|
actionbook app launch "VS Code" --cdp 9223
|
|
```
|
|
|
|
### CDP Not Available
|
|
|
|
Some apps disable CDP by default. Check app documentation or try:
|
|
|
|
```bash
|
|
# Start app manually with CDP flag
|
|
/path/to/app --remote-debugging-port=9222
|
|
|
|
# Then attach
|
|
actionbook app attach 9222
|
|
```
|
|
|
|
### App Crashes on Launch
|
|
|
|
Some apps don't support CDP flags. Try:
|
|
|
|
1. Start app normally first
|
|
2. Check if app has a "Developer Mode" or "Debug Mode"
|
|
3. Consult app-specific documentation
|
|
|
|
## Comparison with Browser Automation
|
|
|
|
| Feature | `actionbook browser` | `actionbook app` |
|
|
|---------|---------------------|-----------------|
|
|
| **Target** | Web browsers | Desktop Electron apps |
|
|
| **Launch** | `browser open <URL>` | `app launch <NAME>` |
|
|
| **Discovery** | Browser paths | App install paths |
|
|
| **Commands** | 35+ commands | Same 35+ commands |
|
|
| **Shadow DOM** | ✅ | ✅ |
|
|
| **IFrames** | ✅ | ✅ |
|
|
| **Hotkeys** | ✅ | ✅ |
|
|
| **Tab Management** | ✅ | ✅ |
|
|
| **Screenshots** | ✅ | ✅ |
|
|
| **Snapshots** | ✅ | ✅ |
|
|
|
|
## Security Considerations
|
|
|
|
- **CDP Access**: CDP provides full control over the app. Only use with trusted apps.
|
|
- **Port Exposure**: CDP port (9222) is localhost-only by default.
|
|
- **Session Storage**: App paths are stored in `~/.actionbook/sessions/`.
|
|
|
|
## Next Steps
|
|
|
|
- Explore the full [CLI Reference](/api-reference/cli#actionbook-app)
|
|
- Learn about [Browser Automation](/guides/browser)
|
|
- Check out [Keyboard Hotkeys](/guides/browser#keyboard-hotkeys)
|
|
- See [Shadow DOM Support](/guides/browser#shadow-dom)
|