62 lines
2.1 KiB
Markdown
62 lines
2.1 KiB
Markdown
|
---
|
|||
|
|
name: linux-gui-control
|
|||
|
|
description: "使用xdotool、wmctrl和dogtail控制Linux桌面GUI。在需要与非浏览器应用程序交互、模拟鼠标/键盘输入、管理窗口或检查X11/GNOME上应用程序的UI层次结构时使用。支持：（1）在应用中点击/输入，（2）调整大小/移动窗口，（3）从应用中提取基于文本的UI树（A11y），（4）截取屏幕截图进行视觉分析。"
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Linux GUI Control
|
|||
|
|
|
|||
|
|
This skill provides tools and procedures for automating interactions with the Linux desktop environment.
|
|||
|
|
|
|||
|
|
## Quick Start
|
|||
|
|
|
|||
|
|
### 1. Identify Target Window
|
|||
|
|
Use `wmctrl` to find the exact name of the window you want to control.
|
|||
|
|
```bash
|
|||
|
|
wmctrl -l
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 2. Inspect UI Hierarchy
|
|||
|
|
For apps supporting accessibility (GNOME apps, Electron apps with `--force-renderer-accessibility`), use the inspection script to find button names without taking screenshots.
|
|||
|
|
```bash
|
|||
|
|
python3 scripts/inspect_ui.py "<app_name>"
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 3. Perform Actions
|
|||
|
|
Use `xdotool` via the helper script for common actions.
|
|||
|
|
```bash
|
|||
|
|
# Activate window
|
|||
|
|
./scripts/gui_action.sh activate "<window_name>"
|
|||
|
|
|
|||
|
|
# Click coordinates
|
|||
|
|
./scripts/gui_action.sh click 500 500
|
|||
|
|
|
|||
|
|
# Type text
|
|||
|
|
./scripts/gui_action.sh type "Hello World"
|
|||
|
|
|
|||
|
|
# Press a key
|
|||
|
|
./scripts/gui_action.sh key "Return"
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Workflows
|
|||
|
|
|
|||
|
|
### Operating an App via Text UI
|
|||
|
|
1. List windows with `wmctrl -l`.
|
|||
|
|
2. Activate the target window.
|
|||
|
|
3. Run `scripts/inspect_ui.py` to get the list of buttons and inputs.
|
|||
|
|
4. Use `xdotool key Tab` and `Return` to navigate, or `click` if coordinates are known.
|
|||
|
|
5. If text-based inspection fails, fallback to taking a screenshot and using vision.
|
|||
|
|
|
|||
|
|
### Forcing Accessibility in Electron Apps
|
|||
|
|
Many modern apps (VS Code, Discord, Cider, Chrome) need a flag to expose their UI tree:
|
|||
|
|
```bash
|
|||
|
|
pkill <app>
|
|||
|
|
nohup <app> --force-renderer-accessibility > /dev/null 2>&1 &
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Tool Reference
|
|||
|
|
|
|||
|
|
- **wmctrl**: Window management (list, activate, move, resize).
|
|||
|
|
- **xdotool**: Input simulation (click, type, key, mousemove).
|
|||
|
|
- **dogtail**: UI tree extraction via AT-SPI (Accessibility bus).
|
|||
|
|
- **scrot**: Lightweight screenshot tool.
|