Initial commit with translated description

This commit is contained in:
2026-03-29 09:39:00 +08:00
commit 952514b082
4 changed files with 138 additions and 0 deletions

61
SKILL.md Normal file
View File

@@ -0,0 +1,61 @@
---
name: linux-gui-control
description: "使用xdotool、wmctrl和dogtail控制Linux桌面GUI。在需要与非浏览器应用程序交互、模拟鼠标/键盘输入、管理窗口或检查X11/GNOME上应用程序的UI层次结构时使用。支持1在应用中点击/输入2调整大小/移动窗口3从应用中提取基于文本的UI树A11y4截取屏幕截图进行视觉分析。"
---
# Linux GUI Control
This skill provides tools and procedures for automating interactions with the Linux desktop environment.
## Quick Start
### 1. Identify Target Window
Use `wmctrl` to find the exact name of the window you want to control.
```bash
wmctrl -l
```
### 2. Inspect UI Hierarchy
For apps supporting accessibility (GNOME apps, Electron apps with `--force-renderer-accessibility`), use the inspection script to find button names without taking screenshots.
```bash
python3 scripts/inspect_ui.py "<app_name>"
```
### 3. Perform Actions
Use `xdotool` via the helper script for common actions.
```bash
# Activate window
./scripts/gui_action.sh activate "<window_name>"
# Click coordinates
./scripts/gui_action.sh click 500 500
# Type text
./scripts/gui_action.sh type "Hello World"
# Press a key
./scripts/gui_action.sh key "Return"
```
## Workflows
### Operating an App via Text UI
1. List windows with `wmctrl -l`.
2. Activate the target window.
3. Run `scripts/inspect_ui.py` to get the list of buttons and inputs.
4. Use `xdotool key Tab` and `Return` to navigate, or `click` if coordinates are known.
5. If text-based inspection fails, fallback to taking a screenshot and using vision.
### Forcing Accessibility in Electron Apps
Many modern apps (VS Code, Discord, Cider, Chrome) need a flag to expose their UI tree:
```bash
pkill <app>
nohup <app> --force-renderer-accessibility > /dev/null 2>&1 &
```
## Tool Reference
- **wmctrl**: Window management (list, activate, move, resize).
- **xdotool**: Input simulation (click, type, key, mousemove).
- **dogtail**: UI tree extraction via AT-SPI (Accessibility bus).
- **scrot**: Lightweight screenshot tool.