Initial commit with translated description
This commit is contained in:
215
SKILL.md
Normal file
215
SKILL.md
Normal file
@@ -0,0 +1,215 @@
|
||||
---
|
||||
name: computer-use
|
||||
description: "用于无头Linux服务器的完整桌面计算机使用。"
|
||||
version: 1.2.1
|
||||
---
|
||||
|
||||
# Computer Use Skill
|
||||
|
||||
Full desktop GUI control for headless Linux servers. Creates a virtual display (Xvfb + XFCE) so you can run and control desktop applications on VPS/cloud instances without a physical monitor.
|
||||
|
||||
## Environment
|
||||
|
||||
- **Display**: `:99`
|
||||
- **Resolution**: 1024x768 (XGA, Anthropic recommended)
|
||||
- **Desktop**: XFCE4 (minimal — xfwm4 + panel only)
|
||||
|
||||
## Quick Setup
|
||||
|
||||
Run the setup script to install everything (systemd services, flicker-free VNC):
|
||||
|
||||
```bash
|
||||
./scripts/setup-vnc.sh
|
||||
```
|
||||
|
||||
This installs:
|
||||
- Xvfb virtual display on `:99`
|
||||
- Minimal XFCE desktop (xfwm4 + panel, no xfdesktop)
|
||||
- x11vnc with stability flags
|
||||
- noVNC for browser access
|
||||
|
||||
All services auto-start on boot and auto-restart on crash.
|
||||
|
||||
## Actions Reference
|
||||
|
||||
| Action | Script | Arguments | Description |
|
||||
|--------|--------|-----------|-------------|
|
||||
| screenshot | `screenshot.sh` | — | Capture screen → base64 PNG |
|
||||
| cursor_position | `cursor_position.sh` | — | Get current mouse X,Y |
|
||||
| mouse_move | `mouse_move.sh` | x y | Move mouse to coordinates |
|
||||
| left_click | `click.sh` | x y left | Left click at coordinates |
|
||||
| right_click | `click.sh` | x y right | Right click |
|
||||
| middle_click | `click.sh` | x y middle | Middle click |
|
||||
| double_click | `click.sh` | x y double | Double click |
|
||||
| triple_click | `click.sh` | x y triple | Triple click (select line) |
|
||||
| left_click_drag | `drag.sh` | x1 y1 x2 y2 | Drag from start to end |
|
||||
| left_mouse_down | `mouse_down.sh` | — | Press mouse button |
|
||||
| left_mouse_up | `mouse_up.sh` | — | Release mouse button |
|
||||
| type | `type_text.sh` | "text" | Type text (50 char chunks, 12ms delay) |
|
||||
| key | `key.sh` | "combo" | Press key (Return, ctrl+c, alt+F4) |
|
||||
| hold_key | `hold_key.sh` | "key" secs | Hold key for duration |
|
||||
| scroll | `scroll.sh` | dir amt [x y] | Scroll up/down/left/right |
|
||||
| wait | `wait.sh` | seconds | Wait then screenshot |
|
||||
| zoom | `zoom.sh` | x1 y1 x2 y2 | Cropped region screenshot |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
```bash
|
||||
export DISPLAY=:99
|
||||
|
||||
# Take screenshot
|
||||
./scripts/screenshot.sh
|
||||
|
||||
# Click at coordinates
|
||||
./scripts/click.sh 512 384 left
|
||||
|
||||
# Type text
|
||||
./scripts/type_text.sh "Hello world"
|
||||
|
||||
# Press key combo
|
||||
./scripts/key.sh "ctrl+s"
|
||||
|
||||
# Scroll down
|
||||
./scripts/scroll.sh down 5
|
||||
```
|
||||
|
||||
## Workflow Pattern
|
||||
|
||||
1. **Screenshot** — Always start by seeing the screen
|
||||
2. **Analyze** — Identify UI elements and coordinates
|
||||
3. **Act** — Click, type, scroll
|
||||
4. **Screenshot** — Verify result
|
||||
5. **Repeat**
|
||||
|
||||
## Tips
|
||||
|
||||
- Screen is 1024x768, origin (0,0) at top-left
|
||||
- Click to focus before typing in text fields
|
||||
- Use `ctrl+End` to jump to page bottom in browsers
|
||||
- Most actions auto-screenshot after 2 sec delay
|
||||
- Long text is chunked (50 chars) with 12ms keystroke delay
|
||||
|
||||
## Live Desktop Viewing (VNC)
|
||||
|
||||
Watch the desktop in real-time via browser or VNC client.
|
||||
|
||||
### Connect via Browser
|
||||
|
||||
```bash
|
||||
# SSH tunnel (run on your local machine)
|
||||
ssh -L 6080:localhost:6080 your-server
|
||||
|
||||
# Open in browser
|
||||
http://localhost:6080/vnc.html
|
||||
```
|
||||
|
||||
### Connect via VNC Client
|
||||
|
||||
```bash
|
||||
# SSH tunnel
|
||||
ssh -L 5900:localhost:5900 your-server
|
||||
|
||||
# Connect VNC client to localhost:5900
|
||||
```
|
||||
|
||||
### SSH Config (recommended)
|
||||
|
||||
Add to `~/.ssh/config` for automatic tunneling:
|
||||
|
||||
```
|
||||
Host your-server
|
||||
HostName your.server.ip
|
||||
User your-user
|
||||
LocalForward 6080 127.0.0.1:6080
|
||||
LocalForward 5900 127.0.0.1:5900
|
||||
```
|
||||
|
||||
Then just `ssh your-server` and VNC is available.
|
||||
|
||||
## System Services
|
||||
|
||||
```bash
|
||||
# Check status
|
||||
systemctl status xvfb xfce-minimal x11vnc novnc
|
||||
|
||||
# Restart if needed
|
||||
sudo systemctl restart xvfb xfce-minimal x11vnc novnc
|
||||
```
|
||||
|
||||
### Service Chain
|
||||
|
||||
```
|
||||
xvfb → xfce-minimal → x11vnc → novnc
|
||||
```
|
||||
|
||||
- **xvfb**: Virtual display :99 (1024x768x24)
|
||||
- **xfce-minimal**: Watchdog that runs xfwm4+panel, kills xfdesktop
|
||||
- **x11vnc**: VNC server with `-noxdamage` for stability
|
||||
- **novnc**: WebSocket proxy with heartbeat for connection stability
|
||||
|
||||
## Opening Applications
|
||||
|
||||
```bash
|
||||
export DISPLAY=:99
|
||||
|
||||
# Chrome — only use --no-sandbox if the kernel lacks user namespace support.
|
||||
# Check: cat /proc/sys/kernel/unprivileged_userns_clone
|
||||
# 1 = sandbox works, do NOT use --no-sandbox
|
||||
# 0 = sandbox fails, --no-sandbox required as fallback
|
||||
# Using --no-sandbox when unnecessary causes instability and crashes.
|
||||
if [ "$(cat /proc/sys/kernel/unprivileged_userns_clone 2>/dev/null)" = "0" ]; then
|
||||
google-chrome --no-sandbox &
|
||||
else
|
||||
google-chrome &
|
||||
fi
|
||||
|
||||
xfce4-terminal & # Terminal
|
||||
thunar & # File manager
|
||||
```
|
||||
|
||||
**Note**: Snap browsers (Firefox, Chromium) have sandbox issues on headless servers. Use Chrome `.deb` instead:
|
||||
|
||||
```bash
|
||||
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
|
||||
sudo dpkg -i google-chrome-stable_current_amd64.deb
|
||||
sudo apt-get install -f
|
||||
```
|
||||
|
||||
## Manual Setup
|
||||
|
||||
If you prefer manual setup instead of `setup-vnc.sh`:
|
||||
|
||||
```bash
|
||||
# Install packages
|
||||
sudo apt install -y xvfb xfce4 xfce4-terminal xdotool scrot imagemagick dbus-x11 x11vnc novnc websockify
|
||||
|
||||
# Run the setup script (generates systemd services, masks xfdesktop, starts everything)
|
||||
./scripts/setup-vnc.sh
|
||||
```
|
||||
|
||||
If you prefer fully manual setup, the `setup-vnc.sh` script generates all systemd service files inline -- read it for the exact service definitions.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### VNC shows black screen
|
||||
- Check if xfwm4 is running: `pgrep xfwm4`
|
||||
- Restart desktop: `sudo systemctl restart xfce-minimal`
|
||||
|
||||
### VNC flickering/flashing
|
||||
- Ensure xfdesktop is masked (check `/usr/bin/xfdesktop`)
|
||||
- xfdesktop causes flicker due to clear→draw cycles on Xvfb
|
||||
|
||||
### VNC disconnects frequently
|
||||
- Check noVNC has `--heartbeat 30` flag
|
||||
- Check x11vnc has `-noxdamage` flag
|
||||
|
||||
### x11vnc crashes (SIGSEGV)
|
||||
- Add `-noxdamage -noxfixes` flags
|
||||
- The DAMAGE extension causes crashes on Xvfb
|
||||
|
||||
## Requirements
|
||||
|
||||
Installed by `setup-vnc.sh`:
|
||||
```bash
|
||||
xvfb xfce4 xfce4-terminal xdotool scrot imagemagick dbus-x11 x11vnc novnc websockify
|
||||
```
|
||||
6
_meta.json
Normal file
6
_meta.json
Normal file
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"ownerId": "kn7cew7yks7cgeynqqjn8asxvx80axjx",
|
||||
"slug": "computer-use",
|
||||
"version": "1.2.1",
|
||||
"publishedAt": 1771195222594
|
||||
}
|
||||
46
scripts/click.sh
Normal file
46
scripts/click.sh
Normal file
@@ -0,0 +1,46 @@
|
||||
#!/bin/bash
|
||||
# click.sh - Click at coordinates
|
||||
# Usage: click.sh X Y [left|right|middle|double|triple]
|
||||
|
||||
export DISPLAY=:99
|
||||
|
||||
X=$1
|
||||
Y=$2
|
||||
BUTTON=${3:-left}
|
||||
|
||||
if [ -z "$X" ] || [ -z "$Y" ]; then
|
||||
echo "ERROR: Usage: click.sh X Y [left|right|middle|double|triple]" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Move to position first
|
||||
xdotool mousemove --sync "$X" "$Y"
|
||||
|
||||
# Click based on button type
|
||||
case "$BUTTON" in
|
||||
left)
|
||||
xdotool click 1
|
||||
;;
|
||||
right)
|
||||
xdotool click 3
|
||||
;;
|
||||
middle)
|
||||
xdotool click 2
|
||||
;;
|
||||
double)
|
||||
xdotool click --repeat 2 --delay 100 1
|
||||
;;
|
||||
triple)
|
||||
xdotool click --repeat 3 --delay 100 1
|
||||
;;
|
||||
*)
|
||||
echo "ERROR: Unknown button type: $BUTTON" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
echo "Clicked $BUTTON at $X,$Y"
|
||||
|
||||
# Auto-screenshot after action (2 sec delay)
|
||||
sleep 2
|
||||
exec "$(dirname "$0")/screenshot.sh"
|
||||
9
scripts/cursor_position.sh
Normal file
9
scripts/cursor_position.sh
Normal file
@@ -0,0 +1,9 @@
|
||||
#!/bin/bash
|
||||
# cursor_position.sh - Get current mouse coordinates
|
||||
|
||||
export DISPLAY=:99
|
||||
|
||||
# Get mouse location
|
||||
eval $(xdotool getmouselocation --shell 2>/dev/null)
|
||||
|
||||
echo "X=$X,Y=$Y"
|
||||
23
scripts/drag.sh
Normal file
23
scripts/drag.sh
Normal file
@@ -0,0 +1,23 @@
|
||||
#!/bin/bash
|
||||
# drag.sh - Drag from start to end coordinates
|
||||
# Usage: drag.sh X1 Y1 X2 Y2
|
||||
|
||||
export DISPLAY=:99
|
||||
|
||||
X1=$1
|
||||
Y1=$2
|
||||
X2=$3
|
||||
Y2=$4
|
||||
|
||||
if [ -z "$X1" ] || [ -z "$Y1" ] || [ -z "$X2" ] || [ -z "$Y2" ]; then
|
||||
echo "ERROR: Usage: drag.sh X1 Y1 X2 Y2" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
xdotool mousemove --sync "$X1" "$Y1" mousedown 1 mousemove --sync "$X2" "$Y2" mouseup 1
|
||||
|
||||
echo "Dragged from $X1,$Y1 to $X2,$Y2"
|
||||
|
||||
# Auto-screenshot after action
|
||||
sleep 2
|
||||
exec "$(dirname "$0")/screenshot.sh"
|
||||
29
scripts/hold_key.sh
Normal file
29
scripts/hold_key.sh
Normal file
@@ -0,0 +1,29 @@
|
||||
#!/bin/bash
|
||||
# hold_key.sh - Hold a key for specified duration
|
||||
# Usage: hold_key.sh "key" duration_seconds
|
||||
|
||||
export DISPLAY=:99
|
||||
|
||||
KEY="$1"
|
||||
DURATION="$2"
|
||||
|
||||
if [ -z "$KEY" ] || [ -z "$DURATION" ]; then
|
||||
echo "ERROR: Usage: hold_key.sh \"key\" duration_seconds" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Validate duration is reasonable
|
||||
if (( $(echo "$DURATION > 100" | bc -l) )); then
|
||||
echo "ERROR: Duration too long (max 100 seconds)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
xdotool keydown "$KEY"
|
||||
sleep "$DURATION"
|
||||
xdotool keyup "$KEY"
|
||||
|
||||
echo "Held $KEY for $DURATION seconds"
|
||||
|
||||
# Auto-screenshot after action
|
||||
sleep 2
|
||||
exec "$(dirname "$0")/screenshot.sh"
|
||||
19
scripts/key.sh
Normal file
19
scripts/key.sh
Normal file
@@ -0,0 +1,19 @@
|
||||
#!/bin/bash
|
||||
# key.sh - Press key or key combination
|
||||
# Usage: key.sh "Return" or key.sh "ctrl+c" or key.sh "alt+F4"
|
||||
|
||||
export DISPLAY=:99
|
||||
|
||||
KEY="$1"
|
||||
|
||||
if [ -z "$KEY" ]; then
|
||||
echo "ERROR: Usage: key.sh \"key_combo\"" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
xdotool key -- "$KEY"
|
||||
echo "Pressed key: $KEY"
|
||||
|
||||
# Auto-screenshot after action
|
||||
sleep 2
|
||||
exec "$(dirname "$0")/screenshot.sh"
|
||||
53
scripts/minimal-desktop.sh
Normal file
53
scripts/minimal-desktop.sh
Normal file
@@ -0,0 +1,53 @@
|
||||
#!/bin/bash
|
||||
# Minimal XFCE desktop without xfdesktop (prevents VNC flickering)
|
||||
# Runs as a watchdog: starts xfwm4+panel, kills xfdesktop if it respawns
|
||||
|
||||
export DISPLAY=:99
|
||||
|
||||
# Wait for X server
|
||||
while ! xdpyinfo -display :99 >/dev/null 2>&1; do
|
||||
sleep 0.5
|
||||
done
|
||||
|
||||
# Kill any existing session/desktop that causes flickering
|
||||
pkill -f xfce4-session 2>/dev/null
|
||||
pkill -f xfdesktop 2>/dev/null
|
||||
sleep 1
|
||||
|
||||
# Set static background (no redraw cycles = no flicker)
|
||||
xsetroot -solid "#2d3436"
|
||||
|
||||
# Disable screen blanking
|
||||
xset s off
|
||||
xset s noblank
|
||||
xset -dpms 2>/dev/null
|
||||
|
||||
# Start window manager (if not running)
|
||||
pgrep -x xfwm4 || xfwm4 &
|
||||
|
||||
# Start panel (if not running)
|
||||
pgrep -x xfce4-panel || xfce4-panel &
|
||||
|
||||
# Watchdog loop: kill flickering processes, respawn essentials
|
||||
while true; do
|
||||
# Kill unwanted processes that cause flickering
|
||||
if pgrep -x xfdesktop >/dev/null; then
|
||||
pkill -f xfdesktop
|
||||
xsetroot -solid "#2d3436"
|
||||
fi
|
||||
if pgrep -x xfce4-session >/dev/null; then
|
||||
pkill -f xfce4-session
|
||||
fi
|
||||
|
||||
# Respawn xfwm4 if it died
|
||||
if ! pgrep -x xfwm4 >/dev/null; then
|
||||
xfwm4 &
|
||||
fi
|
||||
|
||||
# Respawn panel if it died
|
||||
if ! pgrep -x xfce4-panel >/dev/null; then
|
||||
xfce4-panel &
|
||||
fi
|
||||
|
||||
sleep 1
|
||||
done
|
||||
7
scripts/mouse_down.sh
Normal file
7
scripts/mouse_down.sh
Normal file
@@ -0,0 +1,7 @@
|
||||
#!/bin/bash
|
||||
# mouse_down.sh - Press left mouse button (no release)
|
||||
|
||||
export DISPLAY=:99
|
||||
|
||||
xdotool mousedown 1
|
||||
echo "Mouse button pressed"
|
||||
16
scripts/mouse_move.sh
Normal file
16
scripts/mouse_move.sh
Normal file
@@ -0,0 +1,16 @@
|
||||
#!/bin/bash
|
||||
# mouse_move.sh - Move mouse to coordinates
|
||||
# Usage: mouse_move.sh X Y
|
||||
|
||||
export DISPLAY=:99
|
||||
|
||||
X=$1
|
||||
Y=$2
|
||||
|
||||
if [ -z "$X" ] || [ -z "$Y" ]; then
|
||||
echo "ERROR: Usage: mouse_move.sh X Y" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
xdotool mousemove --sync "$X" "$Y"
|
||||
echo "Moved mouse to $X,$Y"
|
||||
11
scripts/mouse_up.sh
Normal file
11
scripts/mouse_up.sh
Normal file
@@ -0,0 +1,11 @@
|
||||
#!/bin/bash
|
||||
# mouse_up.sh - Release left mouse button
|
||||
|
||||
export DISPLAY=:99
|
||||
|
||||
xdotool mouseup 1
|
||||
echo "Mouse button released"
|
||||
|
||||
# Auto-screenshot after action
|
||||
sleep 2
|
||||
exec "$(dirname "$0")/screenshot.sh"
|
||||
24
scripts/screenshot.sh
Normal file
24
scripts/screenshot.sh
Normal file
@@ -0,0 +1,24 @@
|
||||
#!/bin/bash
|
||||
# screenshot.sh - Capture screen and return base64 PNG
|
||||
# Resolution: 1024x768 (XGA)
|
||||
|
||||
export DISPLAY=:99
|
||||
OUTPUT_DIR="/tmp/computer-use"
|
||||
mkdir -p "$OUTPUT_DIR"
|
||||
|
||||
TIMESTAMP=$(date +%s%N)
|
||||
FILE="$OUTPUT_DIR/screenshot_$TIMESTAMP.png"
|
||||
|
||||
# Take screenshot
|
||||
scrot -o "$FILE" 2>/dev/null
|
||||
|
||||
if [ ! -f "$FILE" ]; then
|
||||
echo "ERROR: Failed to take screenshot" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Output base64
|
||||
base64 -w0 "$FILE"
|
||||
|
||||
# Cleanup
|
||||
rm -f "$FILE"
|
||||
49
scripts/scroll.sh
Normal file
49
scripts/scroll.sh
Normal file
@@ -0,0 +1,49 @@
|
||||
#!/bin/bash
|
||||
# scroll.sh - Scroll in a direction
|
||||
# Usage: scroll.sh direction amount [x y]
|
||||
# direction: up, down, left, right
|
||||
# amount: number of scroll units
|
||||
|
||||
export DISPLAY=:99
|
||||
|
||||
DIRECTION="$1"
|
||||
AMOUNT="${2:-3}"
|
||||
X="$3"
|
||||
Y="$4"
|
||||
|
||||
if [ -z "$DIRECTION" ]; then
|
||||
echo "ERROR: Usage: scroll.sh direction [amount] [x y]" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Move to position if specified
|
||||
if [ -n "$X" ] && [ -n "$Y" ]; then
|
||||
xdotool mousemove --sync "$X" "$Y"
|
||||
fi
|
||||
|
||||
# Map direction to button
|
||||
case "$DIRECTION" in
|
||||
up)
|
||||
BUTTON=4
|
||||
;;
|
||||
down)
|
||||
BUTTON=5
|
||||
;;
|
||||
left)
|
||||
BUTTON=6
|
||||
;;
|
||||
right)
|
||||
BUTTON=7
|
||||
;;
|
||||
*)
|
||||
echo "ERROR: Unknown direction: $DIRECTION (use up/down/left/right)" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
xdotool click --repeat "$AMOUNT" "$BUTTON"
|
||||
echo "Scrolled $DIRECTION $AMOUNT times"
|
||||
|
||||
# Auto-screenshot after action
|
||||
sleep 2
|
||||
exec "$(dirname "$0")/screenshot.sh"
|
||||
144
scripts/setup-vnc.sh
Normal file
144
scripts/setup-vnc.sh
Normal file
@@ -0,0 +1,144 @@
|
||||
#!/bin/bash
|
||||
# Setup stable VNC for computer-use skill
|
||||
# Run once to install systemd services for flicker-free VNC desktop
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SKILL_DIR="$(dirname "$SCRIPT_DIR")"
|
||||
USER=$(whoami)
|
||||
HOME_DIR=$(eval echo "~$USER")
|
||||
|
||||
echo "=== Computer Use VNC Setup ==="
|
||||
echo "User: $USER"
|
||||
echo "Skill dir: $SKILL_DIR"
|
||||
echo ""
|
||||
|
||||
# Install packages
|
||||
echo "[1/6] Installing packages..."
|
||||
sudo apt update -qq
|
||||
sudo apt install -y xvfb xfce4 xfce4-terminal xdotool scrot imagemagick dbus-x11 x11vnc novnc websockify
|
||||
|
||||
# Copy minimal-desktop.sh to a stable location
|
||||
echo "[2/6] Installing watchdog script..."
|
||||
sudo mkdir -p /opt/computer-use
|
||||
sudo cp "$SCRIPT_DIR/minimal-desktop.sh" /opt/computer-use/
|
||||
sudo chmod +x /opt/computer-use/minimal-desktop.sh
|
||||
|
||||
# Install systemd services (generated inline)
|
||||
echo "[3/6] Installing systemd services..."
|
||||
|
||||
cat <<EOF | sudo tee /etc/systemd/system/xvfb.service > /dev/null
|
||||
[Unit]
|
||||
Description=Xvfb Virtual Display :99
|
||||
After=graphical.target
|
||||
Wants=graphical.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/usr/bin/Xvfb :99 -screen 0 1024x768x24 -nolisten tcp -dpi 96
|
||||
Restart=always
|
||||
RestartSec=1
|
||||
User=$USER
|
||||
Environment="HOME=$HOME_DIR"
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
|
||||
cat <<EOF | sudo tee /etc/systemd/system/xfce-minimal.service > /dev/null
|
||||
[Unit]
|
||||
Description=XFCE Minimal Desktop for Computer Use
|
||||
After=xvfb.service
|
||||
Requires=xvfb.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/opt/computer-use/minimal-desktop.sh
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
User=$USER
|
||||
Environment="HOME=$HOME_DIR"
|
||||
Environment="DISPLAY=:99"
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
|
||||
cat <<EOF | sudo tee /etc/systemd/system/x11vnc.service > /dev/null
|
||||
[Unit]
|
||||
Description=x11vnc VNC Server
|
||||
After=xfce-minimal.service
|
||||
Requires=xfce-minimal.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/usr/bin/x11vnc -display :99 -forever -shared -rfbport 5900 -noxdamage -noxfixes -noclipboard
|
||||
Restart=always
|
||||
RestartSec=2
|
||||
User=$USER
|
||||
Environment="HOME=$HOME_DIR"
|
||||
Environment="DISPLAY=:99"
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
|
||||
cat <<EOF | sudo tee /etc/systemd/system/novnc.service > /dev/null
|
||||
[Unit]
|
||||
Description=noVNC WebSocket Proxy
|
||||
After=x11vnc.service
|
||||
Requires=x11vnc.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/usr/share/novnc/utils/novnc_proxy --vnc localhost:5900 --listen 6080 --heartbeat 30
|
||||
Restart=always
|
||||
RestartSec=2
|
||||
User=$USER
|
||||
Environment="HOME=$HOME_DIR"
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
|
||||
# Mask xfdesktop to prevent flickering
|
||||
echo "[4/6] Masking xfdesktop (prevents flicker)..."
|
||||
if [ -f /usr/bin/xfdesktop ] && [ ! -f /usr/bin/xfdesktop.real ]; then
|
||||
sudo mv /usr/bin/xfdesktop /usr/bin/xfdesktop.real
|
||||
echo '#!/bin/bash
|
||||
# Masked - xfdesktop causes VNC flickering on Xvfb
|
||||
exit 0' | sudo tee /usr/bin/xfdesktop > /dev/null
|
||||
sudo chmod +x /usr/bin/xfdesktop
|
||||
echo " xfdesktop masked (original at /usr/bin/xfdesktop.real)"
|
||||
else
|
||||
echo " xfdesktop already masked or not found"
|
||||
fi
|
||||
|
||||
# Enable and start services
|
||||
echo "[5/6] Enabling services..."
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable xvfb xfce-minimal x11vnc novnc
|
||||
|
||||
echo "[6/6] Starting services..."
|
||||
sudo systemctl start xvfb
|
||||
sleep 2
|
||||
sudo systemctl start xfce-minimal
|
||||
sleep 3
|
||||
sudo systemctl start x11vnc
|
||||
sleep 1
|
||||
sudo systemctl start novnc
|
||||
|
||||
echo ""
|
||||
echo "=== Setup Complete ==="
|
||||
echo ""
|
||||
echo "Services running:"
|
||||
systemctl is-active xvfb xfce-minimal x11vnc novnc | paste - - - - | awk '{print " xvfb: "$1" xfce-minimal: "$2" x11vnc: "$3" novnc: "$4}'
|
||||
echo ""
|
||||
echo "Access VNC:"
|
||||
echo " 1. SSH tunnel: ssh -L 6080:localhost:6080 $(hostname)"
|
||||
echo " 2. Open: http://localhost:6080/vnc.html"
|
||||
echo ""
|
||||
echo "Or add to ~/.ssh/config:"
|
||||
echo " Host $(hostname)"
|
||||
echo " LocalForward 6080 127.0.0.1:6080"
|
||||
30
scripts/type_text.sh
Normal file
30
scripts/type_text.sh
Normal file
@@ -0,0 +1,30 @@
|
||||
#!/bin/bash
|
||||
# type_text.sh - Type text with realistic delays
|
||||
# Usage: type_text.sh "text to type"
|
||||
# Types in 50 character chunks with 12ms delay between keystrokes
|
||||
|
||||
export DISPLAY=:99
|
||||
|
||||
TEXT="$1"
|
||||
|
||||
if [ -z "$TEXT" ]; then
|
||||
echo "ERROR: Usage: type_text.sh \"text to type\"" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Type in chunks of 50 characters
|
||||
CHUNK_SIZE=50
|
||||
LENGTH=${#TEXT}
|
||||
OFFSET=0
|
||||
|
||||
while [ $OFFSET -lt $LENGTH ]; do
|
||||
CHUNK="${TEXT:$OFFSET:$CHUNK_SIZE}"
|
||||
xdotool type --delay 12 -- "$CHUNK"
|
||||
OFFSET=$((OFFSET + CHUNK_SIZE))
|
||||
done
|
||||
|
||||
echo "Typed ${#TEXT} characters"
|
||||
|
||||
# Auto-screenshot after action
|
||||
sleep 2
|
||||
exec "$(dirname "$0")/screenshot.sh"
|
||||
33
scripts/vnc_start.sh
Normal file
33
scripts/vnc_start.sh
Normal file
@@ -0,0 +1,33 @@
|
||||
#!/bin/bash
|
||||
# Start VNC services for live desktop viewing
|
||||
|
||||
DISPLAY_NUM="${DISPLAY_NUM:-:99}"
|
||||
|
||||
# Kill existing instances
|
||||
pkill -f "x11vnc.*display $DISPLAY_NUM" 2>/dev/null
|
||||
pkill -f "websockify.*6080" 2>/dev/null
|
||||
sleep 1
|
||||
|
||||
# Start x11vnc (VNC server)
|
||||
echo "Starting x11vnc on $DISPLAY_NUM..."
|
||||
x11vnc -display "$DISPLAY_NUM" -forever -shared -nopw -listen localhost &
|
||||
sleep 2
|
||||
|
||||
# Start websockify (noVNC web bridge)
|
||||
echo "Starting noVNC on port 6080..."
|
||||
websockify --web=/usr/share/novnc 6080 localhost:5900 &
|
||||
sleep 1
|
||||
|
||||
# Verify
|
||||
if pgrep -f x11vnc > /dev/null && pgrep -f websockify > /dev/null; then
|
||||
echo ""
|
||||
echo "✓ VNC services started"
|
||||
echo ""
|
||||
echo "To connect:"
|
||||
echo " 1. SSH tunnel: ssh -L 6080:localhost:6080 your-server"
|
||||
echo " 2. Open: http://localhost:6080/vnc.html?autoconnect=true"
|
||||
echo ""
|
||||
else
|
||||
echo "✗ Failed to start VNC services"
|
||||
exit 1
|
||||
fi
|
||||
16
scripts/vnc_stop.sh
Normal file
16
scripts/vnc_stop.sh
Normal file
@@ -0,0 +1,16 @@
|
||||
#!/bin/bash
|
||||
# Stop VNC services
|
||||
|
||||
echo "Stopping VNC services..."
|
||||
|
||||
pkill -f x11vnc 2>/dev/null
|
||||
pkill -f websockify 2>/dev/null
|
||||
|
||||
sleep 1
|
||||
|
||||
if ! pgrep -f x11vnc > /dev/null && ! pgrep -f websockify > /dev/null; then
|
||||
echo "✓ VNC services stopped"
|
||||
else
|
||||
echo "✗ Some processes may still be running"
|
||||
ps aux | grep -E "(x11vnc|websockify)" | grep -v grep
|
||||
fi
|
||||
24
scripts/wait.sh
Normal file
24
scripts/wait.sh
Normal file
@@ -0,0 +1,24 @@
|
||||
#!/bin/bash
|
||||
# wait.sh - Wait for specified duration then screenshot
|
||||
# Usage: wait.sh seconds
|
||||
|
||||
export DISPLAY=:99
|
||||
|
||||
DURATION="$1"
|
||||
|
||||
if [ -z "$DURATION" ]; then
|
||||
echo "ERROR: Usage: wait.sh seconds" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Validate duration is reasonable
|
||||
if (( $(echo "$DURATION > 100" | bc -l) )); then
|
||||
echo "ERROR: Duration too long (max 100 seconds)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
sleep "$DURATION"
|
||||
echo "Waited $DURATION seconds"
|
||||
|
||||
# Screenshot after waiting
|
||||
exec "$(dirname "$0")/screenshot.sh"
|
||||
49
scripts/zoom.sh
Normal file
49
scripts/zoom.sh
Normal file
@@ -0,0 +1,49 @@
|
||||
#!/bin/bash
|
||||
# zoom.sh - Capture cropped region of screen
|
||||
# Usage: zoom.sh X1 Y1 X2 Y2
|
||||
# Returns base64 of the cropped region
|
||||
|
||||
export DISPLAY=:99
|
||||
OUTPUT_DIR="/tmp/computer-use"
|
||||
mkdir -p "$OUTPUT_DIR"
|
||||
|
||||
X1=$1
|
||||
Y1=$2
|
||||
X2=$3
|
||||
Y2=$4
|
||||
|
||||
if [ -z "$X1" ] || [ -z "$Y1" ] || [ -z "$X2" ] || [ -z "$Y2" ]; then
|
||||
echo "ERROR: Usage: zoom.sh X1 Y1 X2 Y2" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
TIMESTAMP=$(date +%s%N)
|
||||
FULL_FILE="$OUTPUT_DIR/full_$TIMESTAMP.png"
|
||||
CROP_FILE="$OUTPUT_DIR/crop_$TIMESTAMP.png"
|
||||
|
||||
# Take full screenshot
|
||||
scrot -o "$FULL_FILE" 2>/dev/null
|
||||
|
||||
if [ ! -f "$FULL_FILE" ]; then
|
||||
echo "ERROR: Failed to take screenshot" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Calculate crop dimensions
|
||||
WIDTH=$((X2 - X1))
|
||||
HEIGHT=$((Y2 - Y1))
|
||||
|
||||
# Crop using ImageMagick
|
||||
convert "$FULL_FILE" -crop "${WIDTH}x${HEIGHT}+${X1}+${Y1}" +repage "$CROP_FILE"
|
||||
|
||||
if [ ! -f "$CROP_FILE" ]; then
|
||||
echo "ERROR: Failed to crop screenshot" >&2
|
||||
rm -f "$FULL_FILE"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Output base64
|
||||
base64 -w0 "$CROP_FILE"
|
||||
|
||||
# Cleanup
|
||||
rm -f "$FULL_FILE" "$CROP_FILE"
|
||||
Reference in New Issue
Block a user