callstack__agent-device/test/skillgym

Skillgym For agent-device

This folder is a starter skillgym setup for benchmarking the agent-device skill with a controlled Expo target app.

Why skillgym fits here

skillgym is useful for agent-device in three layers:

1. Skill-routing checks: verify that the runner loads skills/agent-device/SKILL.md and its required references before it answers.
2. Workflow-planning checks: verify that the agent describes the right agent-device loop for a known fixture app.
3. Optional live-device smoke runs: locally, you can extend prompts so the agent actually drives agent-device against a simulator or device.

The included suite focuses on the first two layers so it stays stable and CI-safe. The suite uses SkillGym v0.8 case tags:

• fixture-smoke: fixture-specific app surface coverage
• skill-guidance: command-planning guidance regressions

Included files

• ../../examples/test-app/: minimal Expo SDK 56 development-build fixture app for broad UI coverage
• skillgym.config.ts: starter config that runs Codex and Claude Haiku against this repo
• suites/agent-device-smoke-suite.ts: planning suite for skill routing, fixture-aware flows, and skill-guidance regressions

Current coverage

The suite keeps the app small while separating coverage into two non-overlapping groups.

Fixture smoke cases cover concrete app surfaces:

• open/snapshot/close defaults with the installed Expo development build
• banners, alerts, toggles, and quick actions on Home
• search debounce, filters, long-list scroll, favorites, and cart updates in Catalog
• detail navigation, quantity edits, note append, and save-to-cart on Product
• form validation, success submit, iOS keyboard-dismiss fallback, and reset on Checkout form
• diagnostics load/error/retry plus reset alert handling in Settings
• accessibility audit via screenshot + snapshot

Skill-guidance regression cases cover distinct command-planning habits:

• read-only inspection versus mutation
• fresh @ref targeting, durable selectors, raw-rect fallbacks, and off-screen scroll recovery
• text replacement, append semantics, supported field clearing, keyboard status, and keyboard fallback
• install/open setup, Expo Go/dev-client launch paths, app discovery, session scoping, and app-owned navigation fallbacks
• Metro reload, logs, network dump, alert fallback, and screenshot evidence
• performance metrics, React DevTools profiling, gestures, settings, and trace capture
• remote config, macOS menu bar surfaces, replay update, same-session mutation ordering, and batch schema/recording

assertAgentDeviceEvidence is intentionally soft when a runner does not expose skill-detection telemetry. When telemetry exists, the suite asserts that agent-device was loaded; when it is absent, the cases still judge command-planning output instead of failing on missing runner metadata.

The codex-mini baseline is a benchmark signal, not a required all-green gate. Its failures should map to command-planning regressions called out by individual case IDs; do not treat the historical pass/fail count as a fixed threshold.

SkillGym v0.8 command assertions are for observed command events. This suite primarily validates the command plan in the final answer, so it converts final-output command lines into a small planned-command report before calling assert.commands.includes or assert.commands.notIncludes. The source-read guardrails use assert.soft.* plus deferred explain questions so one failing run can report multiple routing mistakes and can later be inspected with skillgym explain. Suite types use the v0.8 root export name Case; older TestCase imports no longer typecheck.

Suggested workflow

1. Start with the included smoke suite to benchmark routing and default guidance.
2. Extend the suite with app-specific prompts that cover a new command-planning category rather than duplicating an existing one.
3. Add local-only cases that expect real agent-device shell commands once you are ready to involve a running simulator.

Running the suite

skillgym is installed as a repo dev dependency, so run the starter suite from the project root:

cd /absolute/path/to/agent-device
pnpm install
pnpm test:skillgym

Prefer the package scripts so the environment guard and local CLI build run consistently:

cd /absolute/path/to/agent-device
pnpm test:skillgym
pnpm test:skillgym:case open-and-snapshot

Useful v0.8 filters, reporters, and recovery options:

pnpm test:skillgym -- --tag fixture-smoke
pnpm test:skillgym -- --reporter json
pnpm test:skillgym -- --repeat 3 --repeat-failure 1

Optional Vercel AI Gateway runner:

AI_GATEWAY_API_KEY=<token> \
SKILLGYM_ENABLE_VERCEL_GATEWAY=1 \
pnpm test:skillgym:case open-and-snapshot --runner gpt-nano-gateway

gpt-nano-gateway uses SkillGym's OpenCode adapter with a repo-injected @ai-sdk/openai-compatible provider pointed at https://ai-gateway.vercel.sh/v1 and model openai/gpt-5.4-nano. It is disabled by default so normal runs do not require Gateway credentials, OpenCode auth, or Gateway spend. VERCEL_OIDC_TOKEN can be used instead of AI_GATEWAY_API_KEY; the config passes either token as the bearer credential for Gateway.

If you need to run skillgym directly while developing the runner itself, build first so agents can call node bin/agent-device.mjs help workflow:

pnpm build
pnpm exec skillgym run \
  ./test/skillgym/suites/agent-device-smoke-suite.ts \
  --config ./test/skillgym/skillgym.config.ts \
  --case open-and-snapshot

Use --reporter github-actions in CI when you want annotations in GitHub Actions logs.

The config uses schedule: parallel so the planning suite can run case/runner pairs concurrently up to SkillGym v0.8's default available-machine parallelism cap. This is safe for the included suite because cases validate command plans and local CLI help, not live shared device state or workspace edits. Override with --max-parallel <n> for local experiments that need a different cap. Use --repeat <n> when you want stability sampling rather than a single pass. Use --repeat-failure <n> for local benchmark recovery from transient runner failures; keep it off for strict regression checks unless you explicitly want retry artifacts. When a run fails on an assertion that records explain questions, run pnpm exec skillgym explain <artifact-dir> against the failed repeat-* artifact directory to resume the runner and collect its explanation.

Prerequisites:

• codex CLI installed and authenticated, because the starter config uses the Codex runner
• claude CLI installed and authenticated, because the same cases also run against Claude Haiku
• repo dependencies installed with pnpm install
• if you want the fixture app running locally, use pnpm test-app:install and then pnpm test-app:ios or pnpm test-app:android

Sandbox note:

The configured runners call external Codex and Claude model backends. In Codex sandboxes with CODEX_SANDBOX_NETWORK_DISABLED=1, pnpm test:skillgym and direct skillgym run --config ./test/skillgym/skillgym.config.ts fail fast before building or launching runners. Run the suite from a normal authenticated local shell instead. If you are in a sandbox that has explicitly approved network access and you still want to launch external runners, set SKILLGYM_ALLOW_EXTERNAL_RUNNERS_IN_SANDBOX=1.

Where to extend next

• Add suite cases that ask for selector-based plans against Agent Device Tester.
• Add local-only prompts that expect agent-device open, snapshot, snapshot -i, get, and wait.
• Add regression snapshots once the prompt set stabilizes.