callstack__agent-device/test/skillgym

Skillgym For agent-device

This folder is a starter skillgym setup for benchmarking the agent-device skill with a controlled Expo target app.

Why skillgym fits here

skillgym is useful for agent-device in three layers:

1. Skill-routing checks: verify that the runner loads skills/agent-device/SKILL.md and its required references before it answers.
2. Workflow-planning checks: verify that the agent describes the right agent-device loop for a known fixture app.
3. Optional live-device smoke runs: locally, you can extend prompts so the agent actually drives agent-device against a simulator or device.

The included suite focuses on the first two layers so it stays stable and CI-safe. The suite uses SkillGym v0.8 case tags:

• fixture-smoke: fixture-specific app surface coverage
• skill-guidance: command-planning guidance regressions

Included files

• ../../examples/test-app/: minimal Expo SDK 55 fixture app for broad UI coverage
• skillgym.config.ts: starter config that runs Codex and Claude Haiku against this repo
• suites/agent-device-smoke-suite.ts: planning suite for skill routing, fixture-aware flows, and skill-guidance regressions

Current coverage

The suite keeps the app small while separating coverage into two non-overlapping groups.

Fixture smoke cases cover concrete app surfaces:

• open/snapshot/close defaults with Expo Go
• banners, alerts, toggles, and quick actions on Home
• search debounce, filters, long-list scroll, favorites, and cart updates in Catalog
• detail navigation, quantity edits, note append, and save-to-cart on Product
• form validation, success submit, iOS keyboard-dismiss fallback, and reset on Checkout form
• diagnostics load/error/retry plus reset alert handling in Settings
• accessibility audit via screenshot + snapshot

Skill-guidance regression cases cover distinct command-planning habits:

• read-only inspection versus mutation
• fresh @ref targeting, durable selectors, raw-rect fallbacks, and off-screen scroll recovery
• text replacement, append semantics, supported field clearing, keyboard status, and keyboard fallback
• install/open setup, Expo Go host-shell launch, app discovery, session scoping, and app-owned navigation fallbacks
• Metro reload, logs, network dump, alert fallback, and screenshot evidence
• performance metrics, React DevTools profiling, gestures, settings, and trace capture
• remote config, macOS menu bar surfaces, replay update, same-session mutation ordering, and batch schema/recording

assertAgentDeviceEvidence is intentionally soft when a runner does not expose skill-detection telemetry. When telemetry exists, the suite asserts that agent-device was loaded; when it is absent, the cases still judge command-planning output instead of failing on missing runner metadata.

The codex-mini baseline is a benchmark signal, not a required all-green gate. Its failures should map to command-planning regressions called out by individual case IDs; do not treat the historical pass/fail count as a fixed threshold.

SkillGym v0.8 command assertions are for observed command events. This suite primarily validates the command plan in the final answer, so it converts final-output command lines into a small planned-command report before calling assert.commands.includes or assert.commands.notIncludes. The source-read guardrails use assert.soft.* plus deferred explain questions so one failing run can report multiple routing mistakes and can later be inspected with skillgym explain. Suite types use the v0.8 root export name Case; older TestCase imports no longer typecheck.

Suggested workflow

1. Start with the included smoke suite to benchmark routing and default guidance.
2. Extend the suite with app-specific prompts that cover a new command-planning category rather than duplicating an existing one.
3. Add local-only cases that expect real agent-device shell commands once you are ready to involve a running simulator.

Running the suite

skillgym is installed as a repo dev dependency, so run the starter suite from the project root:

cd /absolute/path/to/agent-device
pnpm install
pnpm test:skillgym

Prefer the package scripts so the environment guard and local CLI build run consistently:

cd /absolute/path/to/agent-device
pnpm test:skillgym
pnpm test:skillgym:case open-and-snapshot

Useful v0.8 filters, reporters, and recovery options:

pnpm test:skillgym -- --tag fixture-smoke
pnpm test:skillgym -- --reporter json
pnpm test:skillgym -- --repeat 3 --repeat-failure 1

If you need to run skillgym directly while developing the runner itself, build first so agents can call node bin/agent-device.mjs help workflow:

pnpm build
pnpm exec skillgym run \
  ./test/skillgym/suites/agent-device-smoke-suite.ts \
  --config ./test/skillgym/skillgym.config.ts \
  --case open-and-snapshot

Use --reporter github-actions in CI when you want annotations in GitHub Actions logs.

The config uses schedule: parallel so the planning suite can run case/runner pairs concurrently up to SkillGym v0.8's default available-machine parallelism cap. This is safe for the included suite because cases validate command plans and local CLI help, not live shared device state or workspace edits. Override with --max-parallel <n> for local experiments that need a different cap. Use --repeat <n> when you want stability sampling rather than a single pass. Use --repeat-failure <n> for local benchmark recovery from transient runner failures; keep it off for strict regression checks unless you explicitly want retry artifacts. When a run fails on an assertion that records explain questions, run pnpm exec skillgym explain <artifact-dir> against the failed repeat-* artifact directory to resume the runner and collect its explanation.

Prerequisites:

• codex CLI installed and authenticated, because the starter config uses the Codex runner
• claude CLI installed and authenticated, because the same cases also run against Claude Haiku
• repo dependencies installed with pnpm install
• if you want the fixture app running locally, use pnpm test-app:install and then pnpm test-app:ios or pnpm test-app:android

Sandbox note:

The configured runners call external Codex and Claude model backends. In Codex sandboxes with CODEX_SANDBOX_NETWORK_DISABLED=1, pnpm test:skillgym and direct skillgym run --config ./test/skillgym/skillgym.config.ts fail fast before building or launching runners. Run the suite from a normal authenticated local shell instead. If you are in a sandbox that has explicitly approved network access and you still want to launch external runners, set SKILLGYM_ALLOW_EXTERNAL_RUNNERS_IN_SANDBOX=1.

Where to extend next

• Add suite cases that ask for selector-based plans against Agent Device Tester.
• Add local-only prompts that expect agent-device open, snapshot, snapshot -i, get, and wait.
• Add regression snapshots once the prompt set stabilizes.