* docs(cli): advertise open --foreground and snapshot --actions in the workflow card
open --foreground (#1670/#1671) and snapshot -i --actions (#1665) shipped
with no mention in the compact `help workflow` card, so a planning model
never discovers either. Add one terse line each: the foreground fast-path
in Bootstrap, and the merged-element custom-action guidance in Validation
and evidence. Stays under the 9,000-byte compact-card budget (8493 -> 8908
bytes).
Adds two help-conformance bench cases per the repo's changed-guidance rule:
foreground-attach-single-sim (correct plan starts with `open --foreground`
in an unambiguous single-sim scenario, fail-closed alternative forbidden)
and merged-card-actions-not-directly-invokable (a merged Bluesky-style feed
card's actions list is evidence, not a selector). Both use a real pinned
sample rebuilt through the production snapshot renderer.
* fix(scripts): accept flag order in the foreground-attach conformance matcher
Flag order after `open` isn't semantically meaningful (`open --platform ios
--foreground` is exactly as correct as `open --foreground --platform ios`),
but startsWithForegroundOpen required --foreground to be the literal next
token after `open`. Rescoring the completed repeat=3 bench report shows this
docked codex:gpt-5.4-mini on all 3 trials even though its plan was
config-order noise, not a real deviation -- the no-positional/no-device
guarantee already comes from the forbidden checks. Loosened to require
--foreground anywhere on the open line; foreground-attach-single-sim now
scores 54/54 across both runners.
* fix: close workflow help conformance gaps
* feat(cli): compact workflow help card + version header
Shrinks the per-task agent protocol tax of the help/skill surface.
`agent-device help workflow` drops from 41025 to 8466 bytes (-79%) by
moving depth into new `help scripting` (save-script, secret-safe
fills, batch JSON, replay divergence/repair) and `help gestures`
(multi-touch shapes/quirks) topics, and folding a few paragraphs into
topics that already owned the subject (help debugging,
help physical-device, help validate). Content is moved, not deleted.
Every `help <topic>` first line is now `agent-device <version> —
<topic>`, so the skill router reads the CLI version off the mandatory
first help read instead of a separate `agent-device --version` call.
SKILL.md is updated to do that and stays a thin router otherwise.
The compact card also gains two terse behavioral rules: chain
confident consecutive steps with `&&` (falling back to one command at
a time when uncertain), and confirm the requested end state is
actually visible on screen before declaring a task done.
help-conformance-bench (22 cases x 2 runners) improves after the
change: 29/44 -> 32/44 passing checks.
* fix(cli): review follow-ups on the compact workflow card (#1663)
Three fixes from PR review:
- Extend the help-conformance plan validator to split a command line
on unquoted && and validate each chained segment independently, so
a plan that follows the workflow card's "chain confident consecutive
steps with &&" guidance is accepted instead of rejected as one
shell-projection violation. && inside a quoted selector value (e.g.
label="A && B") is not a chain boundary and does not split. Adds
unit tests for the splitter and a chains-confident-consecutive-
settle-steps conformance case. batch stays out of this: it is
deliberately stop-only.
- Replace the literal @ref placeholder the compact card used in its
own "snapshot -s @ref" example with a concrete ref
(snapshot -s @e12 (the current concrete ref)), matching the same
card's rule against placeholder targets. Reverts the test to demand
the concrete shape.
- Give help scripting and help gestures real conformance cases
instead of waivers: a secret-safe recorded-fill + publish case, and
an Android transform-then-verify case whose exact verification text
only appears in the gestures topic. Removes both waivers.
help-conformance-bench (25 cases x 2 runners, repeat=1) after these
fixes: two full runs landed at 32/50 and 33/50. That is on par with
the pre-change baseline (29/44) once the topic-untouched cases'
run-to-run swings are accounted for (confirmed noise: one case with
zero exposure to any change here flipped 10/10 -> 1/10 on a runner API
error, and another swung across all three post-fix runs). The new
scripting case now passes 8/8 for both runners; the new chaining case
correctly reports the model's choice not to chain as a soft signal,
not a validator failure.
* fix(cli): update session.test.ts help pointer for moved script-authoring content
* fix(cli): reject empty && chain operands in the plan validator (#1663)
splitOnUnquotedAnd() previously trimmed and filtered out empty
segments, so a plan with a leading (`&& press ...`), trailing
(`press ... &&`), or doubled (`a && && b`) operator passed
validPlanCommands even though a real shell rejects all three as a
syntax error. The validator would bless a plan that fails at
execution.
Empty segments are now surfaced as an `empty-chain-operand` issue
instead of being silently dropped. The quoted-&& non-split behavior
(label="A && B") is unchanged, and a normal single command with no
chain still parses identically to before.
Adds regression tests for all three empty-operand shapes plus the
quoted-&& case.
* feat(bench): renderer-pinned samples, topic-coverage gate, error quizzes; trim skillgym to agentic checks
The help conformance bench's quoted CLI output is now sourced from
scripts/help-conformance-sample-outputs.mjs, and every sample is rebuilt
through the real production renderers (settle output formatters,
printHumanError, formatSnapshotText, refMutationAdmissionResponse) by
scripts/__tests__/help-conformance-sample-outputs.test.ts — a rendering or
message change fails deterministically instead of leaving the bench grading
against output the CLI no longer prints. This retires the fabricated
recoverable-failure envelope (production never throws a textual settle
timeout; that case is replaced by a real DEVICE_IN_USE recovery quiz).
Bench cases move to scripts/help-conformance-cases.mjs and are enumerated
against the help-topic registry: helpTopicIds() is exported from cli-help,
and scripts/__tests__/help-conformance-topic-coverage.test.ts fails when a
help topic has neither a bench case nor an explicit waiver. New case
families: error-envelope recovery quizzes (device-in-use, stale pinned ref,
ambiguous find match, app-not-installed) pinned to real error text, topic
coverage for tv/web/react-native/debugging/workflow, and a metamorphic twin
of the settled-diff quiz.
The skillgym smoke suite shrinks from 119 cases to the 5 that measure what
only an agentic runner can show: skill routing plus output interpretation
with a proven local CLI help probe (local-cli-help-policy). Its embedded
samples now import the same pinned constants, replacing hand-transcribed
output that had already drifted from the renderer. Knowledge checks belong
to the bench; live fixture behavior belongs to the iOS simulator e2e suite.
* review: drive error samples through the real producers; enforce local-help on the routing smoke
The DEVICE_IN_USE, AMBIGUOUS_MATCH, and APP_NOT_INSTALLED parity tests no
longer hand-author the producer message before rendering: each drives the
actual producer — buildDeviceInUseBySessionError (extracted in
session-open.ts and called by the handler), buildAmbiguousMatchError (now
exported from find.ts), and buildAppNotInstalledError (extracted in
app-resolution.ts and thrown by the resolver). Because each factory is
exported from its producer file and called by the production path, dropping
the production call would make it test-only and fail
check:production-exports — the wiring is gate-enforced, not conventional.
open-and-snapshot now sets requireLocalCliHelp and
allowOnlyLocalCliHelpCommands, so the 'skill plus local help' claim is
observed rather than assumed; without them the case can pass on model prior
alone.
* feat: fix codex runner, add override-doc grading, port skillgym quiz cases
The 2026-07-09 evaluation of scripts/help-conformance-bench.mjs found it
structurally right but broken for the codex runner (two bugs), thin on
coverage (4 cases), sequential, and unable to grade a draft help rewrite
without a rebuild.
- Fix runCodex: (1) codex exec reads stdin until EOF when not attached to
a TTY, and execFile never closes the child's stdin, so every codex call
hung until RUN_TIMEOUT_MS with empty output — close stdin right after
spawn. (2) `-o outFile` writes the same final JSON that codex also
prints to stdout, so concatenating both produced two back-to-back JSON
objects that broke every JSON.parse candidate and silently zeroed
extractCommands() — prefer the clean -o payload, fall back to stdout
only when it's empty.
- Add `--override-doc <topicId>=<path>` (repeatable): loads a topic's
text from a file instead of shelling out to
`node bin/agent-device.mjs help <topic>`, so a draft help rewrite can
be A/B graded with zero rebuild.
- Port three cases from
test/skillgym/suites/agent-device-smoke-suite.ts
(settle-diff-is-observation, sample-output-settled-diff-next-target,
sample-output-not-settled-needs-observe) as self-contained
"next-command quiz" cases, generalizing the scorer to support regex
matchers/forbidden patterns alongside the existing named expectations.
Fixture output text matches the CURRENT settle rendering in
src/commands/interaction/output.ts, including the "unchanged
interactive (N):" tail added by #1167/#1172.
- Parallelize the runner x case matrix with a concurrency cap
(HELP_BENCH_CONCURRENCY, default 4); results still print in the
original matrix order.
- Extend test/skillgym/README.md's existing pointer to this bench with
the new flags.
Validated with real LLM calls (both runners, all 7 cases, 14 calls,
~$0.25 total): 13/14 pass; the one fail (claude-haiku-4-5 on
dogfood-mode) is a genuine model miss (returned an empty command plan
asking for the app name instead of committing to a generic plan), not a
bench bug. `--override-doc` demonstrated live: stripping the dogfood
doc's evidence-command examples regresses codex:gpt-5.4-mini from 3/3 to
2/3 on the same case, showing the flag both loads and changes grading.
* fix: apply live-doc post-processing to --override-doc, fail fast on bad overrides
Review findings on the initial version (all reproduced):
- HIGH: an override for the --help:first30 doc id skipped the live path's
firstLines(text, 30) cap, so a 49-line draft leaked lines 31-49 into the
prompt — grading content a live run never shows, on the doc id every case
uses. loadDoc now splits source (live shell-out vs override file) from
post-processing, and the post-processing applies to both, so an override
differs ONLY in where the text comes from.
- MEDIUM: an --override-doc topic id no selected case uses was silently
ignored (exit 0, real doc graded). Now fails fast listing the valid doc
ids for the selection.
- LOW: a missing override file threw a raw ENOENT stack trace; expected
failures now print one clean Error line. Added --help usage text that
documents last-wins semantics for repeated same-topic overrides and the
post-processing parity.
Guard tests (scripts/__tests__/help-conformance-bench.test.ts, wired into
the unit-core vitest project by explicit path): a 49-line fixture whose
prompt must keep line 30 and drop line 31, unknown-topic fail-fast with
valid ids listed, clean no-stack error for a missing file, and last-wins
for repeated overrides. All spawn the script in --dry-run with every
required doc overridden, so they need no LLM calls and no built CLI.
Live re-validation: a 33-line override of --help:first30 whose lines
31-33 instruct the model to emit a sentinel command; neither
claude-haiku-4-5 nor codex:gpt-5.4-mini emitted it (both scored 4/4,
matching the live-doc baseline), proving the cap applies end-to-end.
Michał Pierzchała