* test(bench): falsification fixtures for oracles + typed runner outcomes
Two deterministic PR-time quality gates for the help-conformance bench
(the repo's single non-gating small-model planning oracle):
- Every EXPECTATION_SCORERS entry in help-conformance-case-checks.mjs
now has a falsification fixture (a minimal passing witness plus at
least one known-bad counterexample, and a metamorphic variant where
useful) in the new help-conformance-expectation-fixtures.ts, run
through the real validatePlanCommands/scoreExpectations pipeline.
help-conformance-expectation-falsification.test.ts is the "what
enumerates N" completeness gate: a new named expectation with no
fixture fails it. Counterexamples cover swallowed lifecycle command
prefixes, unsupported flags/selectors, pseudo refs, shell operators,
and invalid positional ordering.
- help-conformance-runner-output.mjs now returns a discriminated
RunnerOutcome ({kind:'success',commands}|{kind:'runner-error',
message,reason}) instead of a raw-string success inference. Only a
'success' outcome ever reaches validatePlanCommands/scoreExpectations
in runCase, so a runner-error result can no longer also carry
model-validation checks, and an all-runner-error aggregate now
reports passRate: null (rendered as "N/A") instead of "0/0 (0%)".
Fixes#1481
* refactor(bench): dedupe RunnerOutcome construction, drop leftover narrowing
Thermo-nuclear pass over 4b2df0a38's diff:
- help-conformance-bench.mjs's runOutcome() catch block was hand-building
the exact {kind:'runner-error', raw, message, reason} shape that
runner-output.mjs's private runnerError() helper already constructs for
its own two error paths. Export it as runnerErrorOutcome so the
discriminated union has exactly one constructor for its error variant,
reused by both error sources instead of duplicated.
- runCase's two return branches repeated the same
{runner, caseId, trial, outputPath} fields; pulled into a shared `base`
object.
- Reverted bench.test.ts's rateLimitedOutcome block: it had an explicit
`: RunnerOutcome` annotation and an if/throw narrowing guard, added only
to give fallow's dead-code checker a "real consumer" of the type before
the actual fix (adding the .d.mts to .fallowrc.json's ignorePatterns,
matching the existing sample-outputs.d.mts precedent) was found. That
workaround is now unnecessary scaffolding — replaced with the same
flat assert.deepEqual style the surrounding assertions already use.
* docs(cli): advertise open --foreground and snapshot --actions in the workflow card
open --foreground (#1670/#1671) and snapshot -i --actions (#1665) shipped
with no mention in the compact `help workflow` card, so a planning model
never discovers either. Add one terse line each: the foreground fast-path
in Bootstrap, and the merged-element custom-action guidance in Validation
and evidence. Stays under the 9,000-byte compact-card budget (8493 -> 8908
bytes).
Adds two help-conformance bench cases per the repo's changed-guidance rule:
foreground-attach-single-sim (correct plan starts with `open --foreground`
in an unambiguous single-sim scenario, fail-closed alternative forbidden)
and merged-card-actions-not-directly-invokable (a merged Bluesky-style feed
card's actions list is evidence, not a selector). Both use a real pinned
sample rebuilt through the production snapshot renderer.
* fix(scripts): accept flag order in the foreground-attach conformance matcher
Flag order after `open` isn't semantically meaningful (`open --platform ios
--foreground` is exactly as correct as `open --foreground --platform ios`),
but startsWithForegroundOpen required --foreground to be the literal next
token after `open`. Rescoring the completed repeat=3 bench report shows this
docked codex:gpt-5.4-mini on all 3 trials even though its plan was
config-order noise, not a real deviation -- the no-positional/no-device
guarantee already comes from the forbidden checks. Loosened to require
--foreground anywhere on the open line; foreground-attach-single-sim now
scores 54/54 across both runners.
* fix: close workflow help conformance gaps
* feat(cli): compact workflow help card + version header
Shrinks the per-task agent protocol tax of the help/skill surface.
`agent-device help workflow` drops from 41025 to 8466 bytes (-79%) by
moving depth into new `help scripting` (save-script, secret-safe
fills, batch JSON, replay divergence/repair) and `help gestures`
(multi-touch shapes/quirks) topics, and folding a few paragraphs into
topics that already owned the subject (help debugging,
help physical-device, help validate). Content is moved, not deleted.
Every `help <topic>` first line is now `agent-device <version> —
<topic>`, so the skill router reads the CLI version off the mandatory
first help read instead of a separate `agent-device --version` call.
SKILL.md is updated to do that and stays a thin router otherwise.
The compact card also gains two terse behavioral rules: chain
confident consecutive steps with `&&` (falling back to one command at
a time when uncertain), and confirm the requested end state is
actually visible on screen before declaring a task done.
help-conformance-bench (22 cases x 2 runners) improves after the
change: 29/44 -> 32/44 passing checks.
* fix(cli): review follow-ups on the compact workflow card (#1663)
Three fixes from PR review:
- Extend the help-conformance plan validator to split a command line
on unquoted && and validate each chained segment independently, so
a plan that follows the workflow card's "chain confident consecutive
steps with &&" guidance is accepted instead of rejected as one
shell-projection violation. && inside a quoted selector value (e.g.
label="A && B") is not a chain boundary and does not split. Adds
unit tests for the splitter and a chains-confident-consecutive-
settle-steps conformance case. batch stays out of this: it is
deliberately stop-only.
- Replace the literal @ref placeholder the compact card used in its
own "snapshot -s @ref" example with a concrete ref
(snapshot -s @e12 (the current concrete ref)), matching the same
card's rule against placeholder targets. Reverts the test to demand
the concrete shape.
- Give help scripting and help gestures real conformance cases
instead of waivers: a secret-safe recorded-fill + publish case, and
an Android transform-then-verify case whose exact verification text
only appears in the gestures topic. Removes both waivers.
help-conformance-bench (25 cases x 2 runners, repeat=1) after these
fixes: two full runs landed at 32/50 and 33/50. That is on par with
the pre-change baseline (29/44) once the topic-untouched cases'
run-to-run swings are accounted for (confirmed noise: one case with
zero exposure to any change here flipped 10/10 -> 1/10 on a runner API
error, and another swung across all three post-fix runs). The new
scripting case now passes 8/8 for both runners; the new chaining case
correctly reports the model's choice not to chain as a soft signal,
not a validator failure.
* fix(cli): update session.test.ts help pointer for moved script-authoring content
* fix(cli): reject empty && chain operands in the plan validator (#1663)
splitOnUnquotedAnd() previously trimmed and filtered out empty
segments, so a plan with a leading (`&& press ...`), trailing
(`press ... &&`), or doubled (`a && && b`) operator passed
validPlanCommands even though a real shell rejects all three as a
syntax error. The validator would bless a plan that fails at
execution.
Empty segments are now surfaced as an `empty-chain-operand` issue
instead of being silently dropped. The quoted-&& non-split behavior
(label="A && B") is unchanged, and a normal single command with no
chain still parses identically to before.
Adds regression tests for all three empty-operand shapes plus the
quoted-&& case.
Michał Pierzchała