* feat(bench): renderer-pinned samples, topic-coverage gate, error quizzes; trim skillgym to agentic checks
The help conformance bench's quoted CLI output is now sourced from
scripts/help-conformance-sample-outputs.mjs, and every sample is rebuilt
through the real production renderers (settle output formatters,
printHumanError, formatSnapshotText, refMutationAdmissionResponse) by
scripts/__tests__/help-conformance-sample-outputs.test.ts — a rendering or
message change fails deterministically instead of leaving the bench grading
against output the CLI no longer prints. This retires the fabricated
recoverable-failure envelope (production never throws a textual settle
timeout; that case is replaced by a real DEVICE_IN_USE recovery quiz).
Bench cases move to scripts/help-conformance-cases.mjs and are enumerated
against the help-topic registry: helpTopicIds() is exported from cli-help,
and scripts/__tests__/help-conformance-topic-coverage.test.ts fails when a
help topic has neither a bench case nor an explicit waiver. New case
families: error-envelope recovery quizzes (device-in-use, stale pinned ref,
ambiguous find match, app-not-installed) pinned to real error text, topic
coverage for tv/web/react-native/debugging/workflow, and a metamorphic twin
of the settled-diff quiz.
The skillgym smoke suite shrinks from 119 cases to the 5 that measure what
only an agentic runner can show: skill routing plus output interpretation
with a proven local CLI help probe (local-cli-help-policy). Its embedded
samples now import the same pinned constants, replacing hand-transcribed
output that had already drifted from the renderer. Knowledge checks belong
to the bench; live fixture behavior belongs to the iOS simulator e2e suite.
* review: drive error samples through the real producers; enforce local-help on the routing smoke
The DEVICE_IN_USE, AMBIGUOUS_MATCH, and APP_NOT_INSTALLED parity tests no
longer hand-author the producer message before rendering: each drives the
actual producer — buildDeviceInUseBySessionError (extracted in
session-open.ts and called by the handler), buildAmbiguousMatchError (now
exported from find.ts), and buildAppNotInstalledError (extracted in
app-resolution.ts and thrown by the resolver). Because each factory is
exported from its producer file and called by the production path, dropping
the production call would make it test-only and fail
check:production-exports — the wiring is gate-enforced, not conventional.
open-and-snapshot now sets requireLocalCliHelp and
allowOnlyLocalCliHelpCommands, so the 'skill plus local help' claim is
observed rather than assumed; without them the case can pass on model prior
alone.
* test(output-economy): add routine-workflow output-behavior oracle
Add a deterministic routine-workflow measurement (#1180) that pairs
response bytes with follow-up behavior: fallback-observation count,
retry count, and whether an actionable failure preserves the session.
Refs chain across one recorded checkout session and counts derive from
the real formatters, so dropping settled-diff refs, the unchanged-
interactive tail, or a recovery handle fails the suite. Adds a matching
non-gating help-conformance next-command case. Response defaults
unchanged.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(output-economy): make routine-workflow ref-surfacing depend on rendered output
Address review on #1190:
- Drop the raw mutation-confirm/failure MCP data samples that leaked e4/e5
and mislabeled the surface; e4/e5 now surface only from the rendered CLI
settled-diff, so dropping added refs genuinely raises the fallback count.
- Track the failure once as its projection-invariant normalized payload
(workflow.failure.shared.json) instead of duplicate cli/mcp raw copies.
- Reuse the shared REF_TOKEN_PATTERN from economy-metrics.ts.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(output-economy): reuse shared fixtures in routine-workflow oracle
Address review finding #2 on #1190: chain the workflow onto the shared
per-surface fixtures instead of copy-pasting them.
- orient/recheck reuse SNAPSHOT_RESULT + SNAPSHOT_DAEMON_RESULT; the
first mutation reuses SETTLE_ADDED_REF_RESULT, so session identity and
ref generations come from ./fixtures.ts and the two suites cannot drift.
- Only the genuinely workflow-specific pieces remain local: the unchanged
recheck, a tail retargeted onto a settled-diff ref (SETTLE_TAIL_RESULT
taps an unsurfaced @e6 and can't chain), the in-session timeout failure,
and its recovered retry. routine-workflow.ts drops ~110 LOC.
- Rendered-output ref guard preserved: @e5 (settled diff) and @e7 (tail)
surface only from formatter output; recovery semantics unchanged.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Michał Pierzchała <thymikee@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat: fix codex runner, add override-doc grading, port skillgym quiz cases
The 2026-07-09 evaluation of scripts/help-conformance-bench.mjs found it
structurally right but broken for the codex runner (two bugs), thin on
coverage (4 cases), sequential, and unable to grade a draft help rewrite
without a rebuild.
- Fix runCodex: (1) codex exec reads stdin until EOF when not attached to
a TTY, and execFile never closes the child's stdin, so every codex call
hung until RUN_TIMEOUT_MS with empty output — close stdin right after
spawn. (2) `-o outFile` writes the same final JSON that codex also
prints to stdout, so concatenating both produced two back-to-back JSON
objects that broke every JSON.parse candidate and silently zeroed
extractCommands() — prefer the clean -o payload, fall back to stdout
only when it's empty.
- Add `--override-doc <topicId>=<path>` (repeatable): loads a topic's
text from a file instead of shelling out to
`node bin/agent-device.mjs help <topic>`, so a draft help rewrite can
be A/B graded with zero rebuild.
- Port three cases from
test/skillgym/suites/agent-device-smoke-suite.ts
(settle-diff-is-observation, sample-output-settled-diff-next-target,
sample-output-not-settled-needs-observe) as self-contained
"next-command quiz" cases, generalizing the scorer to support regex
matchers/forbidden patterns alongside the existing named expectations.
Fixture output text matches the CURRENT settle rendering in
src/commands/interaction/output.ts, including the "unchanged
interactive (N):" tail added by #1167/#1172.
- Parallelize the runner x case matrix with a concurrency cap
(HELP_BENCH_CONCURRENCY, default 4); results still print in the
original matrix order.
- Extend test/skillgym/README.md's existing pointer to this bench with
the new flags.
Validated with real LLM calls (both runners, all 7 cases, 14 calls,
~$0.25 total): 13/14 pass; the one fail (claude-haiku-4-5 on
dogfood-mode) is a genuine model miss (returned an empty command plan
asking for the app name instead of committing to a generic plan), not a
bench bug. `--override-doc` demonstrated live: stripping the dogfood
doc's evidence-command examples regresses codex:gpt-5.4-mini from 3/3 to
2/3 on the same case, showing the flag both loads and changes grading.
* fix: apply live-doc post-processing to --override-doc, fail fast on bad overrides
Review findings on the initial version (all reproduced):
- HIGH: an override for the --help:first30 doc id skipped the live path's
firstLines(text, 30) cap, so a 49-line draft leaked lines 31-49 into the
prompt — grading content a live run never shows, on the doc id every case
uses. loadDoc now splits source (live shell-out vs override file) from
post-processing, and the post-processing applies to both, so an override
differs ONLY in where the text comes from.
- MEDIUM: an --override-doc topic id no selected case uses was silently
ignored (exit 0, real doc graded). Now fails fast listing the valid doc
ids for the selection.
- LOW: a missing override file threw a raw ENOENT stack trace; expected
failures now print one clean Error line. Added --help usage text that
documents last-wins semantics for repeated same-topic overrides and the
post-processing parity.
Guard tests (scripts/__tests__/help-conformance-bench.test.ts, wired into
the unit-core vitest project by explicit path): a 49-line fixture whose
prompt must keep line 30 and drop line 31, unknown-topic fail-fast with
valid ids listed, clean no-stack error for a missing file, and last-wins
for repeated overrides. All spawn the script in --dry-run with every
required doc overridden, so they need no LLM calls and no built CLI.
Live re-validation: a 33-line override of --help:first30 whose lines
31-33 instruct the model to emit a sentinel command; neither
claude-haiku-4-5 nor codex:gpt-5.4-mini emitted it (both scored 4/4,
matching the live-doc baseline), proving the cap applies end-to-end.
Michał Pierzchała