* feat: fix codex runner, add override-doc grading, port skillgym quiz cases
The 2026-07-09 evaluation of scripts/help-conformance-bench.mjs found it
structurally right but broken for the codex runner (two bugs), thin on
coverage (4 cases), sequential, and unable to grade a draft help rewrite
without a rebuild.
- Fix runCodex: (1) codex exec reads stdin until EOF when not attached to
a TTY, and execFile never closes the child's stdin, so every codex call
hung until RUN_TIMEOUT_MS with empty output — close stdin right after
spawn. (2) `-o outFile` writes the same final JSON that codex also
prints to stdout, so concatenating both produced two back-to-back JSON
objects that broke every JSON.parse candidate and silently zeroed
extractCommands() — prefer the clean -o payload, fall back to stdout
only when it's empty.
- Add `--override-doc <topicId>=<path>` (repeatable): loads a topic's
text from a file instead of shelling out to
`node bin/agent-device.mjs help <topic>`, so a draft help rewrite can
be A/B graded with zero rebuild.
- Port three cases from
test/skillgym/suites/agent-device-smoke-suite.ts
(settle-diff-is-observation, sample-output-settled-diff-next-target,
sample-output-not-settled-needs-observe) as self-contained
"next-command quiz" cases, generalizing the scorer to support regex
matchers/forbidden patterns alongside the existing named expectations.
Fixture output text matches the CURRENT settle rendering in
src/commands/interaction/output.ts, including the "unchanged
interactive (N):" tail added by #1167/#1172.
- Parallelize the runner x case matrix with a concurrency cap
(HELP_BENCH_CONCURRENCY, default 4); results still print in the
original matrix order.
- Extend test/skillgym/README.md's existing pointer to this bench with
the new flags.
Validated with real LLM calls (both runners, all 7 cases, 14 calls,
~$0.25 total): 13/14 pass; the one fail (claude-haiku-4-5 on
dogfood-mode) is a genuine model miss (returned an empty command plan
asking for the app name instead of committing to a generic plan), not a
bench bug. `--override-doc` demonstrated live: stripping the dogfood
doc's evidence-command examples regresses codex:gpt-5.4-mini from 3/3 to
2/3 on the same case, showing the flag both loads and changes grading.
* fix: apply live-doc post-processing to --override-doc, fail fast on bad overrides
Review findings on the initial version (all reproduced):
- HIGH: an override for the --help:first30 doc id skipped the live path's
firstLines(text, 30) cap, so a 49-line draft leaked lines 31-49 into the
prompt — grading content a live run never shows, on the doc id every case
uses. loadDoc now splits source (live shell-out vs override file) from
post-processing, and the post-processing applies to both, so an override
differs ONLY in where the text comes from.
- MEDIUM: an --override-doc topic id no selected case uses was silently
ignored (exit 0, real doc graded). Now fails fast listing the valid doc
ids for the selection.
- LOW: a missing override file threw a raw ENOENT stack trace; expected
failures now print one clean Error line. Added --help usage text that
documents last-wins semantics for repeated same-topic overrides and the
post-processing parity.
Guard tests (scripts/__tests__/help-conformance-bench.test.ts, wired into
the unit-core vitest project by explicit path): a 49-line fixture whose
prompt must keep line 30 and drop line 31, unknown-topic fail-fast with
valid ids listed, clean no-stack error for a missing file, and last-wins
for repeated overrides. All spawn the script in --dry-run with every
required doc overridden, so they need no LLM calls and no built CLI.
Live re-validation: a 33-line override of --help:first30 whose lines
31-33 instruct the model to emit a sentinel command; neither
claude-haiku-4-5 nor codex:gpt-5.4-mini emitted it (both scored 4/4,
matching the live-doc baseline), proving the cap applies end-to-end.
Michał Pierzchała