* feat(bench): renderer-pinned samples, topic-coverage gate, error quizzes; trim skillgym to agentic checks
The help conformance bench's quoted CLI output is now sourced from
scripts/help-conformance-sample-outputs.mjs, and every sample is rebuilt
through the real production renderers (settle output formatters,
printHumanError, formatSnapshotText, refMutationAdmissionResponse) by
scripts/__tests__/help-conformance-sample-outputs.test.ts — a rendering or
message change fails deterministically instead of leaving the bench grading
against output the CLI no longer prints. This retires the fabricated
recoverable-failure envelope (production never throws a textual settle
timeout; that case is replaced by a real DEVICE_IN_USE recovery quiz).
Bench cases move to scripts/help-conformance-cases.mjs and are enumerated
against the help-topic registry: helpTopicIds() is exported from cli-help,
and scripts/__tests__/help-conformance-topic-coverage.test.ts fails when a
help topic has neither a bench case nor an explicit waiver. New case
families: error-envelope recovery quizzes (device-in-use, stale pinned ref,
ambiguous find match, app-not-installed) pinned to real error text, topic
coverage for tv/web/react-native/debugging/workflow, and a metamorphic twin
of the settled-diff quiz.
The skillgym smoke suite shrinks from 119 cases to the 5 that measure what
only an agentic runner can show: skill routing plus output interpretation
with a proven local CLI help probe (local-cli-help-policy). Its embedded
samples now import the same pinned constants, replacing hand-transcribed
output that had already drifted from the renderer. Knowledge checks belong
to the bench; live fixture behavior belongs to the iOS simulator e2e suite.
* review: drive error samples through the real producers; enforce local-help on the routing smoke
The DEVICE_IN_USE, AMBIGUOUS_MATCH, and APP_NOT_INSTALLED parity tests no
longer hand-author the producer message before rendering: each drives the
actual producer — buildDeviceInUseBySessionError (extracted in
session-open.ts and called by the handler), buildAmbiguousMatchError (now
exported from find.ts), and buildAppNotInstalledError (extracted in
app-resolution.ts and thrown by the resolver). Because each factory is
exported from its producer file and called by the production path, dropping
the production call would make it test-only and fail
check:production-exports — the wiring is gate-enforced, not conventional.
open-and-snapshot now sets requireLocalCliHelp and
allowOnlyLocalCliHelpCommands, so the 'skill plus local help' claim is
observed rather than assumed; without them the case can pass on model prior
alone.
Michał Pierzchała