Resolve the runner conflict by retaining the neutral working directory and closing child stdin. Preserve the contributor commit and both regression tests.
Validation: 43 unit tests passed; eval catalog valid; real subprocess smoke check passed; both regressions reproduced against unpatched main. No paid provider calls.
Owner-directed amendment prepared with Codex (GPT-6).
The claude runner is invoked without a stdin redirect, so the child inherits
the parent's stdin. In a non-interactive parent the CLI waits 3s per call and
warns; worse, when that stdin carries data the CLI reads it and appends it to
the prompt. One observed call billed 155,004 cache-creation input tokens for a
4-token prompt (USD 1.58 against a USD 0.03 median), which is silent unbounded
cost in a harness whose budget flag is its only guard.
The same warning can land after the JSON document on stdout, and
_parse_response used json.loads on the whole buffer, aborting the run with
"Extra data: line 2 column 1" and discarding the responses still in flight.
Close the child's stdin and decode only the first JSON document.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`score` aggregates rows a human already wrote, so running the harness end to end
meant hand-scoring every response across five dimensions -- 420 numbers for the
documented 3-trial pass. That is the step that kept the harness unrun.
scripts/judge.py grades a whole `(case, trial)` group in one call, so conditions
are compared against each other rather than scored in isolation. Blinding is
structural rather than a convention the grader is asked to honour: conditions
are relabelled A/B/C and the label order is permuted per group from a digest of
the group key, so position carries no signal and a resumed run reproduces the
labels it used before. Over the 42 groups of a 3-trial pass the candidate landed
in each position 19/23 times.
Only the region of rubric.md between the new judge:begin/judge:end markers
reaches the grader. The release-gate rules below them name both conditions, so
sending the whole document to a blind grader would hand over the exact
vocabulary the blinding exists to hide -- caught by the end-to-end test, after
every unit test had passed on a prompt that was still leaking.
The prompt goes in on stdin. Appending it as a trailing argument let the claude
runner's `--tools ""` consume it as that option's value; generation only escapes
this because `--max-budget-usd` happens to sit in between. Stdin also removes an
argv ceiling that matters when prompts embed whole responses.
Graders occasionally drop a field or wrap JSON in prose. Verdicts are validated
with the offending case named, retried on the same budget as a failed process,
and if still unusable the group is skipped, reported on stderr, and the exit
code is non-zero -- one bad reply should not discard the groups already written
or the ones queued behind it. Groups missing a condition are reported the same
way rather than dropped, since conditions are only comparable when judged on
identical rows.
Stdlib only, matching the existing scripts. 18 tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two ways the harness graded something other than what it claimed to. Both
surfaced on the first real run; neither is visible without executing it.
Working directory: the runner subprocess inherited `cwd=ROOT`, so an agent CLI
adopted this checkout as project context and answered prompts by inspecting the
harness instead of the task. One baseline response opened by identifying
`scripts/run_evals.py` and emitted tool-call syntax for tools it did not have.
Over a 14-case pass it hit 8 baseline responses and 2 candidate ones -- the
asymmetry favours whichever condition discourages exploration, so it lands
directly on the number under test. The runners already pass `--setting-sources ""`
for exactly this reason; the working directory was an unguarded channel for the
same class of leak. Both conditions now run in an empty scratch directory.
Frontmatter: `_condition_prompt` injected SKILL.md verbatim, so the candidate
prompt carried `disable-model-invocation: true` and the `hermes` metadata block.
hooks/always-on.sh strips that frontmatter before injecting the ruleset, so the
eval was grading text that never reaches a user. `_strip_frontmatter` mirrors
the hook's awk -- byte-identical output on the current SKILL.md -- and leaves a
file alone if it opens a block without closing it, rather than emptying it.
Tests cover both: the working-directory one uses a stub runner that reports its
own `pwd`, and fails against the previous behaviour.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The "Runner did not report dollar cost" rejection fired after the subprocess
had already run — but for the text and codex-jsonl response formats
_parse_response returns cost None unconditionally, so the outcome is fully
determined by the runner config. First run with the shipped codex runner and
no --allow-unmetered paid for one full API call, then discarded the response;
the rerun paid for it again.
Pre-flight the check from response_format before the loop. The per-call check
stays as a backstop for claude-json responses that omit total_cost_usd, which
is only knowable at runtime.
The message interpolated the condition name and told the user to pass
--candidate-skill / --comparator-skill; the parser only defines
--condition-skill, so following the error verbatim produced
"unrecognized arguments".
An unknown id filtered out every case, so the run wrote nothing and exited 0
with "Reported cost: $0.0000" — indistinguishable from a resumed run that had
nothing left to do. The operator believes the condition was measured, and
score then compares whatever happens to be in the results file.
Validate --case against the loaded catalog before the loop.
summarize_scores averaged each condition independently, so a baseline judged
on two hard cases and a candidate judged on one easy case returned
"passed": true with no warning. evals/README.md already requires identical
cases across conditions; nothing enforced it, and a partial candidate run or a
skipped judge row silently produced a confident verdict.
Compare the (case_id, trial) multiset of every condition against baseline and
raise with the concrete difference; reject duplicate rows within a condition,
which otherwise double-count into the average.
14 cases across direct answers, agent autonomy, debugging, explanations,
destructive actions, ambiguity, progress, output contracts, planning,
and medical boundaries. Weighted correctness/autonomy/actionability/
safety/concision rubric, baseline/candidate/comparator matrix, budget
ceiling, blind scoring, resumability. Trimmed to the evaluation
foundation; the repo-governance files from the original branch were
split out. Groundwork for #4.
ayghri