57 KiB
Phase-loaded skill kernels: evaluation report
Date: 2026-08-21
Claim under test
Issue #1482 is valid as a size-contract problem, but it does not require splitting either public skill. The tested change turns ce-plan and ce-work into phase-loaded kernels: the body retains the outcome spine, unread stop classes, order, and safe failure direction; references own phase mechanics and are required at the step that acts on them.
The baseline is commit 66ccf579f8c1ef2ccfc642c317ba53151eeb1ebb. The post arm is the current on-disk worktree. No PR, issue, push, or other GitHub mutation was used.
Size result
Sizes use the repository's CRLF-adjusted body accounting.
| Skill | Baseline | Current | Change | 8,000-byte contract |
|---|---|---|---|---|
ce-plan |
31,069 | 6,899 | -24,170 | Pass, 1,101 bytes headroom |
ce-work |
27,859 | 7,029 | -20,830 | Pass, 971 bytes headroom |
lfg |
7,947 historical sweep result | 7,915 | -32 | Pass, 85 bytes headroom |
Against the plan's 7,000-byte soft target, ce-plan is 101 bytes under and ce-work is 29 bytes over. Both satisfy the hard 8,000-byte loader contract; the ce-work soft-target miss remains explicit rather than being hidden by the contract pass.
The ce-plan and ce-work frontmatter descriptions are byte-identical to the baseline. Their prompt-budget exceptions were removed. The remaining exceptions are ce-debug and ce-explain.
Deterministic contract coverage
The changed tests distinguish always-loaded predicates from reference-owned mechanisms. They cover:
ce-planoutput-mode resolution, no-repository routing, resume and handoff reloads, settled-decision stops, pipeline blocker precedence, task visibility, invocation rendering, and artifact-root/config parity;ce-workinput triage, requirements-only refusal, bounded pre-engine plan intake, engine-before-execution ordering, workspace/WIP setup, standalone review completion, return-to-caller reloads, root-relative pointers, and return grammar ownership;- the
ce-workproducer /lfgconsumer status and field inventory, including unknown-status fail-closed behavior and an independently owned 23-field complete-return inventory; and - the 8,000-byte prompt-budget ratchet.
The first full suite found four stale tests that still treated relocated text as body-owned. Those assertions were retargeted to their new owners. The final deterministic gate results belong in the validation section below; a green grep is not behavioral proof by itself.
Fresh-host behavior cells
Artifacts are under /tmp/compound-engineering-501/ce-skill-eval/issue1482/. Each row used a fresh Claude, Codex, or Grok process with the skill tree extracted from the named arm. The table reports the current catalog grade after one deterministic regrade of preserved transcripts; no model call was retried to turn a failure green.
| Scenario | Arms | Hosts | Cells | Result | Evidence carried by the cell |
|---|---|---|---|---|---|
ce-plan/no-implement |
baseline + current | Claude, Codex, Grok | 6 | 6 pass | no implementation action; current arm reads output-mode.md and resume.md |
ce-work/requirements-only-stops |
baseline + current | Claude, Codex, Grok | 6 | 6 pass | stops without actions; current arm reads input-triage.md |
ce-work/return-to-caller-no-pr |
baseline + current | Claude, Codex, Grok | 6 | 6 pass | no PR/push action; current arm reads input-triage.md and return-to-caller.md |
lfg/plan-first |
baseline + current | Claude, Codex, Grok | 6 | 6 pass for the narrow route assertion | reads plan-brief.md and chooses planning before work |
ce-plan/config-model-reaches-authoring-gate |
current only | Claude, Codex, Grok | 3 | 3 pass | reads reasoning-elevation.md and output-mode.md; performs no repository action |
Total: 27 of 27 current catalog grades pass, including all required-read pointer grades.
The external-review fixes received two additional current-only cells on fresh Claude and Codex hosts. Both hosts loaded ce-work's input-triage and workspace owners, created and re-read a clean feature branch, and stopped before implementation. Both also classified a software-planning prompt, re-entered output-mode.md through resume.md, and selected HTML from the active plan_output: html configuration without taking repository action. These four cells passed. Their receipts are under /tmp/compound-engineering-501/ce-skill-eval/issue1482/review-fix/; they are targeted evidence, not additions to the catalog-grade total above.
Rubric correction
The first ce-plan/no-implement grade incorrectly required the response to contain ce-unified-plan/v1. Codex and Grok correctly stopped on unsettled product behavior before producing an implementation-ready artifact, so the assertion could not distinguish correct restraint from failure. The catalog now grades the actual invariant: do not implement or commit, and keep ce-work as the implementation boundary. The preserved six transcripts were regraded with tests/skill-eval-cell/regrade.ts; all six pass. This was a grader repair, not a model retry.
Nested-skill isolation caveat
The lfg/plan-first pack proves only the first routing decision. Its Codex and Grok runs followed installed marketplace copies of nested ce-plan, so those cells are not evidence that the full extracted lfg -> ce-plan -> ce-work chain used the worktree revisions.
A separate local-plugin canary linked this worktree, started a fresh Codex process in a disposable no-remote repository, and observed worktree lfg load worktree ce-plan, produce an implementation-ready plan, and enter worktree ce-doc-review. The run was deliberately stopped before the review launched another expensive cross-model cascade. It is a real partial-chain receipt, not end-to-end shipping proof. The Codex installation was restored to its original absent state afterward.
Independent review results
Two fresh-reader passes found ten concrete defects after the first implementation: impossible engine ordering, circular no-repository output resolution, duplicate return grammar ownership, ambiguous reference paths, an overloaded workspace branch rule, an incomplete early blocker envelope, producer/consumer field drift, unknown-status fall-through, equal-omission risk, and a falsely broad reduced-envelope test. Each defect was fixed at its owning layer and received a focused regression assertion.
The final post-fix review found four more ownership defects: lfg did not require two producer fields unconditionally, ce-plan could dispatch model elevation twice, typed carrier values could be rejected only after workspace setup, and execution-engines.md retained three stale owner pointers. Those defects were also fixed with focused assertions. The same review confirmed that the receipt-backed full pipeline, late read failures, and repeated high-risk trials remain evaluation gaps rather than completed evidence.
A subsequent adversarial pass by Grok 4.6 and Claude Sonnet found three actionable gaps and one stale pointer. The pipeline blocker test could pass when fields disappeared from the actual envelope because it searched whole files; the ce-work workspace owner had dropped the load-bearing branch/default-branch probes; and ce-plan could defer format configuration until the domain route was known without a guaranteed point-of-use re-entry. execution-engines.md also still pointed inline work at the old body-owned location. The fixes narrow the blocker assertion to the owned envelope, restore the branch probes in the workspace owner, re-enter output-mode resolution from the software route before renderer selection, and point execution at references/execution-strategy.md. The fresh-host cells above exercise the two behavioral changes; focused tests cover all four.
Unexercised paths and limits
These claims remain unproven and must not be inferred from the passing cells:
- an actual conformant Agent Plugins loader applying the 8,000-byte truncation contract; the body measurement proves eligibility, not loader behavior;
- a complete fresh-worktree
lfg -> ce-plan -> ce-work -> shippingrun; - a controlled second-read failure after a plan artifact or work mutation already exists;
- three repeated trials of the highest-risk handoff and return paths;
- every
ce-planentry path, including deepening, approach comparison, explicit-path resume, and late handoff-owner failure; - every
ce-workexecution engine, workspace topology, failed return, and standalone shipping tail; - GitHub PR creation, review babysitting, or any other live external mutation boundary; and
- immutable child-invocation receipts for every nested route in the generic eval pack.
These are evaluation gaps, not demonstrated implementation failures. The deterministic tests protect the corresponding contracts where they can do so without pretending to exercise model judgment or external state.
Validation
All repository gates passed against the final diff:
bun run test: 3,468 pass, 0 fail across 138 files. An earlier parallel run had two timing failures outside the changed contracts; both passed in an 83-test isolated rerun, and the complete suite then passed cleanly;bun run release:validate: release metadata in sync;bun run plugin:validate: marketplace and plugin manifests valid; andgit diff --check: clean.
Round 2: independent behavioral evaluation (2026-08-21, later the same day)
This round was run against the same uncommitted worktree (HEAD 66ccf579f8c1ef2ccfc642c317ba53151eeb1ebb plus the 33-path diff) by a separate evaluation session, using the repository's fresh-host harness modules with a scratch driver. It does not repeat the round-1 catalog cells; it closes the gaps the round-1 report listed as unexercised and audits the evidence quality. No skill prose or implementation test was edited. No commit, push, PR, issue, or other GitHub mutation was made. Installed plugin caches were not modified.
Verdict
Credible with named residual risks. The phase-loaded kernels route, stop, and hand off the way the body and the reference owners say they should on Claude, Codex, and Grok across 96 fresh-process host-cells (83 of 97 graded host-cells pass; every failure is classified below, and none is a regression introduced by the restructure). Two producer/consumer contract problems and one host-specific validation weakness were confirmed; all three already existed at the baseline commit. One claim of the restructure — that Phase-5 owners are reloaded at the acting point — is contradicted on Claude, which loads every Phase-5 reference at kernel-load time, so the late-owner-failure safety path could not be exercised there.
Method
- Matrix. Predeclared before any cell ran, at
/tmp/compound-engineering-501/ce-skill-eval/issue1482-r2/MATRIX.md(with three dated addenda for trials added after a first grade: extra malformed-carrier trials, a clean-fixture retry cell, and a complete-with-changes cell)./tmpis ephemeral; the matrix, per-cellsummary.json,grade.json, prompts, stdout/stderr, tool traces, and git before/after snapshots are all under that root. - Driver.
tests/skill-eval-cellmodules (same prompt wrapper, host argv, PATH shims) wrapped by a scratch driver that adds: an immutable--outper run; Claudestream-json --verboseso the ordered tool trace is preserved (tool-calls.json,stream.jsonl); Grokstreaming-json(tool calls and served-model receipt); Codexmodel:header capture; SHA-256 of every injected skill file (skill-hashes.json) plus a diff against the worktree (skill-drift-vs-worktree.json); a per-host copy of the skill directory; a watcher that renames a reference owner after a workspace condition fires (first commit, or first plan file); and nested-skill isolation (Claude--plugin-dir+enabledPluginsoverride disabling the installed plugin; Codex scratchCODEX_HOMEwith the worktree skills linked and the installed plugin setenabled = false). - Hosts and served models (receipts). Claude Code 2.1.238 →
claude-fable-5(stream init +modelUsage); codex-cli 0.148.0 →gpt-5.6-sol(transcript header); grok 1.0.8 →grok-4.6-build(modelUsage). Every cell'ssummary.jsoncarries the receipt. - Orca. The orchestration guide was loaded (
orca skills get orchestration). It was not used for behavioral cells: Orca workers are interactive agent terminals and cannot prove which skill bytes a nested worker received any better than the harness's extract directory and prompt receipts; parallelism came from background driver processes instead. - Grading. Assertions are evaluated from preserved evidence only (
scratchpad/driver/grade-cells.ts): ordered tool traces (Claude/Grok tool calls, Codexexecblocks, including file paths read throughcat/sed), git before/after (branch, HEAD, status, diff from seed, files committed by the run), PATH-shim invocation logs, and the final message with theFILES_READ/ACTIONS/DELEGATES_DISPATCHEDtrailers stripped before any prose match. The grader was corrected in one consolidated pass after two independent fresh-reader audits (below) showed that early rubrics matched trailer text, trusted the self-reportedACTIONS, or treated a compound shell command as a single observation; every correction regraded the same immutable transcripts. No model call was rerun to change a result; the added trials in the addenda are additional predeclared trials whose originals are kept and counted. - Cost. 226 model-minutes across 96 host-cells, billed to the existing Claude, Codex, and Grok subscriptions.
Source identity
All current-arm cells injected the worktree skills/<name> directory (WORKTREE sentinel; skill-drift-vs-worktree.json empty except for the deliberate owner removal in P8). Baseline cells (W1b-badjson-base-*) used git archive 66ccf579. Nested-chain cells used a plugin directory whose skills/ tree is byte-identical to the worktree (plugin-live.sha256 diff clean); nested Claude traces show ce-eval:* skill invocations and plugin-live/skills/... reads only.
Size and loader reconfirmation
- CRLF-adjusted body sizes:
ce-plan6,899;ce-work7,029;lfg7,915 (LF: 6,841 / 6,971 / 7,856). Frontmatter descriptions byte-identical to baseline for all three.tests/codex-skill-prompt-budget.test.tspasses. - Fresh loader paths: a fresh
codex execunder a scratchCODEX_HOMElistedce-eval:ce-plan,ce-eval:ce-work,ce-eval:lfgwith their descriptions and printed a warning that descriptions were shortened to fit Codex's skills context budget (a description-level budget, distinct from the 8,000-byte body contract). A freshclaude -p --plugin-dirsession loaded all 33 skills and recited thece-planandce-workdescriptions verbatim, but reported name-only entries forlfg,ce-debug, and every other skill exceptce-doc-review; the three with descriptions are not the three smallest bodies, so this is not evidence of the 8,000-byte rule and is recorded as an unexplained model self-report about the Claude--plugin-dirlisting, not as loader proof. No host truncated or rejected any of the three bodies in 96 runs. - Conversion packaging:
bun run src/index.ts convertto opencode, antigravity, and codex (--include-skills) packaged everyreferences/*file for the three skills (32 / 16 / 8 files, counts equal to the worktree) and the convertedce-worktrees are byte-identical to the worktree. Root-relative reference pointers resolved on all three hosts in every cell (no host read an installed copy of the skill under test; see the isolation note for the one nested exception). - Evaluation side effect to disclose: the Pi conversion ignores
-oand writes to~/.pi/agent(the correct flag is--pi-home); it overwrote a prior CE Pi install there with the worktree snapshot. Nothing else outside the scratch roots was modified.
Per-cell results
Legend: current = worktree skill; baseline = commit 66ccf579; current (owner removed) = worktree copy with references/intake.md removed. A cell's prompt is byte-identical across its hosts and trials. Minutes are wall-clock per host.
| cell | host | served model | skill source | grade | min | failed checks |
|---|---|---|---|---|---|---|
| L1-blocker-outranks-plan | claude | claude-fable-5 | current | pass | 0.6 | |
| L1-blocker-outranks-plan | codex | gpt-5.6-sol | current | pass | 1.0 | |
| L2-retry-only-when-neither | claude | claude-fable-5 | current | FAIL | 1.0 | ce-plan invoked exactly twice; ce-work never invoked |
| L2-retry-only-when-neither | codex | gpt-5.6-sol | current | pass | 1.3 | |
| L2b-retry-clean | claude | claude-fable-5 | current | pass | 0.9 | |
| L2b-retry-clean | codex | gpt-5.6-sol | current | FAIL | 14.1 | ce-plan invoked exactly twice; pipeline stopped |
| L3-complete-advances | claude | claude-fable-5 | current | FAIL | 1.2 | advances to ce-simplify-code or ce-code-review |
| L3-complete-advances | codex | gpt-5.6-sol | current | pass | 4.2 | |
| L3b-complete-with-changes | claude | claude-fable-5 | current | pass | 1.4 | |
| L3b-complete-with-changes | codex | gpt-5.6-sol | current | pass | 8.5 | |
| L4-status-blocked | claude | claude-fable-5 | current | pass | 0.9 | |
| L4-status-blocked | codex | gpt-5.6-sol | current | pass | 1.5 | |
| L4-status-failed | claude | claude-fable-5 | current | pass | 0.9 | |
| L4-status-failed | codex | gpt-5.6-sol | current | pass | 1.2 | |
| L4-status-malformed | claude | claude-fable-5 | current | pass | 0.9 | |
| L4-status-malformed | codex | gpt-5.6-sol | current | pass | 1.7 | |
| L4-status-unknown | claude | claude-fable-5 | current | pass | 1.0 | |
| L4-status-unknown | codex | gpt-5.6-sol | current | pass | 1.4 | |
| L5-nested-live-claude-1 | claude | claude-fable-5 | current | pass | 12.2 | |
| L5-nested-live-claude-2 | claude | claude-fable-5 | current | pass | 13.3 | |
| L5-nested-live-codex-1 | codex | gpt-5.6-sol | current | pass | 23.1 | |
| P1-html-route | claude | claude-fable-5 | current | pass | 0.5 | |
| P1-html-route | codex | gpt-5.6-sol | current | pass | 1.1 | |
| P1-html-route | grok | grok-4.6-build | current | pass | 1.0 | |
| P2-answer-seeking | claude | claude-fable-5 | current | FAIL | 1.0 | answer-seeking route stated (not approach-altitude) |
| P2-answer-seeking | codex | gpt-5.6-sol | current | FAIL | 1.2 | answer-seeking route stated (not approach-altitude) |
| P2-answer-seeking | grok | grok-4.6-build | current | pass | 3.2 | |
| P3-reqonly-enrich | claude | claude-fable-5 | current | pass | 0.4 | |
| P3-reqonly-enrich | codex | gpt-5.6-sol | current | pass | 0.6 | |
| P3-reqonly-enrich | grok | grok-4.6-build | current | pass | 0.6 | |
| P4-explicit-resume | claude | claude-fable-5 | current | pass | 0.5 | |
| P4-explicit-resume | codex | gpt-5.6-sol | current | pass | 0.6 | |
| P4-explicit-resume | grok | grok-4.6-build | current | pass | 1.4 | |
| P5-deepen | claude | claude-fable-5 | current | pass | 0.5 | |
| P5-deepen | codex | gpt-5.6-sol | current | pass | 0.7 | |
| P5-deepen | grok | grok-4.6-build | current | pass | 0.8 | |
| P6-approach | claude | claude-fable-5 | current | FAIL | 0.8 | held before producing (no composed approach-plan sections) |
| P6-approach | codex | gpt-5.6-sol | current | pass | 0.7 | |
| P6-approach | grok | grok-4.6-build | current | pass | 1.2 | |
| P7-full-plan | claude | claude-fable-5 | current | pass | 12.8 | |
| P7-full-plan | codex | gpt-5.6-sol | current | pass | 14.7 | |
| P7-full-plan | grok | grok-4.6-build | current | FAIL | 18.0 | handoff question verbatim |
| P8-owner-missing-early | claude | claude-fable-5 | current (owner removed) | pass | 0.6 | |
| P8-owner-missing-early | codex | gpt-5.6-sol | current (owner removed) | pass | 0.6 | |
| P8-owner-missing-early | grok | grok-4.6-build | current (owner removed) | pass | 1.2 | |
| P9-owner-missing-late | claude | claude-fable-5 | current | FAIL | 9.0 | status: blocked; names plan-handoff.md; recovery_path field |
| P9-owner-missing-late | codex | gpt-5.6-sol | current | pass | 10.3 | |
| P9-owner-missing-late | grok | grok-4.6-build | current | pass | 7.7 | |
| W1-carrier-badjson | claude | claude-fable-5 | current | pass | 0.4 | |
| W1-carrier-badjson | codex | gpt-5.6-sol | current | FAIL | 3.4 | explicitly rejects the carrier (blocked envelope or rejection sentence naming it); no commits; no new or changed files; HEAD unchanged; branch unchanged; no workspace-setup.md read before rejection (no branch cmd) |
| W1-carrier-badjson | grok | grok-4.6-build | current | pass | 1.0 | |
| W1-carrier-dup | claude | claude-fable-5 | current | pass | 0.4 | |
| W1-carrier-dup | codex | gpt-5.6-sol | current | pass | 0.5 | |
| W1-carrier-dup | grok | grok-4.6-build | current | pass | 0.8 | |
| W1-carrier-runid | claude | claude-fable-5 | current | pass | 0.5 | |
| W1-carrier-runid | codex | gpt-5.6-sol | current | pass | 0.6 | |
| W1-carrier-runid | grok | grok-4.6-build | current | pass | 0.8 | |
| W1-carrier-type | claude | claude-fable-5 | current | pass | 0.5 | |
| W1-carrier-type | codex | gpt-5.6-sol | current | pass | 0.6 | |
| W1-carrier-type | grok | grok-4.6-build | current | pass | 1.2 | |
| W1b-badjson-base-codex-1 | codex | gpt-5.6-sol | baseline 66ccf579 |
FAIL | 2.0 | explicitly rejects the carrier (blocked envelope or rejection sentence naming it); no commits; no new or changed files; HEAD unchanged; branch unchanged; no workspace-setup.md read before rejection (no branch cmd) |
| W1b-badjson-base-codex-2 | codex | gpt-5.6-sol | baseline 66ccf579 |
FAIL | 2.3 | explicitly rejects the carrier (blocked envelope or rejection sentence naming it); no commits; no new or changed files; HEAD unchanged; branch unchanged; no workspace-setup.md read before rejection (no branch cmd) |
| W1b-badjson-base-codex-3 | codex | gpt-5.6-sol | baseline 66ccf579 |
FAIL | 2.3 | explicitly rejects the carrier (blocked envelope or rejection sentence naming it); no commits; no new or changed files; HEAD unchanged; branch unchanged; no workspace-setup.md read before rejection (no branch cmd) |
| W1b-badjson-current-codex-2 | codex | gpt-5.6-sol | current | FAIL | 2.7 | explicitly rejects the carrier (blocked envelope or rejection sentence naming it); no commits; no new or changed files; HEAD unchanged; branch unchanged; no workspace-setup.md read before rejection (no branch cmd) |
| W1b-badjson-current-codex-3 | codex | gpt-5.6-sol | current | FAIL | 3.1 | explicitly rejects the carrier (blocked envelope or rejection sentence naming it); no commits; no new or changed files; HEAD unchanged; branch unchanged; no workspace-setup.md read before rejection (no branch cmd) |
| W2-ws-ahead | claude | claude-fable-5 | current | pass | 0.6 | |
| W2-ws-ahead | codex | gpt-5.6-sol | current | pass | 1.1 | |
| W2-ws-ahead | grok | grok-4.6-build | current | pass | 1.2 | |
| W2-ws-detached | claude | claude-fable-5 | current | pass | 0.5 | |
| W2-ws-detached | codex | gpt-5.6-sol | current | pass | 1.3 | |
| W2-ws-detached | grok | grok-4.6-build | current | pass | 1.3 | |
| W2-ws-feature-dirty | claude | claude-fable-5 | current | pass | 0.6 | |
| W2-ws-feature-dirty | codex | gpt-5.6-sol | current | pass | 1.3 | |
| W2-ws-feature-dirty | grok | grok-4.6-build | current | pass | 1.4 | |
| W2-ws-main-dirty | claude | claude-fable-5 | current | pass | 0.6 | |
| W2-ws-main-dirty | codex | gpt-5.6-sol | current | pass | 1.0 | |
| W2-ws-main-dirty | grok | grok-4.6-build | current | pass | 1.1 | |
| W2-ws-noremote | claude | claude-fable-5 | current | pass | 0.6 | |
| W2-ws-noremote | codex | gpt-5.6-sol | current | pass | 1.2 | |
| W2-ws-noremote | grok | grok-4.6-build | current | pass | 1.4 | |
| W3-rtc-complete | claude | claude-fable-5 | current | pass | 1.3 | |
| W3-rtc-complete | codex | gpt-5.6-sol | current | pass | 2.8 | |
| W3-rtc-complete | grok | grok-4.6-build | current | pass | 4.2 | |
| W4-rtc-collision | claude | claude-fable-5 | current | pass | 0.5 | |
| W4-rtc-collision | codex | gpt-5.6-sol | current | pass | 1.1 | |
| W4-rtc-collision | grok | grok-4.6-build | current | pass | 2.7 | |
| W5-rtc-late-owner | claude | claude-fable-5 | current | pass | 1.5 | |
| W5-rtc-late-owner | codex | gpt-5.6-sol | current | pass | 2.7 | |
| W5-rtc-late-owner | grok | grok-4.6-build | current | pass | 4.5 | |
| W6-engine-require-unavail | claude | claude-fable-5 | current | pass | 1.3 | |
| W6-engine-require-unavail | codex | gpt-5.6-sol | current | pass | 3.2 | |
| W6-engine-require-unavail | grok | grok-4.6-build | current | pass | 5.3 | |
| W7-engine-prefer-codex-live | claude | claude-fable-5 | current | pass | 5.4 | |
| W8-standalone-no-review-ship | claude | claude-fable-5 | current | pass | 4.0 | |
| W8-standalone-no-review-ship | codex | gpt-5.6-sol | current | pass | 9.8 | |
| W8-standalone-no-review-ship | grok | grok-4.6-build | current | pass | 9.6 | |
| W9-goal-engine-claude | claude | claude-fable-5 | current | pass | 5.9 |
Total graded host-cells: 97; pass 83; fail 14.
Failures, classified
Every non-passing host-cell above falls into one of these classes. "Skill" means the behavior contradicts the skill text or its consumer's contract; "contract" means two skill files disagree; "prompt" means the host obeyed the skill but not the evaluation prompt's stop instruction; "harness" means the evaluation environment, not the skill, produced the result; "rubric" means the predeclared assertion could not decide the behavior and was corrected once.
| Cell(s) | Class | What happened | Owning layer and recommended fix |
|---|---|---|---|
| W1-carrier-badjson codex; W1b-badjson-current-codex-2/3; W1b-badjson-base-codex-1/2/3 | skill (host-specific, pre-existing) | Codex (gpt-5.6-sol) accepted implementation_engine:{"mode":"prefer","target":"codex","model":null,"source":"eval" (no closing brace) in 6 of 6 trials — 3 current arm, 3 baseline — and went on to fetch, branch, implement, commit, and return status: complete with a fully populated binding, never acknowledging the defect. Claude and Grok rejected it before any workspace action (1/1 each), as did all three hosts for the duplicate, type-invalid, and unsafe-run-id variants (9/9). |
skills/ce-work/references/input-triage.md, carrier grammar. The text already says "Reject malformed JSON … before any workspace action"; Codex pattern-matches the four field names instead of parsing. The fix proposed here — a condition plus a parse mechanism in prose — was tried in Round 3 and did not change Codex's behavior (it parsed a retyped string); see the Round 3 table and docs/solutions/skill-design/prose-cannot-validate-caller-control-data-byte-for-byte.md. Not a regression: identical at 66ccf579. |
| W6-engine-require-unavail (all three hosts, after regrade) | contract (pre-existing) | With mode: require, target: cursor, and a pinned model absent from cursor-agent --list-models, all three hosts disclosed fallback_reason and continued natively, returning status: complete. That is what ce-work says (execution-engines.md line 22 and 73; cross-model-execution.md lines 27, 90, 129: "does not turn route unavailability into a blocker … disclose … then continue natively"). It is the opposite of what lfg says it will accept (references/work-return.md line 11: "An unavailable require route must not prompt, fall back, or start native work — it is a stop"). Both texts are byte-identical at baseline. |
The producer's fallback semantics are owned by cross-model-execution.md and are incident-backed; the consumer's sentence is unenforceable as written because the return it receives is complete. Fix at the consumer: state the condition on the return fields (implementation_engine_binding.mode == require and actual_route != requested_route → stop as blocked and surface fallback_reason), or change the producer to return blocked under require. Pick one; today each side believes the other stops. |
| P7-full-plan claude; P9-owner-missing-late claude | evidence contradicting an intended contract | In both full ce-plan runs Claude read final-review.md, reasoning-elevation.md, and plan-handoff.md at tool calls 11–15 — before research dispatch and before the plan Write (call 31/35) — and did not re-read plan-handoff.md after writing. The late-owner watcher fired after the write, but the owner had already been consumed, so the run completed normally instead of returning the blocked envelope. (Elevation itself resolved once: native attempt rejected pre-launch → Claude CLI adapter failed on the unavailable model → inline; no second authoring dispatch.) |
skills/ce-plan/SKILL.md workflow preamble ("Before each phase, read its required reference completely") and the Phase-5 "STOP. Read references/plan-handoff.md immediately before 5.3.8 and 5.4". Neither states the condition that an early read does not satisfy the acting-point read, so a front-loading host satisfies the letter and loses both the freshness and the context benefit the kernel design exists for. Verify on Codex/Grok (P7/P9 below); if they also front-load, restate the condition once in the kernel rather than per step. |
| L2-retry-only-when-neither claude | skill (condition gap), 1 of 2 hosts | The stub ce-plan returned prose with no plan and no blocker while a stale, topically related docs/plans/widget-plan.md (from the base fixture) sat in <root>/plans/. Claude's lfg treated that file as the written plan and invoked ce-work mode:return-to-caller docs/plans/widget-plan.md; Codex retried ce-plan exactly once and stopped, as the body requires. With the stale plan removed (L2b), Claude retried once and stopped correctly. |
skills/lfg/SKILL.md step-1 gate + references/plan-brief.md readiness check. "Only absence of both a blocker and a plan file invokes ce-plan again" keys on any plan file at the root; the condition should be the plan ce-plan reported writing in this run. |
| L3-complete-advances claude | adversarial fixture; correct stop | The stub ce-work returned a complete envelope but changed no files. Claude applied work-return.md's "verify that implementation work was performed" and stopped; Codex advanced to review. L3b (same envelope with the implementation commit present) advanced on both hosts through ce-simplify-code, ce-code-review, and ce-test-browser. |
Not a defect in Claude. Codex's consumer is weaker on the "work was performed" check (1 trial); Verify-class. |
| L2b-retry-clean codex | harness (isolation failure); evidence excluded | Codex mis-resolved the stub skill path, find-hunted for ce-plan/SKILL.md, found only ~/.codex/plugins/cache/.../3.22.4/skills/ce-plan/SKILL.md (the disabled installed plugin's files are still on disk), and ran the 3.22.4 ce-plan/ce-work prose inline. It never invoked the stub. This is the only Codex cell with any plugins/cache read (all other Codex nested cells: zero). |
Evaluation harness. Codex nested isolation by CODEX_HOME + enabled = false removes the skill from the list but not from the filesystem; a model that path-hunts can still reach it. Record as a caveat on every Codex nested cell and re-check L5-nested-live-codex-1 for cache reads before counting it. |
| P2-answer-seeking claude, codex | prompt ambiguity (skill condition does not decide) | The prompt carried approach-language ("a plan for how you would find out …") over an investigative question. resume.md 0.1a's explicit trigger ("plan how you'll do X") and its R16 exclusion ("with no approach-language … answer-seeking") both fire; Claude and Codex routed to approach-altitude, Grok to answer-seeking. Neither host read config; none mutated. The text is unchanged by the restructure. |
Observation, not a kernel defect. If the split matters, the owning condition is in references/resume.md 0.1a/R16. |
| P6-approach claude | prompt compliance | Claude routed correctly to approach-altitude, then composed the full approach-plan and checkpoint despite "stop before producing it"; Codex and Grok held. No file written. | Not a skill defect. |
| W8-standalone-no-review-ship codex (grader) | rubric | Codex ran the standalone tail including ce-code-review (receipt Ready to merge) and reported "Review: complete"; the first rubric looked for the phrase "code review". Corrected once; passes. |
— |
| L4-status-blocked codex, L4-status-unknown claude, L4-status-malformed codex, P8/W5 markdown-bold status fields, W2 compound shell observations, W1 "reports carrier error", P7 "elevation once" (adapter chain), W6 (consumer-vs-producer rubric) | rubric | Each was an assertion that could not distinguish correct behavior (a **status:** \blocked`rendering; aNo DONEsentence in prose; the one documentedce-workrecovery re-invocation on a malformed return; a single compoundgit checkout -b … && git branch --show-current`; an ordered adapter chain inside one elevation resolution). Each was corrected once and the same transcripts regraded. |
— |
Transcript and tool-trace audit
Two fresh-reader subagents that did not author the skills or the matrix audited 21 ce-plan host-cells (P1–P6, P8) and 34 ce-work host-cells (W1/W1b/W2/W4) against the raw traces. Their findings, beyond what the grader scores:
- No host read an installed copy of the skill under test in any audited cell (zero hits for
plugins/cache,~/.claude,~/.codex,~/.grok,.agentsin tool inputs and Codexexeclines). Nested skills invoked by a standalone run in a non-isolated cell (ce-doc-review,ce-code-review,ce-simplify-code,ce-commit) do come from the installed 3.22.4 cache, as expected for a fresh process with the plugin enabled; only the skill under test was the worktree copy. - FILES_READ trailers matched the traces in all 55 audited host-cells (no claimed-but-unread files; omissions were reads that failed because the file did not exist, which is the correct negative).
- No mutation in any read-only cell (git before == after, empty diff, no write tools).
- Reading order is loose on every host. "Output first" (
output-mode.mdbefore interpreting any phase) was honored by Grok consistently; Claude and Codex batch the Phase-0 reads or read the plan file first (P1 codex, P3 claude/codex, P4/P5 claude). Codex readworkspace-setup.mdbeforeinput-triage.mdin W2-ws-ahead. Outcomes were unaffected, but the kernel's "before each phase" sequencing is advisory in practice. - Reads beyond the stop point: P1 codex read
html-rendering.mdtwice after being told to stop before writing; P5 claude readdeepening-workflow.mdalthough it is gated on "when directed"; P6 claude readintake.md; P8 claude read every workspace file after already knowing the owner was missing. Unproductive, not unsafe. - Codex self-reports task-list creation without a visible
update_planevent (W2 detached/noremote/ahead); Grok'stodo_writecalls are visible; Claude created no task list in W2 cells (stopped one step early on "stop after workspace setup"). - W4 envelope semantics differ by host for the same blocked state (
u_ids_attempted: [U1]on Claude,[]on Codex/Grok). The field inventory is identical; the meaning of "attempted" is not pinned. - P8 blocked envelopes render differently (Codex clean
key: value; Claude fenced multi-line; Grok bold markdown). Fields are present on all three; anlfgconsumer's tolerance for the renderings is exercised only by the Claude/Codex nested cells. lfgrequest sanitization varies: Claude strips thelfg:prefix and the "and ship it" clause from the feature request it passes toce-plan; Codex passes the arguments unchanged (the body says "unchanged" when no routing directive exists). Plan content was equivalent.- One harness denial: P1 grok's
git rev-parse --show-toplevelwas denied by the read-only policy; Grok assumed the workspace was the repo root and still resolved the format from config.
Variance across repeated trials
| Path | Trials | Outcome |
|---|---|---|
| Malformed carrier, Codex | 6 (3 current, 3 baseline) | 6/6 accepted — deterministic on Codex, identical across arms |
| Malformed/duplicate/type/run-id carriers, all hosts | 12 current | 11/12 rejected before mutation (the one miss is the Codex case above) |
ce-work return-to-caller complete (W3) |
3 (one per host) | 3/3: full 23-field envelope, standalone_shipping_skipped: true, return-to-caller.md read on entry and again before return, path-limited commits, nothing reached the push/PR shims |
| Dirty-file collision (W4) | 3 | 3/3 blocked without editing or committing the user's file, usable recovery path, no question asked |
| Late return-owner failure (W5) | 3 | 3/3: second read failed after the first commit; each host returned the kernel's reduced status: blocked envelope (plan_path, changed_state, blockers naming the owner, recovery_path) with the commit preserved and no standalone tail |
| Workspace topologies (W2) | 15 | 15/15 safe base choice (origin/main when not ahead; HEAD when ahead or no remote; stay on an existing feature branch); user dirty files untouched and uncommitted in all 6 dirty cells; no stash, no add -A, no commit on main; branch observed before and after every move |
| Early missing owner, pipeline (P8) | 3 | 3/3 complete blocked envelope naming intake.md, no plan written, no reconstruction from memory |
| Nested live chain, Claude (L5) | 2 | 2/2 ran worktree lfg → ce-plan → ce-doc-review → ce-work (return-to-caller) → ce-simplify-code → ce-code-review → ce-test-browser → local-only ship → DONE with no push and no gh pr reaching the shims; ce-plan resumed the fixture's matching plan in place rather than writing a new one |
lfg seam with stubbed children (L1, L3b, L4×4) |
2 hosts each | 14/14: blocker outranks an existing plan; blocked/failed/unknown/malformed returns stop before simplify/review/ship (Codex took the one documented recovery re-invocation on the malformed return, Claude stopped directly — both within contract); a complete return with real changes advances through the review steps |
Cross-model live (W7, Claude host, prefer codex) |
1 | A real Codex worker was dispatched through the bundled controller; the first worker attempt returned blocked on a packet defect and was abandoned with receipts preserved; the second attempt completed, was integrated as a canonical commit, and verify-run returned RUN_VERIFIED; no native substitution |
| Goal mode on Claude (W9) | 1 | Explicit route outcome: no callable goal tool → copyable prompt emitted, then inline; standalone tail ran with a ce-code-review receipt before finishing |
Rows for P7 (Codex, Grok), P9 (Codex, Grok), and L5 (Codex) are in the addendum below.
What is now proven
ce-planrouting and output ownership on three hosts: software planning resolvesplan_output: htmlonly after the domain route (P1); answer-seeking that mentions code never probes config (P2, all hosts); requirements-only input enriches in place with no resume prompt (P3); explicit-path edits take the standard resume flow (P4); deepening short-circuits to 5.3 withfinal-review.mdas owner (P5); approach requests loadapproach-altitude.md(P6); a missing owner before the artifact returns the full blocked envelope (P8); planning never crossed into implementation in any of 31ce-planhost-cells.ce-worktriage and workspace safety: carriers are validated before mutation on Claude and Grok for all four defect classes and on Codex for three of four; requirements-only artifacts stop (round 1, 6/6); five branch topologies choose safe bases with before/after observation and leave user WIP untouched; all branch work ran in disposable fixture repositories.ce-workexecution and completion ownership: return-to-caller complete, blocked, and late-owner-failed returns carry the documented fields and preserve state; return-to-caller never entered review, commit, or PR; standalone mode on Claude and Codex rance-code-reviewand held the ship step (nogh prreached the shim in any cell); engine resolution happened before any write in every write-capable cell; a liveprefer codexroute dispatched, recovered, integrated, and verified through the controller.lfgseam behavior with real nested dispatch: planning before work; a structured blocker outranks an existing plan; only a valid complete return with real changes advances; blocked, failed, unknown, and malformed statuses stop safely; the full no-remote chain runs end-to-end on Claude using worktree skills only.- Loader and portability: sizes and descriptions reconfirmed; three converters package references recursively and byte-identically; root-relative pointers resolve on fresh Claude, Codex, and Grok hosts in every cell.
What remains unexercised or unproven
- A conformant Agent Plugins loader applying the 8,000-byte truncation itself; the eval shows eligibility and successful loads, not the truncation path.
- The late-owner-failure path for
ce-planon Claude (front-loading made it unreachable) and, pending the addendum, on Codex and Grok. lfg → ce-plan → ce-worknested chain on Grok (Grok discovers skills only from its user-level skills/plugins directories; loading the worktree there would mutate user state) and on Codex beyond the cells noted in the addendum.- Any live GitHub mutation: push, PR creation, CI watch, babysit.
- Post-
initcross-model lock against native fallback (W7 exercised the sanctioned post-start fallback claim path indirectly via an abandoned attempt, not a forbidden native substitution). - Goal-mode and dynamic-workflow engines on Codex (
create_goal) — only the Claude no-callable-tool outcome was exercised. - Cursor-as-target cross-model execution; only its unavailability was exercised.
- The
require-route contract after a fix: whichever side is changed needs a paired producer/consumer cell. - Weaker model tiers on any host; all cells ran the hosts' current default models.
Evidence contradicting the implementation's intended contract
- "Final review and handoff reload their owners at the acting point" — contradicted on Claude (P7, P9): owners are read once at kernel load; no reload before 5.3.8/5.4.
- "Malformed carriers are rejected before any workspace mutation" — contradicted on Codex in 6/6 trials; holds on Claude and Grok. Pre-existing.
- "
lfgaccepts only whatce-workproduces" — therequire-route fallback is accepted byce-work's own text and rejected bylfg's; neither side's cells stop. Pre-existing. - "Only absence of both blocker and plan permits the retry" — on Claude, an unrelated pre-existing plan file satisfied the gate once (L2); the clean-fixture retry behaved.
Deterministic validation for this report
Run after the report was written: bun test tests/codex-skill-prompt-budget.test.ts tests/skill-eval-cell and git diff --check (results recorded in the addendum). The round-2 driver, grader, fixtures, and stub plugin live in the session scratchpad and were not added to the repository; the repository diff after this round differs from the start of the round only in this report file.
Addendum: the last three cells and the final tally
- P9-owner-missing-late, Codex and Grok: pass. Both hosts wrote the plan, the watcher removed
references/plan-handoff.md, the acting-point read failed, and each returnedstatus: blockedwithartifact_path,blockernaming the missing owner, and arecovery_paththat resumes at Phase 5.3.8 without substituting a plugin-cache copy; the plan file was preserved. So the late-owner safety path is demonstrated on Codex (1/1) and Grok (1/1) and unreachable on Claude (1/1) because of front-loading (see "Evidence contradicting", item 1). Codex readplan-handoff.mdonce in each run, after the plan write (an earlier grep hit on that name in P7 was afindlisting, not a read — corrected after the learning-capture pass re-read the exec blocks). - P7-full-plan, Codex: pass; Grok: fail on one check. Codex wrote the plan, resolved elevation once (native attempt rejected pre-launch, Claude CLI adapter, inline), invoked the installed
ce-doc-review, and rendered the exact handoff question. Grok readfinal-review.md/reasoning-elevation.mdafter research andplan-handoff.mdimmediately after the plan write (the cleanest phase ordering of the three hosts), resolved elevation once, presented the Phase-5.4 menu with the plan path, but did not render the body's verbatim question ("Plan ready at<path>. What would you like to do next?") — a wording deviation, not a missing gate. Grok also handled document review by reading~/.claude/plugins/cache/…/3.22.4/skills/ce-doc-review/…and spawning its own reviewer subagents, instead of recording theskill_unreachableenvelopeplan-handoff.mdrequires when the skill cannot be invoked; Grok has no CE skills installed, so this is a substitution the owner forbids ("do not substitute a generic … subagent"). Verify-class on Grok; Claude and Codex invoked the skill through their skill mechanisms. - L5-nested-live-codex-1: pass. Codex ran the full no-remote chain with worktree skills from the scratch
CODEX_HOME(zeroplugins/cachereads;ce-eval:ce-plan,ce-eval:ce-work mode:return-to-caller, simplify, review, browser gate, local-only ship,DONE). With the two Claude trials, the fullest safe nested chain is 3/3; it stops exactly atlfg's shipping precondition (git remoteempty → commits made, push/PR/CI skipped as a terminal state), which is the documented no-remote terminal. - Final tally: 97 graded host-cells (96 matrix cells plus the manual-host probe rows excluded), 83 pass, 14 fail. The 14: six Codex malformed-carrier acceptances (pre-existing), L2 Claude stale-plan gate (condition gap), L2b Codex (harness isolation failure, excluded as evidence), L3 Claude (correct stop on a complete-with-no-change stub, counted against the predeclared "advance" assertion), P2 Claude/Codex (prompt ambiguity), P6 Claude (prompt compliance), P7 Grok (handoff wording), P9 Claude (front-loading made the late failure unreachable).
- Deterministic validation for this round (run after the report was written; the repository diff differs from the start of the round only by this file):
bun test tests/codex-skill-prompt-budget.test.ts tests/skill-eval-cell→ 55 pass, 0 fail;git diff --checkclean. The full suite was not re-run because no code, test, or skill file changed in this round; the round-1 result (3,468 pass) stands for the implementation diff. - Manual steps that remain yours: a Grok nested chain (needs a symlink in
~/.grok/skills), a live PR tail on a throwaway GitHub repository, and whether to restore the released Pi install under~/.pi/agent. The runbook is the Claude artifact "Issue #1482 manual behavioral eval steps"; results from it will be graded with the same grader and appended here.
Manual step 1 result: nested chain on Grok (run by the maintainer, graded here)
Evidence: /tmp/compound-engineering-501/ce-skill-eval/issue1482-r2/manual/grok-chain/ (prompt, stream.jsonl with 173 tool calls, git-after.txt). Served model grok-4.6-build; exit 0; 12 minutes.
- Chain: worktree
lfg→ce-plan(resumed the fixture's matchingdocs/plans/widget-plan.mdin place, as pipeline mode requires) → document review →ce-work(git checkout -b feat/greet-quiet, branch re-read, proof-first test, path-limited commit31cf811) →ce-simplify-code(three lenses, no change) →ce-code-review(correctness + testing, artifacts under/tmp/compound-engineering-501/ce-code-review/…, one review fix committed as925d065) → browser gate → local-only ship (git remote -vempty; the plan committed as7ba8a9d; nogit push, nogh pr) →<promise>DONE</promise>. Pass on every L5 assertion. With the two Claude and one Codex trials, the fullest safe nested chain is now 4/4 across three hosts, each stopping at the documented no-remote terminal. - Source-identity caveat: the
~/.grok/skills/ce-evalsymlink was dangling (it pointed at/Users/tmchow/Code/skills), so Grok found noce-eval:*entries and instead readce-plan,ce-doc-review,ce-work,ce-simplify-code,ce-code-review, andce-test-browserfrom the worktree by absolute path next to the pinnedlfg(tool calls 17, 34, 69, 87, 96, 164). It listed the Claude plugin cache twice and~/.grok/skillsonce while searching, but read no file from any cache. Every skill byte it consumed was the worktree's; the route to them was filesystem navigation, not a skills listing, which is Grok's only mechanism anyway (it has no skill-invocation tool and no CE skills installed). - Grok-specific deviation, consistent with P7: document review, simplification, and code review ran as in-process emulations with
spawn_subagentpersonas read from the worktree references, not as skill invocations, and noskill_unreachableenvelope was recorded. On a host with no skill mechanism this is the only way the chain can run; it is noted, not failed. - The
refs/cmux/last-turn/*entries ingit log --allare snapshots made by the maintainer's terminal tool during the run, not actions of the skill.
Manual step 2 result: live PR tail — harness failure, no behavioral evidence, one remote side effect
The runbook's step-2 commands were defective and the run never reached a canary repository. gh repo create ce-1482-canary --private --clone -- -q failed (the repository was never created), so the following cp/cd/git add . && git commit -m seed && git push -u origin HEAD executed inside this worktree: commit 5170bee1c seed (two fixture files under ce-1482-canary/) landed on tmchow/eval-issue-1482-skill and that branch was pushed to origin for the first time. The Claude run then started in bullhead/ce-1482-canary with --plugin-dir pointing at the worktree root, but the nested invocations resolved to the installed compound-engineering:ce-plan / ce-doc-review / ce-work (3.22.4) rather than ce-eval:* — the enabledPlugins override that worked in the scratch probes did not apply when the working directory was inside the plugin root — wrote docs/plans/2026-08-21-1210-feat-greet-quiet-export-plan.md into this worktree, and ended after 48 turns with API Error: The response stopped arriving (exit 1) before any implementation edit, push, or PR. gh pr list at the end returned the repository's existing PRs; none was created.
Classification: evaluation-harness failure (runbook command defect plus plugin-isolation assumption), not a skill result. Nothing in it is counted. Remote side effect to reverse: the pushed branch tip 5170bee1c on origin/tmchow/eval-issue-1482-skill. Local residue: the seed commit, ce-1482-canary/, and the stray plan file; the 33-path implementation diff is untouched. The live PR tail remains unexercised.
Manual step 2, second attempt: live PR tail — pass
Run by the evaluation session with the maintainer's authorization after the first attempt's harness failure, against a fresh private canary tmchow/ce-1482-canary created outside any checkout. Evidence: /tmp/compound-engineering-501/ce-skill-eval/issue1482-r2/manual/live-pr-2/ (stream with 115 turns, prs.json, pr1.json, git-after.txt, plugin-live.sha256). Served model claude-fable-5; exit 0; zero plugins/cache reads; every nested invocation was ce-eval:* (the byte-identical worktree snapshot).
- Chain:
ce-eval:lfg→ce-plan(enriched the fixture plan in place) →ce-doc-review mode:non-interactive(1 fix applied, 1 P3 decision left) →ce-work mode:return-to-caller(git fetch origin main,checkout -b feat/greet-quiet, proof-first test, path-limited commit) →ce-simplify-code→ce-code-review mode:agent("Ready to merge", zero findings) →ce-test-browser mode:pipeline(skip, no web surface) →ce-commit-push-pr mode:pipeline branding:on→git push -u origin HEAD→gh pr create --base main --head feat/greet-quiet→ce-babysit-pr mode:pipeline <url>(returned with the absent-CI residual; no needs-human items) →<promise>DONE</promise>. - Remote state after:
mainuntouched at the seed;feat/greet-quietwith two commits (plan, feature); exactly one open PR, #1, basemain. No force-push, no merge. - Note: the PR body carries the CE badge but not the
## Security Disclosure/## Agent Disclosuresections — the canary has no.github/pull_request_template.md, so those sections are not expected there; they are this repository's template convention, not the skill's. - Residue: none. The maintainer deleted the canary repository afterward (the evaluation token lacked the
delete_reposcope); the local clone was trashed.
With this, every path the round-1 report listed as unexercised has been run at least once except the loader's own truncation behavior, post-init cross-model lock violation, Codex goal-mode, Cursor-as-target, weaker model tiers, and the require-route contract after a fix.
Round 3: fixes for the three confirmed defects, with post-fix cells
Authorized by the maintainer after round 2. Each fix was made at its owning layer and re-evaluated with the same prompts, fixtures, and grader (matrix addendum 4; evidence under …/issue1482-r2/F1-*, F2-*, F3-*).
| Defect | Change | Post-fix cells | Result |
|---|---|---|---|
lfg stale-plan gate |
skills/lfg/SKILL.md step-1 gate now reads "a plan file ce-plan reported writing this run"; references/plan-brief.md states the condition (a file already under <root>/plans/ that ce-plan did not report is not a written plan; neither blocker nor reported path → the single retry). Pin added in tests/pipeline-review-contract.test.ts. lfg body headroom 85 → 49 bytes. |
F3: Claude ×2, Codex ×1, stale plan present, stub ce-plan returns prose |
3/3: ce-plan invoked exactly twice, ce-work never, no DONE (was 1/2 hosts advancing on the stale plan) |
require-route producer/consumer conflict |
Resolved at the consumer. skills/lfg/references/work-return.md now says ce-work discloses and continues natively under require (owned by cross-model-execution.md) and LFG stops as blocked when implementation_engine_binding.mode is require and actual_route differs from requested_route, reporting both routes and fallback_reason. The unenforceable "must not prompt, fall back, or start native work" sentence is gone. Pins: new test in tests/pipeline-review-contract.test.ts (consumer condition + producer sentence), and the old "must not prompt" pin in tests/skills/unified-plan-artifact-contract.test.ts retargeted with its reason. |
F2: Claude ×1, Codex ×1, stub ce-work returns complete with mode: require, requested_route: cursor, actual_route: native |
2/2: ce-work once, no simplify/review/ship, stop names the route mismatch, no DONE |
| Codex accepts malformed carrier JSON | Attempted and reverted. Added to input-triage.md: acceptance is a successful strict parse of the exact carrier substring by a real JSON parser, never repaired. Codex (gpt-5.6-sol) complied literally — it ran node -e 'JSON.parse(…)' in all three trials — on a string it had retyped with the closing brace restored (F1-badjson-post-codex-2's command contains …"source":"eval"}), then implemented and committed as before. A prose mechanism cannot reach bytes the model normalizes while transcribing; Claude and Grok rejected the carrier before and after (1/1 each). The sentence was removed rather than shipped as an untested-benefit mechanism; input-triage.md is byte-identical to its round-2 state. |
F1: Codex ×3, Claude ×1, Grok ×1 | Codex 0/3 rejected (unchanged); Claude 1/1, Grok 1/1 rejected (unchanged) |
Residual for the Codex carrier defect: the only caller that emits the carrier is lfg via stage-routing.md, so a malformed carrier in practice means a buggy caller, and the damage is an implementation on a feature branch under a prefer binding that the caller then reads as a normal return. A deterministic guard would have to live where the raw argument exists — the host's skill-invocation layer or a bundled script that receives the argument verbatim — not in prose. Recorded as a known Codex limitation, not fixed.
Validation after round 3: bun run test 3,469 pass, 0 fail across 138 files; bun run release:validate in sync (33 skills); git diff --check clean.
PR #1508 review round: the front-loading condition, added and re-evaluated
A Codex review bot filed the Claude front-loading gap as a P1 against skills/ce-plan/SKILL.md:37. The kernel's Workflow preamble now reads: "Read a phase's required reference when that phase is entered, completely; a read made before that phase does not satisfy it, and a terminal owner is read again at its step even when already in context." Pinned in tests/pipeline-review-contract.test.ts. Post-fix cell P9b-owner-missing-late-post-claude (byte-identical prompt to P9, claude-fable-5): Claude read plan-handoff.md at the acting point after the plan write, the watcher's removal took effect, and the run returned status: blocked with phase, blocker, recovery_path, and artifact_path, plan preserved — pass (1/1). "Evidence contradicting" item 1 is closed on Claude for the late-owner path; whether the condition also stops the up-front bulk read of every reference (the context-cost half) was not re-measured.