skills/boshu2__agentops
1
0
Fork 0
mirror of https://github.com/boshu2/agentops.git synced 2026-09-14 15:08:13 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
main
boshu2__agentops/tests/scripts/probe-skill.bats
T
Bo 1cebfee94d Measure one skill: sealed probe dispatch and the first current ledger row (#1101) 
## Measure one skill: filesystem-sealed probe dispatch and the first
current ledger row

Provenance: the instrument-panel roadmap
(`docs/plans/2026-08-26-instrument-panel-roadmap.md`) closed out with
zero skills measured because control-arm reps read
`skills/<skill>/SKILL.md` from the checkout and the operator's skill
roots mid-run. Its named successor was filesystem-sealed dispatch. This
PR builds the seal, then uses it, and the instrument caught one more
leak on the way.

**The seal.** Codex's own seatbelt does not nest inside an outer
`sandbox-exec` profile, so a rep now runs under an outer profile with
Codex's sandbox bypassed for exactly that reason: `file-read*` denied on
the checkout and every skill root (`~/.agents`, `~/.claude/skills`,
`~/.gemini/skills`, `~/.codex/skills`), `file-write*` denied everywhere
except the rep's workspace, a scratch HOME, and the temp roots. HOME and
CODEX_HOME are scratch (auth symlinked). Dispatch refuses without
`sandbox-exec` unless `PROBE_SEAL=none` is set, which marks the run
coverage-ineligible. The seal is recorded in the capture contract
(`agentops-skill-probe-capture.v3`, a `seal` block), coverage
eligibility requires `seal.mode == seatbelt`, and `verify-scorecard`
re-checks that the denied roots include the repository and the four
skill roots. Existing 08-26 sets become `legacy-unsealed` and
ineligible, which changes nothing they were counting.

**The leak the seal did not cover.** The first sealed capture showed the
xhigh control-2 rep listing the shared live workspace, finding
`treatment-1.prompt`, and reading it; the treatment prompt embeds the
skill. So every rep now gets its own empty workspace, prompts reach a
rep on stdin only, raw JSONL and stderr live in a harness-private
dispatch directory the rep can only stat, and a `sibling-prompt-read`
trap degrades any rep whose successful command names a prompt, contract,
seal, or stage file or whose output lists a sibling's artifacts.
Replaying the contaminated set under the new trap turns its REGRESSIVE
into INERT with the contaminated rep degraded; that set is superseded
and not committed.

**The second pass, after the judges read the first.** A Fable validator
passed the first sealed capture with four findings and a Codex read
failed it with seven, all real: the whole per-user temp tree stayed
readable (hundreds of stale probe directories held prompt files with the
canonical skill bytes), sibling checkouts and the real `~/.codex`
sessions were readable because seatbelt matches the traversed path and
the skill roots are symlinks into the main checkout, the dispatch
directory was writable so a rep could rename an earlier prompt into its
workspace, the contract trusted asserted seal metadata (a hand-made
Linux record was coverage-eligible), the only real-Darwin test nested
seatbelt and proved nothing, and the operator's `config.toml` was
symlinked in whole, so every rep started the operator's MCP servers. The
seal is now one run directory removed on exit (`home/ ws/ tmp/
dispatch/`, reset per rep); reads are denied under the real HOME, the
real TMPDIR, `/tmp`, the checkout, the git common directory, and each
skill root's resolved target; `dispatch/` is write-denied with
metadata-only reads; `file-link` and `file-clone` are denied on every
denied root; the rep's config is rebuilt from top-level scalars only and
`auth.json` is copied, not linked. The contract binds platform,
mechanism, wrapper, the denied and allowed roots, the rep environment,
and the sanitization, and coverage refuses any record that omits them;
the Darwin test performs its reads inside the seal. One recorded hole
remains by necessity: the codex launcher under `~/.local/bin` and
`~/dev/dotfiles/bin` is re-allowed by literal path, and coverage refuses
any allowed path inside the checkout.

**The third pass, after the judges read the second.** Both judges failed
the hardened tip again, and the headline was network: bypassing codex's
own sandbox to nest the outer seal had dropped every inner deny, not
only the filesystem ones, so a rep could fetch the canonical skill from
GitHub while the disk was sealed. A rep now reaches the network only
through a harness-owned CONNECT proxy
(`scripts/lib/probe-connect-proxy.py`) with a host allowlist discovered
from real runs (`chatgpt.com`, `ab.chatgpt.com`, the rotating
`.oaiusercontent.com` suffix, and the API-key hosts), behind a seatbelt
`network*` deny that admits only the proxy; every attempt is logged, a
refused one degrades the rep, and web search is disabled in the rep's
config. The profile is now rendered from the bound seal block by one
function, so `verify-scorecard` reconstructs it and requires the digest
to match; the recorded launcher chain is the only allowed read under a
denied root; the wrap invokes `/usr/bin/sandbox-exec` by absolute path;
the real CODEX_HOME and the Darwin cache directory are denied and bound;
every non-stdio descriptor is closed and `/dev` writes narrowed to four
devices; each rep runs in its own process group that must be empty
before the next reset; one minimal config is generated per run, bound by
text and digest, and re-parsed after each rep; one guarded trap covers
the run root and any unpublished stage; and the coverage gate prints an
eligibility row for every ledger row that names a scorecard. What stays
open is stated in the README: allow-default still governs what the
profile does not deny.

**The fourth pass, after the judges read the third.** Both judges failed
the third tip on properties the record carried but the verifier did not
check. GNU `timeout` sets its own process group and ran outside the
seal, so the per-rep reap was vacuous (four sleeping survivors after the
test passed); the seal is now the outermost process, timeout runs inside
it with `--foreground`, its absolute path is bound, and the reap counts
survivors before signalling. The verifier now pins, not merely records:
the egress host set (a module constant), port 443 only, no unix sockets,
a loopback proxy, the generated config text equal to the renderer's
output, the real CODEX_HOME and cache root inside the denied roots,
writable roots under the run root, the four device paths, the
environment allowlist, the launcher chain's digest against the producer
identity, and it refuses unknown record fields. The proxy logs an
attempt before dialing, captures the rep at accept time, refuses names
that resolve to private or loopback ranges, and its source joins the
evaluator hash set; the per-rep egress log is published with the fixture
set. Withdrawn ledger rows get their own eligibility reason. The README
lists which seal fields the verifier pins and which it only records; the
cross-capture aggregate is recounted to 4 of 14. Still open and
disclosed: Mach IPC and Apple Events beyond the filesystem and network
seals, exact-byte environment sealing beyond the allowlist, and cleanup
ownership by inode.

**The fifth pass, after the judges read the fourth.** Both judges found
the seal, proxy, reap ordering, transcripts, and scorecards sound and
failed the tip on claims the tree did not hold: the egress log the docs
called published was caught by the repository's `*.log` ignore rule, a
`setsid()` child escaped the process-group reap, and one ledger sentence
said no rep ran a command when a control rep had run `rg` over its empty
workspace. The log is now tracked and required, strictly parsed, with
both allowed and refused counts recomputed per rep; the reap is fatal to
the capture when it cannot prove emptiness and an `lsof` sweep of the
run root catches a session escape; the rep launches through a real `env
-i` boundary; the timeout budget, the launcher chain's adjacency, every
required root in both read and link denies, the evaluator match, and the
trust-table value are pinned by the verifier; and every sentence in the
ledger, README, and runbook names the check that enforces it. The
remaining gap is stated: a child that both escapes the session and holds
nothing under the run root is not detected.

**The sixth pass, after CI read the fifth.** The first CI run of this PR
failed on Linux: the launcher-chain pin walked the capturing Mac's
filesystem, so on the runner both sets read as unsealed and the coverage
headline fell to zero. A pin that holds on one host is not a pin. The
chain is now bound as structure in the seal record (each link's path,
kind, symlink target, and the final file's digest, with the invoked path
as the head), the verifier checks that record on any host, and it
cross-checks the live filesystem only where the head exists, reporting
which happened. A gate-level test runs the coverage gate twice, once
with the chain made absent, and requires byte-identical output.

**The measurement.** `premortem-plan-shape-t2` (two seeded structural
defects, band [1,4]) captured seven times on 2026-09-03, once under each
seal; the first six sets were deleted with their scorecards when a judge
or CI broke that seal. The rows that count are the seventh capture: low
INERT (control 0/2, treatment 0/2) and xhigh INERT (control 1/2,
treatment 1/2); no rep ran a command, no egress was refused, both
per-rep egress logs are tracked and verified, and both scorecards verify
with every pinned field, the launcher chain checked as a record and
cross-checked on the capturing host. The superseded captures read, at
low, BEHAVIORAL 1/1, INERT, INERT, BEHAVIORAL 1/2, INERT, BEHAVIORAL 1/2
and, at xhigh, INERT, BEHAVIORAL 2/2, INERT, INERT, INERT (control 1/2),
INERT. The reversals across captures are recorded as an unresolved N=2
observation, not variance; across all seven the treatment arm put the
marks in band in 7 of 26 usable reps and the control arm in 2 of 28, and
the ledger marks that aggregate as disclosure, not a row. Headroom is
SEPARATED at both levels. This is the first current, manifest-backed
ledger row; it records response-shape behavior change, never quality
uplift, and what it records today is INERT at both levels.

**Evidence on the tip.** Probe, exec-lib, metadata, coverage, headroom,
and adapter bats suites green with the Darwin seal, survivor,
session-escape, and environment-boundary tests running;
`verify-scorecard` on both scorecards; `check-skill-probe-coverage.sh`
reports measured 1 of 12 with the 2026-08-26 row labelled
`verdict-withdrawn`; regen check clean; full gate green with a
HEAD-built binary; CI's bats command green; Go bar, lint, security, and
the gate-tightening ratchet green.

**Judgment, stated plainly.** The caller elected Fable 5.1 as the
binding validator. Its fifth-round verdict on the pre-CI tip was PASS
with no verdict-driving findings; its read of the portability delta
failed on one test-only point (the coverage test had dropped the
`measured` pin that CI tripped on) which this tip restores. The
cross-family Codex reads of the same tips are FAIL. They closed the
network seal, the environment boundary, the fatal reap, the data roots,
the config binding, and the documentation, and they keep these open,
recorded here as the next pass rather than hidden: the evaluator
identity check still accepts the pre-proxy four-file shape for legacy
sets; an inherited absolute `CODEX_EXEC_TIMEOUT_BIN` is trusted without
a probe; a refusal logged under a rep label outside the expected set is
not rejected; an empty top-level `[projects]` table passes the drift
check; the `lsof` sweep treats a failing `lsof` as a clean result; and
the launcher chain, now bound as a record that any host can check, is
corroborated against the filesystem only on the capturing host, so a
record whose head does not exist there is taken on its word (the docs
say so; Codex wants the live walk required at capture and the invoked
head bound into the producer identity). Fable also noted a test that
mis-resolves when TMPDIR sits under `/private/tmp`, a `2026-09-04` date
in one refusal message, and that the low control replies used a `S2:
FINDING <reason>` shape the discriminator scores as "plan not walked"
rather than by band (outcome-invariant). Each of those changes a harness
or verifier file and would orphan the two rows again, so they are filed
as the follow-up instead of an eighth capture today. Across six judging
rounds the open set shrank only from round four on; the convergence
law's stop signal was reached at round three and the run continued on
the caller's instruction to land the instrument. The two ledger rows are
honest about what they measure, and the seal they ran under is the one
this tree can verify on any host.
2026-09-04 01:40:41 +00:00

1773 lines
75 KiB
Bash
Raw Permalink Blame History

#!/usr/bin/env bats
bats_require_minimum_version 1.5.0
setup() {
REPO_ROOT="$(cd "$BATS_TEST_DIRNAME/../.." && pwd)"
HARNESS="$REPO_ROOT/scripts/probe-skill.sh"
META_TOOL="$REPO_ROOT/scripts/lib/probe-fixture-metadata.py"
PREAMBLE="$REPO_ROOT/scripts/lib/preamble.sh"
DISPATCH_HELPER="$REPO_ROOT/scripts/lib/codex-exec.sh"
PROBES="$BATS_TEST_TMPDIR/probes"
SKILLS="$BATS_TEST_TMPDIR/skills"
PROBE_DIR="$PROBES/demo"
mkdir -p "$PROBE_DIR" "$SKILLS/demo-skill"
export SKILL_PROBES_DIR="$PROBES"
export SKILL_PROBE_SKILLS_DIR="$SKILLS"
cat > "$PROBE_DIR/probe.json" <<'JSON'
{"id":"demo","skill":"demo-skill","reps":2,"discriminator":"discriminator.sh","treatment_source":"injected-prelude"}
JSON
cat > "$SKILLS/demo-skill/SKILL.md" <<'MD'
---
name: demo-skill
description: Fixture skill.
---
# Demo skill
CANONICAL_ACTION
MD
printf 'QUESTION\n' > "$PROBE_DIR/question.md"
printf 'PRELUDE_ACTION\n' > "$PROBE_DIR/treatment-prelude.md"
cat > "$PROBE_DIR/discriminator.sh" <<'SH'
#!/usr/bin/env bash
if grep -q '^INFRA$' "$1"; then exit 2; fi
grep -q '^ACTION$' "$1"
SH
chmod +x "$PROBE_DIR/discriminator.sh"
# Filesystem seal (move 2): live dispatch fails closed without a seatbelt.
# Linux CI has no sandbox-exec, so every live test runs with a pass-through
# stand-in that honors the exact `sandbox-exec -p <profile> cmd...` shape
# the harness builds and simply runs cmd. The darwin-only seal test drops
# it from PATH and uses the real /usr/bin/sandbox-exec.
SEATBELT_STUB_DIR="$BATS_TEST_TMPDIR/seatbelt-stub"
mkdir -p "$SEATBELT_STUB_DIR"
cat > "$SEATBELT_STUB_DIR/sandbox-exec" <<'SH'
#!/usr/bin/env bash
while [[ $# -gt 0 ]]; do
case "$1" in
-p|-f|-n|-D) shift 2;;
*) break;;
esac
done
exec "$@"
SH
chmod +x "$SEATBELT_STUB_DIR/sandbox-exec"
export PATH="$SEATBELT_STUB_DIR:$PATH"
# The harness takes /usr/bin/sandbox-exec by absolute path, so a PATH stub
# no longer shadows it (that was the point of the fix). Tests that are not
# ABOUT the seal ask for the pass-through through the documented seam; the
# seal tests unset it and get the real kernel sandbox.
export PROBE_SEAL_SANDBOX_EXEC="$SEATBELT_STUB_DIR/sandbox-exec"
RUNTIME_STUB="$BATS_TEST_TMPDIR/codex-runtime-identity"
cat > "$RUNTIME_STUB" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then
printf 'codex-cli probe-test\n'
exit 0
fi
printf 'runtime identity stub is not a live producer\n' >&2
exit 70
SH
chmod +x "$RUNTIME_STUB"
}
write_transcript() {
local directory="$1" name="$2" model="$3" effort="$4" body="$5"
python3 - "$directory" "$name" "$body" <<'PY'
import json, pathlib, sys
directory = pathlib.Path(sys.argv[1])
name = sys.argv[2]
body = sys.argv[3]
arm, rep_text = name.rsplit("-", 1)
rep = int(rep_text)
contract = json.loads((directory / "capture-contract.json").read_text())
prompt = contract["prompts"][0 if arm == "control" else 1]
position = next(
item["position"]
for item in contract["schedule"]
if item["arm"] == arm and item["rep"] == rep
)
events = [
{
"type": "agentops.probe-input.v1",
"arm": arm,
"rep": rep,
"position": position,
"prompt": prompt,
},
{"type": "thread.started", "thread_id": f"thread-{directory.name}-{name}"},
{"type": "turn.started"},
{
"type": "item.completed",
"item": {"id": f"item-{name}", "type": "agent_message", "text": body},
},
{
"type": "turn.completed",
"usage": {"input_tokens": 1, "cached_input_tokens": 0, "output_tokens": 1},
},
]
(directory / f"{name}.txt").write_text(
"".join(json.dumps(event, sort_keys=True, separators=(",", ":")) + "\n" for event in events)
)
PY
}
write_legacy_transcript() {
local directory="$1" name="$2" model="$3" effort="$4" body="$5"
{
printf 'OpenAI Codex fixture\n'
printf '%s\n' '--------'
printf 'workdir: /fixture\n'
printf 'model: %s\n' "$model"
printf 'reasoning effort: %s\n' "$effort"
printf '%s\n' '--------'
printf 'user\nQUESTION\n'
printf 'codex\n'
printf '%s\n' "$body"
printf 'tokens used: 1\n'
} > "$directory/$name.txt"
}
make_fixture_set() {
local name="$1" model="$2" effort="$3" control_body="$4" treatment_body="$5"
local directory="$PROBE_DIR/$name" rep
mkdir -p "$directory"
python3 "$META_TOOL" snapshot \
--fixture-dir "$directory" \
--probe-dir "$PROBE_DIR" \
--skills-dir "$SKILLS" \
--probe demo \
--requested-model "$model" \
--requested-effort "$effort" \
--producer-override-bin "$RUNTIME_STUB" >/dev/null
for rep in 1 2; do
write_transcript "$directory" "control-$rep" "$model" "$effort" "$control_body"
write_transcript "$directory" "treatment-$rep" "$model" "$effort" "$treatment_body"
done
python3 "$META_TOOL" create \
--fixture-dir "$directory" \
--probe-dir "$PROBE_DIR" \
--skills-dir "$SKILLS" \
--harness "$HARNESS" \
--preamble "$PREAMBLE" \
--dispatch-helper "$DISPATCH_HELPER" \
--probe demo \
--reps 2 \
--requested-model "$model" \
--requested-effort "$effort" >/dev/null
}
json_field() {
python3 -c '
import json, sys
value = json.loads(sys.argv[1])
for part in sys.argv[2].split("."):
value = value[int(part)] if isinstance(value, list) else value[part]
if value is None:
print("null")
elif isinstance(value, bool):
print("true" if value else "false")
else:
print(value)
' "$1" "$2"
}
@test "git attributes preserve transcript bytes and normalize all hashed text inputs" {
run git -C "$REPO_ROOT" check-attr text whitespace -- \
evals/skill-probes/example/fixtures/control-1.txt \
evals/skill-probes/example/fixtures-xhigh-2026-08-04/treatment-2.txt
[ "$status" -eq 0 ]
[ "$(printf '%s\n' "$output" | grep -c ': text: unset$')" -eq 2 ]
[ "$(printf '%s\n' "$output" | grep -c ': whitespace: unset$')" -eq 2 ]
run git -C "$REPO_ROOT" check-attr text eol -- \
evals/skill-probes/example/probe.json \
evals/skill-probes/example/question.md \
evals/skill-probes/example/treatment-prelude.md \
evals/skill-probes/example/fixtures-v3/capture-contract.json \
evals/skill-probes/example/discriminator.sh \
skills/example/SKILL.md \
scripts/probe-skill.sh \
scripts/lib/preamble.sh \
scripts/lib/codex-exec.sh \
scripts/lib/probe-fixture-metadata.py
[ "$status" -eq 0 ]
[ "$(printf '%s\n' "$output" | grep -c ': text: set$')" -eq 10 ]
[ "$(printf '%s\n' "$output" | grep -c ': eol: lf$')" -eq 10 ]
}
@test "replay takes producer provenance and binding from verified fixture metadata" {
make_fixture_set fixtures observed-model low ABSENT ACTION
run bash "$HARNESS" --probe demo --replay
[ "$status" -eq 0 ]
[ "$(json_field "$output" schema)" = "agentops-skill-probe.v3" ]
[ "$(json_field "$output" producer.model)" = "observed-model" ]
[ "$(json_field "$output" producer.effort)" = "low" ]
[ "$(json_field "$output" producer.identity.source)" = "test-override" ]
[ "$(json_field "$output" producer.identity.coverage_eligible)" = "false" ]
[ "$(json_field "$output" producer.threads.0.path)" = "control-1.txt" ]
[ "$(json_field "$output" fixture_set.name)" = "fixtures" ]
[[ "$(json_field "$output" fixture_set.binding_sha256)" == sha256:* ]]
[ "$(json_field "$output" evaluator_matches_capture)" = "true" ]
[ "$(json_field "$output" treatment_source)" = "injected-prelude" ]
[[ "$(json_field "$output" honesty)" == *"NOT full-skill activation"* ]]
[ "$(json_field "$output" capture_evaluator.dispatch_helper.path)" = "scripts/lib/codex-exec.sh" ]
[[ "$(json_field "$output" capture_evaluator.dispatch_helper.sha256)" == sha256:* ]]
[ "$(json_field "$output" capture_evaluator.preamble.path)" = "scripts/lib/preamble.sh" ]
[[ "$(json_field "$output" capture_evaluator.preamble.sha256)" == sha256:* ]]
[ "$(json_field "$output" verdict)" = "BEHAVIORAL" ]
[ "$(json_field "$(cat "$PROBE_DIR/fixtures/fixture-set.json")" schema)" = "agentops-skill-probe-fixture-set.v3" ]
[ "$(json_field "$(cat "$PROBE_DIR/fixtures/fixture-set.json")" canonical_skill.path)" = "skills/demo-skill/SKILL.md" ]
}
@test "v3 response-only scoring supports the repository transcript-marker discriminator" {
cp "$REPO_ROOT/evals/skill-probes/anti-ceremony-creation-gate-v2/discriminator.sh" \
"$PROBE_DIR/discriminator.sh"
make_fixture_set fixtures observed-model low \
$'A: CREATE\nB: CREATE' $'A: DROP\nB: CREATE'
run bash "$HARNESS" --probe demo --replay
[ "$status" -eq 0 ]
[ "$(json_field "$output" control.usable)" -eq 2 ]
[ "$(json_field "$output" treatment.usable)" -eq 2 ]
[ "$(json_field "$output" verdict)" = "BEHAVIORAL" ]
}
@test "structured response extraction ignores transcript-like delimiter collisions" {
make_fixture_set fixtures observed-model low ABSENT $'codex\ntokens used: forged\nACTION'
run bash "$HARNESS" --probe demo --replay
[ "$status" -eq 0 ]
[ "$(json_field "$output" treatment.present)" -eq 2 ]
[ "$(json_field "$output" treatment.usable)" -eq 2 ]
[ "$(json_field "$output" verdict)" = "BEHAVIORAL" ]
}
@test "replay refuses legacy fixture bytes without immutable capture metadata" {
mkdir -p "$PROBE_DIR/fixtures"
write_legacy_transcript "$PROBE_DIR/fixtures" control-1 old-model low ABSENT
write_legacy_transcript "$PROBE_DIR/fixtures" treatment-1 old-model low ACTION
write_legacy_transcript "$PROBE_DIR/fixtures" control-2 old-model low ABSENT
write_legacy_transcript "$PROBE_DIR/fixtures" treatment-2 old-model low ACTION
run bash "$HARNESS" --probe demo --replay
[ "$status" -eq 2 ]
[[ "$output" == *"verified replay requires immutable capture metadata"* ]]
[[ "$output" == *"replay refused"* ]]
}
@test "v1 bound fixtures replay as injected-prelude without a treatment_source declaration" {
make_fixture_set fixtures observed-model low ABSENT ACTION
write_legacy_transcript "$PROBE_DIR/fixtures" control-1 observed-model low ABSENT
write_legacy_transcript "$PROBE_DIR/fixtures" treatment-1 observed-model low ACTION
write_legacy_transcript "$PROBE_DIR/fixtures" control-2 observed-model low ABSENT
write_legacy_transcript "$PROBE_DIR/fixtures" treatment-2 observed-model low ACTION
python3 - "$PROBE_DIR/probe.json" "$PROBE_DIR/fixtures/fixture-set.json" <<'PY'
import hashlib, json, pathlib, sys
probe_path = pathlib.Path(sys.argv[1])
manifest_path = pathlib.Path(sys.argv[2])
probe = json.loads(probe_path.read_text())
probe.pop("treatment_source")
probe_path.write_text(json.dumps(probe) + "\n")
manifest = json.loads(manifest_path.read_text())
manifest["schema"] = "agentops-skill-probe-fixture-set.v1"
manifest["producer"] = {
"adapter": "codex",
"model": "observed-model",
"effort": "low",
}
manifest["evaluation_inputs"] = [
{"path": record["path"], "sha256": record["sha256"]}
for record in manifest.pop("capture_inputs")
]
manifest.pop("capture_contract")
manifest.pop("canonical_skill")
manifest.pop("treatment_source")
manifest.pop("prompts")
manifest.pop("schedule")
manifest.pop("scoring")
manifest["capture_evaluator"].pop("preamble")
manifest["capture_evaluator"].pop("dispatch_helper")
manifest["capture_evaluator"].pop("network_proxy")
for record in manifest["transcripts"]:
transcript_path = manifest_path.parent / record["path"]
record["sha256"] = "sha256:" + hashlib.sha256(transcript_path.read_bytes()).hexdigest()
for record in manifest["evaluation_inputs"]:
if record["path"] == "probe.json":
record["sha256"] = "sha256:" + hashlib.sha256(probe_path.read_bytes()).hexdigest()
payload = {key: value for key, value in manifest.items() if key != "binding_sha256"}
canonical = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode()
manifest["binding_sha256"] = "sha256:" + hashlib.sha256(canonical).hexdigest()
manifest_path.write_text(json.dumps(manifest) + "\n")
manifest_path.with_name("capture-contract.json").unlink()
PY
run bash "$HARNESS" --probe demo --replay
[ "$status" -eq 0 ]
[ "$(json_field "$output" treatment_source)" = "injected-prelude" ]
[[ "$(json_field "$output" honesty)" == *"NOT full-skill activation"* ]]
[ "$(json_field "$output" verdict)" = "BEHAVIORAL" ]
}
@test "replay rejects transcript tampering before discrimination" {
make_fixture_set fixtures observed-model low ABSENT ACTION
printf 'ACTION\n' >> "$PROBE_DIR/fixtures/control-1.txt"
run bash "$HARNESS" --probe demo --replay
[ "$status" -eq 2 ]
[[ "$output" == *"transcript digest mismatch for control-1.txt"* ]]
}
@test "replay rejects producer-config tampering through the fixture-set binding" {
make_fixture_set fixtures observed-model low ABSENT ACTION
python3 - "$PROBE_DIR/fixtures/fixture-set.json" <<'PY'
import json, pathlib, sys
path = pathlib.Path(sys.argv[1])
value = json.loads(path.read_text())
value["requested_producer"]["model"] = "relabeled-model"
path.write_text(json.dumps(value))
PY
run bash "$HARNESS" --probe demo --replay
[ "$status" -eq 2 ]
[[ "$output" == *"requested_producer disagrees with bound producer request"* ]]
}
@test "fixture creation rejects prompt-event tampering before manifest binding" {
local directory="$PROBE_DIR/fixtures"
local rep
mkdir -p "$directory"
python3 "$META_TOOL" snapshot \
--fixture-dir "$directory" --probe-dir "$PROBE_DIR" --skills-dir "$SKILLS" \
--probe demo --requested-model observed-model --requested-effort low \
--producer-override-bin "$RUNTIME_STUB" >/dev/null
for rep in 1 2; do
write_transcript "$directory" "control-$rep" observed-model low ABSENT
write_transcript "$directory" "treatment-$rep" observed-model low ACTION
done
python3 - "$directory/control-1.txt" <<'PY'
import base64, json, pathlib, sys
path = pathlib.Path(sys.argv[1])
events = [json.loads(line) for line in path.read_text().splitlines()]
events[0]["prompt"]["content_base64"] = base64.b64encode(b"TAMPERED\n").decode()
path.write_text("".join(json.dumps(event) + "\n" for event in events))
PY
run python3 "$META_TOOL" create \
--fixture-dir "$directory" --probe-dir "$PROBE_DIR" --skills-dir "$SKILLS" \
--harness "$HARNESS" --preamble "$PREAMBLE" \
--dispatch-helper "$DISPATCH_HELPER" --probe demo --reps 2 \
--requested-model observed-model --requested-effort low
[ "$status" -eq 2 ]
[[ "$output" == *"probe input event does not match bound control-1 prompt"* ]]
[ ! -e "$directory/fixture-set.json" ]
}
@test "fixture creation refuses retroactive capture-contract creation" {
local directory="$PROBE_DIR/fixtures"
mkdir -p "$directory"
write_legacy_transcript "$directory" control-1 observed-model low ABSENT
write_legacy_transcript "$directory" treatment-1 observed-model low ACTION
write_legacy_transcript "$directory" control-2 observed-model low ABSENT
write_legacy_transcript "$directory" treatment-2 observed-model low ACTION
run python3 "$META_TOOL" create \
--fixture-dir "$directory" --probe-dir "$PROBE_DIR" --skills-dir "$SKILLS" \
--harness "$HARNESS" --preamble "$PREAMBLE" \
--dispatch-helper "$DISPATCH_HELPER" --probe demo --reps 2 \
--requested-model observed-model --requested-effort low
[ "$status" -eq 2 ]
[[ "$output" == *"requires a pre-existing capture contract written before execution"* ]]
[ ! -e "$directory/capture-contract.json" ]
[ ! -e "$directory/fixture-set.json" ]
}
@test "v3 replay scores captured discriminator bytes after current probe drift" {
make_fixture_set fixtures observed-model low ABSENT ACTION
printf '# changed scoring input\n' >> "$PROBE_DIR/discriminator.sh"
run bash "$HARNESS" --probe demo --replay
[ "$status" -eq 0 ]
[ "$(json_field "$output" verdict)" = "BEHAVIORAL" ]
}
@test "v3 replay remains self-contained after current canonical skill drift" {
make_fixture_set fixtures observed-model low ABSENT ACTION
printf '\nchanged canonical guidance\n' >> "$SKILLS/demo-skill/SKILL.md"
run bash "$HARNESS" --probe demo --replay
[ "$status" -eq 0 ]
[ "$(json_field "$output" verdict)" = "BEHAVIORAL" ]
}
@test "capture rejects a canonical skill whose declared identity mismatches probe.json" {
sed -i.bak 's/name: demo-skill/name: unrelated-skill/' "$SKILLS/demo-skill/SKILL.md"
rm -f "$SKILLS/demo-skill/SKILL.md.bak"
mkdir -p "$PROBE_DIR/fixtures"
run python3 "$META_TOOL" snapshot \
--fixture-dir "$PROBE_DIR/fixtures" \
--probe-dir "$PROBE_DIR" \
--skills-dir "$SKILLS" \
--probe demo \
--requested-model observed-model \
--requested-effort low \
--producer-override-bin "$RUNTIME_STUB"
[ "$status" -eq 2 ]
[[ "$output" == *"canonical skill identity mismatch"* ]]
[ ! -e "$PROBE_DIR/fixtures/capture-contract.json" ]
}
@test "replay flags are constraints and mismatches fail rather than relabel fixtures" {
make_fixture_set fixtures observed-model low ABSENT ACTION
run bash "$HARNESS" --probe demo --replay --model another-model
[ "$status" -eq 2 ]
[[ "$output" == *"does not match bound fixture producer request"* ]]
run bash "$HARNESS" --probe demo --replay --effort xhigh
[ "$status" -eq 2 ]
[[ "$output" == *"does not match bound fixture producer request"* ]]
run env PROBE_MODEL=ambient-relabel SKILL_PROBES_DIR="$PROBES" \
bash "$HARNESS" --probe demo --replay
[ "$status" -eq 2 ]
[[ "$output" == *"does not match bound fixture producer request"* ]]
}
@test "a zero-usable control arm cannot produce a false BEHAVIORAL verdict" {
make_fixture_set fixtures observed-model low INFRA ACTION
run bash "$HARNESS" --probe demo --replay
[ "$status" -eq 0 ]
[ "$(json_field "$output" control.usable)" = "0" ]
[ "$(json_field "$output" treatment.usable)" = "2" ]
[ "$(json_field "$output" verdict)" = "UNMEASURED" ]
}
@test "both zero-usable arms yield UNMEASURED with null rates" {
make_fixture_set fixtures observed-model low INFRA INFRA
run bash "$HARNESS" --probe demo --replay
[ "$status" -eq 0 ]
[ "$(json_field "$output" control.rate)" = "null" ]
[ "$(json_field "$output" treatment.rate)" = "null" ]
[ "$(json_field "$output" verdict)" = "UNMEASURED" ]
}
@test "named alternate fixture sets are independently selectable and verified" {
make_fixture_set fixtures default-model low ABSENT ACTION
make_fixture_set fixtures-xhigh-2026-08-04 alternate-model xhigh ABSENT ABSENT
run bash "$HARNESS" --probe demo --replay \
--fixtures fixtures-xhigh-2026-08-04 \
--model alternate-model \
--effort xhigh
[ "$status" -eq 0 ]
[ "$(json_field "$output" fixture_set.name)" = "fixtures-xhigh-2026-08-04" ]
[ "$(json_field "$output" producer.model)" = "alternate-model" ]
[ "$(json_field "$output" producer.effort)" = "xhigh" ]
[ "$(json_field "$output" verdict)" = "INERT" ]
}
@test "producer strings are JSON serialized without shell interpolation" {
make_fixture_set fixtures 'gpt-"quoted"' low ABSENT ACTION
run bash "$HARNESS" --probe demo --replay --model 'gpt-"quoted"'
[ "$status" -eq 0 ]
[ "$(json_field "$output" producer.model)" = 'gpt-"quoted"' ]
}
@test "replay discloses when the current evaluator differs from capture" {
make_fixture_set fixtures observed-model low ABSENT ACTION
python3 - "$PROBE_DIR/fixtures/fixture-set.json" <<'PY'
import hashlib, json, pathlib, sys
path = pathlib.Path(sys.argv[1])
value = json.loads(path.read_text())
value["capture_evaluator"]["harness"]["sha256"] = "sha256:" + "0" * 64
payload = {key: item for key, item in value.items() if key != "binding_sha256"}
canonical = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode()
value["binding_sha256"] = "sha256:" + hashlib.sha256(canonical).hexdigest()
path.write_text(json.dumps(value))
PY
run bash "$HARNESS" --probe demo --replay
[ "$status" -eq 0 ]
[ "$(json_field "$output" evaluator_matches_capture)" = "false" ]
[ "$(json_field "$output" capture_evaluator.harness.sha256)" = "sha256:0000000000000000000000000000000000000000000000000000000000000000" ]
[[ "$(json_field "$output" evaluator.harness.sha256)" == sha256:* ]]
}
@test "v3 replay requires the dispatch helper in capture evaluator identity" {
make_fixture_set fixtures observed-model low ABSENT ACTION
python3 - "$PROBE_DIR/fixtures/fixture-set.json" <<'PY'
import hashlib, json, pathlib, sys
path = pathlib.Path(sys.argv[1])
value = json.loads(path.read_text())
value["capture_evaluator"].pop("dispatch_helper")
payload = {key: item for key, item in value.items() if key != "binding_sha256"}
canonical = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode()
value["binding_sha256"] = "sha256:" + hashlib.sha256(canonical).hexdigest()
path.write_text(json.dumps(value))
PY
run bash "$HARNESS" --probe demo --replay
[ "$status" -eq 2 ]
[[ "$output" == *"capture_evaluator must contain exactly"* ]]
[[ "$output" == *"dispatch_helper"* ]]
}
@test "v3 replay requires the sourced preamble in capture evaluator identity" {
make_fixture_set fixtures observed-model low ABSENT ACTION
python3 - "$PROBE_DIR/fixtures/fixture-set.json" <<'PY'
import hashlib, json, pathlib, sys
path = pathlib.Path(sys.argv[1])
value = json.loads(path.read_text())
value["capture_evaluator"].pop("preamble")
payload = {key: item for key, item in value.items() if key != "binding_sha256"}
canonical = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode()
value["binding_sha256"] = "sha256:" + hashlib.sha256(canonical).hexdigest()
path.write_text(json.dumps(value))
PY
run bash "$HARNESS" --probe demo --replay
[ "$status" -eq 2 ]
[[ "$output" == *"capture_evaluator must contain exactly"* ]]
[[ "$output" == *"preamble"* ]]
}
@test "replay discloses dispatch helper drift from a bound v3 capture" {
make_fixture_set fixtures observed-model low ABSENT ACTION
python3 - "$PROBE_DIR/fixtures/fixture-set.json" <<'PY'
import hashlib, json, pathlib, sys
path = pathlib.Path(sys.argv[1])
value = json.loads(path.read_text())
value["capture_evaluator"]["dispatch_helper"]["sha256"] = "sha256:" + "0" * 64
payload = {key: item for key, item in value.items() if key != "binding_sha256"}
canonical = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode()
value["binding_sha256"] = "sha256:" + hashlib.sha256(canonical).hexdigest()
path.write_text(json.dumps(value))
PY
run bash "$HARNESS" --probe demo --replay
[ "$status" -eq 0 ]
[ "$(json_field "$output" evaluator_matches_capture)" = "false" ]
[ "$(json_field "$output" capture_evaluator.dispatch_helper.sha256)" = "sha256:0000000000000000000000000000000000000000000000000000000000000000" ]
[[ "$(json_field "$output" evaluator.dispatch_helper.sha256)" == sha256:* ]]
}
@test "live capture refuses to overwrite an existing immutable fixture set" {
make_fixture_set fixtures stale-model low ABSENT ACTION
cp "$PROBE_DIR/fixtures/fixture-set.json" "$BATS_TEST_TMPDIR/manifest.before"
cp "$PROBE_DIR/fixtures/treatment-1.txt" "$BATS_TEST_TMPDIR/treatment.before"
run --separate-stderr env CODEX_EXEC_BIN=definitely-missing-probe-producer \
SKILL_PROBES_DIR="$PROBES" \
bash "$HARNESS" --probe demo --live --reps 2
[ "$status" -eq 2 ]
[ -z "$output" ]
[[ "$stderr" == *"refusing to overwrite immutable fixture set"* ]]
[[ "$stderr" == *"choose a new --fixtures name"* ]]
cmp "$BATS_TEST_TMPDIR/manifest.before" "$PROBE_DIR/fixtures/fixture-set.json"
cmp "$BATS_TEST_TMPDIR/treatment.before" "$PROBE_DIR/fixtures/treatment-1.txt"
}
@test "publish refuses a fixture destination that appears during dispatch" {
local producer="$BATS_TEST_TMPDIR/codex-publish-race"
local count="$BATS_TEST_TMPDIR/publish-race-count"
local target="$PROBE_DIR/fixtures-raced"
cat > "$producer" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then printf 'codex-cli race-stub\n'; exit 0; fi
n=$(cat "$PROBE_STUB_COUNT" 2>/dev/null || printf '0')
n=$((n + 1))
printf '%s\n' "$n" > "$PROBE_STUB_COUNT"
if [[ "$n" -eq 4 ]]; then
mkdir "$PROBE_RACE_TARGET"
printf 'external-writer\n' > "$PROBE_RACE_TARGET/sentinel"
fi
prompt="$(cat)"
if [[ "$prompt" == *PRELUDE_ACTION* ]]; then response=ACTION; else response=ABSENT; fi
printf '{"type":"thread.started","thread_id":"race-%s"}\n' "$n"
printf '{"type":"turn.started"}\n'
printf '{"type":"item.completed","item":{"id":"item-%s","type":"agent_message","text":"%s"}}\n' "$n" "$response"
printf '{"type":"turn.completed","usage":{"input_tokens":1,"cached_input_tokens":0,"output_tokens":1}}\n'
SH
chmod +x "$producer"
run --separate-stderr env CODEX_EXEC_BIN="$producer" \
PROBE_STUB_COUNT="$count" \
PROBE_RACE_TARGET="$target" \
bash "$HARNESS" --probe demo --live --reps 2 --fixtures fixtures-raced
[ "$status" -eq 1 ]
[ "$(cat "$target/sentinel")" = "external-writer" ]
[ ! -e "$target/fixture-set.json" ]
[ "$(find "$target" -mindepth 1 -maxdepth 1 | wc -l | tr -d ' ')" -eq 1 ]
[[ "$stderr" == *"refusing to replace existing immutable fixture set"* ]]
[[ "$stderr" == *"failed to publish fixture set"* ]]
}
@test "publisher rejects an externally mutated hidden-stage transcript without exposing target" {
make_fixture_set fixtures observed-model low ABSENT ACTION
local stage="$PROBE_DIR/.fixtures-publish-stage"
local target="$PROBE_DIR/fixtures-publish-mutated"
mv "$PROBE_DIR/fixtures" "$stage"
printf 'MUTATED_BEFORE_PUBLISH\n' >> "$stage/treatment-1.txt"
run python3 "$META_TOOL" publish \
--stage-dir "$stage" \
--target-dir "$target" \
--probe-dir "$PROBE_DIR" \
--skills-dir "$SKILLS" \
--probe demo
local publish_status="$status"
[ "$publish_status" -eq 2 ]
[[ "$output" == *"transcript digest mismatch"* ]]
[ ! -e "$target" ]
[ -f "$stage/fixture-set.json" ]
}
@test "publisher never exposes or deletes an externally replaced hidden-stage transcript" {
make_fixture_set fixtures observed-model low ABSENT ACTION
local stage="$PROBE_DIR/.fixtures-replaced-transcript-stage"
local target="$PROBE_DIR/fixtures-replaced-transcript"
local replacement="$BATS_TEST_TMPDIR/replacement-transcript"
mv "$PROBE_DIR/fixtures" "$stage"
printf 'replacement-owner\n' > "$replacement"
mv "$replacement" "$stage/control-1.txt"
run python3 "$META_TOOL" publish \
--stage-dir "$stage" \
--target-dir "$target" \
--probe-dir "$PROBE_DIR" \
--skills-dir "$SKILLS" \
--probe demo
local publish_status="$status"
[ "$publish_status" -eq 2 ]
[[ "$output" == *"transcript digest mismatch"* ]]
[ ! -e "$target" ]
[ "$(cat "$stage/control-1.txt")" = "replacement-owner" ]
[ -f "$stage/fixture-set.json" ]
}
@test "atomic publisher preserves an externally owned destination" {
make_fixture_set fixtures observed-model low ABSENT ACTION
local stage="$PROBE_DIR/.fixtures-raced-stage"
local target="$PROBE_DIR/fixtures-raced-at-rename"
mv "$PROBE_DIR/fixtures" "$stage"
mkdir "$target"
printf 'external-writer\n' > "$target/sentinel"
run python3 "$META_TOOL" publish \
--stage-dir "$stage" \
--target-dir "$target" \
--probe-dir "$PROBE_DIR" \
--skills-dir "$SKILLS" \
--probe demo
local publish_status="$status"
[ "$publish_status" -eq 2 ]
[[ "$output" == *"refusing to replace existing immutable fixture set"* ]]
[ "$(cat "$target/sentinel")" = "external-writer" ]
[ ! -e "$target/fixture-set.json" ]
[ "$(find "$target" -mindepth 1 -maxdepth 1 | wc -l | tr -d ' ')" -eq 1 ]
[ -f "$stage/fixture-set.json" ]
}
@test "publisher rejects a bound evaluator hash that does not match the repository file" {
make_fixture_set fixtures observed-model low ABSENT ACTION
local stage="$PROBE_DIR/.fixtures-bad-evaluator-stage"
local target="$PROBE_DIR/fixtures-bad-evaluator"
mv "$PROBE_DIR/fixtures" "$stage"
python3 - "$stage/fixture-set.json" <<'PY'
import hashlib
import json
import sys
path = sys.argv[1]
manifest = json.load(open(path, encoding="utf-8"))
digest = manifest["capture_evaluator"]["harness"]["sha256"]
manifest["capture_evaluator"]["harness"]["sha256"] = (
digest[:-1] + ("0" if digest[-1] != "0" else "1")
)
payload = dict(manifest)
payload.pop("binding_sha256")
manifest["binding_sha256"] = "sha256:" + hashlib.sha256(
json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
).hexdigest()
with open(path, "w", encoding="utf-8") as handle:
json.dump(manifest, handle, sort_keys=True, separators=(",", ":"))
handle.write("\n")
PY
run python3 "$META_TOOL" publish \
--stage-dir "$stage" \
--target-dir "$target" \
--probe-dir "$PROBE_DIR" \
--skills-dir "$SKILLS" \
--probe demo
[ "$status" -eq 2 ]
[[ "$output" == *"capture evaluator hashes do not match the exact repo-local evaluator files"* ]]
[ ! -e "$target" ]
[ -f "$stage/fixture-set.json" ]
}
@test "publisher rejects current probe inputs that differ from the capture" {
make_fixture_set fixtures observed-model low ABSENT ACTION
local stage="$PROBE_DIR/.fixtures-input-race-stage"
local target="$PROBE_DIR/fixtures-publish-input-mutated"
mv "$PROBE_DIR/fixtures" "$stage"
printf 'MUTATED_BEFORE_PUBLISH\n' >> "$PROBE_DIR/question.md"
run python3 "$META_TOOL" publish \
--stage-dir "$stage" \
--target-dir "$target" \
--probe-dir "$PROBE_DIR" \
--skills-dir "$SKILLS" \
--probe demo
local publish_status="$status"
[ "$publish_status" -eq 2 ]
[[ "$output" == *"current probe inputs differ from the self-contained capture"* ]]
[ ! -e "$target" ]
[ -f "$stage/fixture-set.json" ]
}
@test "failed live dispatch does not publish or score any prior transcript" {
local producer="$BATS_TEST_TMPDIR/codex-failing"
cat > "$producer" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then printf 'codex-cli failing-stub\n'; exit 0; fi
cat >/dev/null
printf 'producer failure\n' >&2
exit 77
SH
chmod +x "$producer"
run --separate-stderr env CODEX_EXEC_BIN="$producer" \
SKILL_PROBES_DIR="$PROBES" \
bash "$HARNESS" --probe demo --live --reps 2 --fixtures fixtures-failed
# An incomplete live dispatch is a FAILED run: nonzero, no scorecard, no
# fixture set, and no capture stage left in the probe directory.
[ "$status" -ne 0 ]
[ -z "$output" ]
[ ! -e "$PROBE_DIR/fixtures-failed" ]
[ -z "$(find "$PROBE_DIR" -mindepth 1 -maxdepth 1 -name '.fixtures-failed.capture.*')" ]
[[ "$stderr" == *"producer failure"* ]]
[[ "$stderr" == *"incomplete live run; fixture set not published"* ]]
}
@test "a partial live run stays UNMEASURED even when each arm has one usable rep" {
local producer="$BATS_TEST_TMPDIR/codex-partial"
local count="$BATS_TEST_TMPDIR/partial-count"
cat > "$producer" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then printf 'codex-cli partial-stub\n'; exit 0; fi
n=$(cat "$PROBE_STUB_COUNT" 2>/dev/null || printf '0')
n=$((n + 1))
printf '%s\n' "$n" > "$PROBE_STUB_COUNT"
if [[ "$n" -gt 2 ]]; then printf 'producer failure\n' >&2; exit 77; fi
prompt="$(cat)"
if [[ "$prompt" == *PRELUDE_ACTION* ]]; then response=ACTION; else response=ABSENT; fi
printf '{"type":"thread.started","thread_id":"partial-%s"}\n' "$n"
printf '{"type":"turn.started"}\n'
printf '{"type":"item.completed","item":{"id":"item-%s","type":"agent_message","text":"%s"}}\n' "$n" "$response"
printf '{"type":"turn.completed","usage":{"input_tokens":1,"cached_input_tokens":0,"output_tokens":1}}\n'
SH
chmod +x "$producer"
run --separate-stderr env CODEX_EXEC_BIN="$producer" PROBE_STUB_COUNT="$count" \
SKILL_PROBES_DIR="$PROBES" \
bash "$HARNESS" --probe demo --live --reps 2 --fixtures fixtures-partial
# Half a capture is not a null result: it exits nonzero and writes nothing,
# so a caller cannot read a partial dispatch as an honest measurement.
[ "$status" -ne 0 ]
[ -z "$output" ]
[ ! -e "$PROBE_DIR/fixtures-partial" ]
[ -z "$(find "$PROBE_DIR" -mindepth 1 -maxdepth 1 -name '.fixtures-partial.capture.*')" ]
[[ "$stderr" == *"incomplete live run"* ]]
}
@test "multi-rep live capture fails closed when canonical input mutates during dispatch" {
python3 - "$PROBE_DIR/probe.json" <<'PY'
import json, pathlib, sys
path = pathlib.Path(sys.argv[1])
value = json.loads(path.read_text())
value["treatment_source"] = "canonical-skill"
path.write_text(json.dumps(value))
PY
local producer="$BATS_TEST_TMPDIR/codex-mutating"
local count="$BATS_TEST_TMPDIR/mutating-count"
cat > "$producer" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then printf 'codex-cli mutating-stub\n'; exit 0; fi
n=$(cat "$PROBE_STUB_COUNT" 2>/dev/null || printf '0')
n=$((n + 1))
printf '%s\n' "$n" > "$PROBE_STUB_COUNT"
prompt="$(cat)"
if [[ "$n" -eq 1 ]]; then printf '\nMUTATED_DURING_CAPTURE\n' >> "$PROBE_MUTATE_FILE"; fi
if [[ "$prompt" == *CANONICAL_ACTION* ]]; then response=ACTION; else response=ABSENT; fi
printf '{"type":"thread.started","thread_id":"mutating-%s"}\n' "$n"
printf '{"type":"turn.started"}\n'
printf '{"type":"item.completed","item":{"id":"item-%s","type":"agent_message","text":"%s"}}\n' "$n" "$response"
printf '{"type":"turn.completed","usage":{"input_tokens":1,"cached_input_tokens":0,"output_tokens":1}}\n'
SH
chmod +x "$producer"
run --separate-stderr env CODEX_EXEC_BIN="$producer" \
PROBE_STUB_COUNT="$count" \
PROBE_MUTATE_FILE="$SKILLS/demo-skill/SKILL.md" \
bash "$HARNESS" --probe demo --live --reps 2 --fixtures fixtures-mutated
[ "$status" -eq 2 ]
[[ "$stderr" == *"live capture inputs changed during dispatch"* ]]
[ ! -e "$PROBE_DIR/fixtures-mutated" ]
}
@test "scorecard output refuses to overwrite existing evidence" {
make_fixture_set fixtures observed-model low ABSENT ACTION
local scorecard="$BATS_TEST_TMPDIR/scorecard.json"
printf 'sentinel\n' > "$scorecard"
run --separate-stderr bash "$HARNESS" --probe demo --replay --output "$scorecard"
[ "$status" -eq 2 ]
[ "$(cat "$scorecard")" = "sentinel" ]
[[ "$stderr" == *"refusing to overwrite immutable scorecard output"* ]]
}
@test "successful live capture publishes transcripts and manifest as one verified set" {
local producer="$BATS_TEST_TMPDIR/codex-success"
local count="$BATS_TEST_TMPDIR/success-count"
cat > "$producer" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then printf 'codex-cli success-stub\n'; exit 0; fi
n=$(cat "$PROBE_STUB_COUNT" 2>/dev/null || printf '0')
n=$((n + 1))
printf '%s\n' "$n" > "$PROBE_STUB_COUNT"
prompt="$(cat)"
if [[ "$prompt" == *PRELUDE_ACTION* ]]; then response=ACTION; else response=ABSENT; fi
printf '{"type":"thread.started","thread_id":"success-%s"}\n' "$n"
printf '{"type":"turn.started"}\n'
printf '{"type":"item.completed","item":{"id":"item-%s","type":"agent_message","text":"%s"}}\n' "$n" "$response"
printf '{"type":"turn.completed","usage":{"input_tokens":1,"cached_input_tokens":0,"output_tokens":1}}\n'
SH
chmod +x "$producer"
run --separate-stderr env CODEX_EXEC_BIN="$producer" PROBE_STUB_COUNT="$count" \
SKILL_PROBES_DIR="$PROBES" \
bash "$HARNESS" --probe demo --live --capture --reps 2 \
--model requested-live --effort high
[ "$status" -eq 0 ]
[ -f "$PROBE_DIR/fixtures/fixture-set.json" ]
[ ! -e "$PROBE_DIR/fixtures/.capture-inputs" ]
[ "$(json_field "$output" producer.model)" = "requested-live" ]
[ "$(json_field "$output" producer.identity.source)" = "test-override" ]
[ "$(json_field "$output" producer.identity.coverage_eligible)" = "false" ]
[ "$(json_field "$output" requested_producer.model)" = "requested-live" ]
[[ "$(json_field "$output" fixture_set.binding_sha256)" == sha256:* ]]
[ "$(json_field "$output" evaluator_matches_capture)" = "true" ]
[ "$(json_field "$output" verdict)" = "BEHAVIORAL" ]
run python3 "$META_TOOL" verify \
--fixture-dir "$PROBE_DIR/fixtures" \
--probe-dir "$PROBE_DIR" \
--skills-dir "$SKILLS" \
--probe demo
[ "$status" -eq 0 ]
}
@test "canonical-skill treatment mode injects the bound SKILL.md rather than the prelude" {
python3 - "$PROBE_DIR/probe.json" <<'PY'
import json, pathlib, sys
path = pathlib.Path(sys.argv[1])
value = json.loads(path.read_text())
value["treatment_source"] = "canonical-skill"
path.write_text(json.dumps(value))
PY
rm "$PROBE_DIR/treatment-prelude.md"
local producer="$BATS_TEST_TMPDIR/codex-canonical"
local count="$BATS_TEST_TMPDIR/canonical-count"
cat > "$producer" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then printf 'codex-cli canonical-stub\n'; exit 0; fi
n=$(cat "$PROBE_STUB_COUNT" 2>/dev/null || printf '0')
n=$((n + 1))
printf '%s\n' "$n" > "$PROBE_STUB_COUNT"
prompt="$(cat)"
if [[ "$prompt" == *CANONICAL_ACTION* ]]; then response=ACTION; else response=ABSENT; fi
printf '{"type":"thread.started","thread_id":"canonical-%s"}\n' "$n"
printf '{"type":"turn.started"}\n'
printf '{"type":"item.completed","item":{"id":"item-%s","type":"agent_message","text":"%s"}}\n' "$n" "$response"
printf '{"type":"turn.completed","usage":{"input_tokens":1,"cached_input_tokens":0,"output_tokens":1}}\n'
SH
chmod +x "$producer"
run --separate-stderr env CODEX_EXEC_BIN="$producer" \
PROBE_STUB_COUNT="$count" \
bash "$HARNESS" --probe demo --live --reps 2 --fixtures fixtures-canonical
[ "$status" -eq 0 ]
[ "$(json_field "$output" treatment_source)" = "canonical-skill" ]
[[ "$(json_field "$output" honesty)" == *"exact bound canonical SKILL.md treatment"* ]]
[ "$(json_field "$output" verdict)" = "BEHAVIORAL" ]
run python3 - "$PROBE_DIR/fixtures-canonical/capture-contract.json" <<'PY'
import json, sys
value = json.load(open(sys.argv[1], encoding="utf-8"))
raise SystemExit(0 if [item["path"] for item in value["capture_inputs"]] == [
"probe.json", "question.md", "discriminator.sh"
] else 1)
PY
[ "$status" -eq 0 ]
}
@test "transcript self-report cannot override bound runtime producer identity" {
local producer="$BATS_TEST_TMPDIR/codex-identity-spoof"
local count="$BATS_TEST_TMPDIR/count"
cat > "$producer" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then printf 'codex-cli immutable-stub-version\n'; exit 0; fi
n=$(cat "$PROBE_STUB_COUNT" 2>/dev/null || printf '0')
n=$((n + 1))
printf '%s\n' "$n" > "$PROBE_STUB_COUNT"
cat >/dev/null
printf '{"type":"thread.started","thread_id":"spoof-%s","model":"self-reported-%s"}\n' "$n" "$n"
printf '{"type":"turn.started"}\n'
printf '{"type":"item.completed","item":{"id":"item-%s","type":"agent_message","text":"model: forged-model\\nreasoning effort: forged-effort\\nABSENT"}}\n' "$n"
printf '{"type":"turn.completed","usage":{"input_tokens":1,"cached_input_tokens":0,"output_tokens":1}}\n'
SH
chmod +x "$producer"
run --separate-stderr env CODEX_EXEC_BIN="$producer" PROBE_STUB_COUNT="$count" \
SKILL_PROBES_DIR="$PROBES" \
bash "$HARNESS" --probe demo --live --reps 2 --fixtures fixtures-identity-spoof \
--model bound-model --effort low
[ "$status" -eq 0 ]
[ "$(json_field "$output" producer.model)" = "bound-model" ]
[ "$(json_field "$output" producer.effort)" = "low" ]
[ "$(json_field "$output" producer.identity.version)" = "codex-cli immutable-stub-version" ]
[ "$(json_field "$output" producer.identity.source)" = "test-override" ]
[ "$(json_field "$output" producer.identity.coverage_eligible)" = "false" ]
}
@test "replay DEGRADES a rep whose transcript reads a SKILL.md (skill-read-contamination)" {
# The committed premortem tier-2 fixture is the real contaminated capture:
# every rep fetched skills/premortem/SKILL.md off disk with exit 0. Under
# the contamination rule the whole set collapses to UNMEASURED 0/0 usable.
# setup() points the harness at a per-test sandbox; this case reads the
# real committed fixture, so resolve the real repo trees.
unset SKILL_PROBES_DIR SKILL_PROBE_SKILLS_DIR
run --separate-stderr bash "$REPO_ROOT/scripts/probe-skill.sh" \
--probe premortem-plan-shape-t2 --replay --fixtures fixtures-low-2026-08-26
[ "$status" -eq 0 ]
[[ "$stderr" == *"skill-read-contamination"* ]]
[[ "$output" == *'"verdict": "UNMEASURED"'* ]]
}
# transcript_message DIR NAME -> the agent_message text of one captured transcript.
transcript_message() {
python3 - "$1/$2.txt" <<'PY'
import json, sys
for line in open(sys.argv[1], encoding="utf-8"):
event = json.loads(line)
item = event.get("item") or {}
if event.get("type") == "item.completed" and item.get("type") == "agent_message":
print(item["text"])
PY
}
# path_without_seatbelt -> a PATH with no executable sandbox-exec anywhere on it:
# drops the pass-through stub dir and replaces any real dir that carries one
# with a symlink farm of that dir minus sandbox-exec.
path_without_seatbelt() {
local dir out="" farm entry
while IFS= read -r dir; do
[[ -n "$dir" && "$dir" != "$SEATBELT_STUB_DIR" ]] || continue
if [[ -x "$dir/sandbox-exec" ]]; then
farm="$BATS_TEST_TMPDIR/no-seatbelt$(printf '%s' "$dir" | tr '/' '_')"
mkdir -p "$farm"
for entry in "$dir"/*; do
[[ "$(basename "$entry")" != "sandbox-exec" ]] || continue
ln -s "$entry" "$farm/$(basename "$entry")"
done
dir="$farm"
fi
out="${out:+$out:}$dir"
done < <(printf '%s\n' "${PATH//:/$'\n'}")
printf '%s' "$out"
}
@test "sealed dispatch gives the rep a scratch HOME and CODEX_HOME with no operator skill roots" {
local fake_codex_home="$BATS_TEST_TMPDIR/real-codex-home"
mkdir -p "$fake_codex_home/skills"
printf '{"token":"fixture"}\n' > "$fake_codex_home/auth.json"
# The operator config the rep must NOT inherit. It is not filtered any more,
# it is not read at all: the rep runs a GENERATED config.
cat > "$fake_codex_home/config.toml" <<'TOML'
model = "operator-model"
web_search = "live"
notify = ["/Users/operator/hook", "turn-ended"]
[mcp_servers.smart-connections]
command = "npx"
args = ["-y", "smart-connections-mcp"]
[projects."/somewhere/else"]
trust_level = "trusted"
TOML
local producer="$BATS_TEST_TMPDIR/codex-seal-env"
cat > "$producer" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then printf 'codex-cli seal-env-stub\n'; exit 0; fi
cat >/dev/null
leak=no-leak
if test -e "$HOME/.agents/skills"; then leak=LEAK; fi
auth=no-auth
if test -f "${CODEX_HOME:-/nonexistent}/auth.json" && ! test -L "${CODEX_HOME}/auth.json"; then
auth="copy:$(cat "$CODEX_HOME/auth.json" | tr -d '\n\"{} ')"
fi
config=no-config
if test -f "${CODEX_HOME:-/nonexistent}/config.toml"; then
config="tables:$(grep -c '^\[' "$CODEX_HOME/config.toml" || true)"
config="$config:websearch:$(grep -c 'web_search = "disabled"' "$CODEX_HOME/config.toml" || true)"
config="$config:operator:$(grep -c 'operator-model\|notify\|mcp_servers' "$CODEX_HOME/config.toml" || true)"
fi
names="$(env | sed 's/=.*//' | sort | tr '\n' ' ')"
text="home=$HOME|codex_home=${CODEX_HOME:-unset}|tmpdir=${TMPDIR:-unset}|$leak|auth=$auth|config=$config|env=$names|cwd=$PWD"
printf '{"type":"thread.started","thread_id":"seal-env-%s"}\n' "$$"
printf '{"type":"turn.started"}\n'
printf '{"type":"item.completed","item":{"id":"item-seal","type":"agent_message","text":"%s"}}\n' "$text"
printf '{"type":"turn.completed","usage":{"input_tokens":1,"cached_input_tokens":0,"output_tokens":1}}\n'
SH
chmod +x "$producer"
run --separate-stderr env CODEX_EXEC_BIN="$producer" CODEX_HOME="$fake_codex_home" \
bash "$HARNESS" --probe demo --live --reps 2 --fixtures fixtures-seal
[ "$status" -eq 0 ]
[ "$(json_field "$output" seal.seal_mode)" = "seatbelt" ]
[ "$(json_field "$output" seal.coverage_eligible)" = "true" ]
[ "$(json_field "$output" verdict)" = "INERT" ]
local text rep_home
text="$(transcript_message "$PROBE_DIR/fixtures-seal" control-1)"
rep_home="${text#home=}"; rep_home="${rep_home%%|*}"
[ -n "$rep_home" ]
[ "$rep_home" != "$HOME" ]
[[ "$rep_home" != "$REPO_ROOT"/* ]]
[[ "$text" == *"|codex_home=$rep_home/.codex|"* ]]
[[ "$text" == *"|no-leak|"* ]]
[[ "$text" != *LEAK* ]]
# auth.json is COPIED, not symlinked: the real home is read-denied, so a
# symlink into it cannot resolve inside the seal.
[[ "$text" == *"|auth=copy:token:fixture|"* ]]
# The generated config carries no table, turns web search off, and holds
# nothing of the operator's file.
[[ "$text" == *"|config=tables:0:websearch:1:operator:0|"* ]]
# The rep's environment is exactly what the seal discloses.
local names
names="${text#*|env=}"; names="${names%%|*}"
local declared
declared="$(python3 -c '
import json, sys
print(" ".join(sorted(json.loads(sys.argv[1])["seal"]["env_allowlist"])))
' "$output")"
local name
for name in $names; do
[[ " $declared " == *" $name "* ]] || {
echo "rep saw undeclared variable: $name" >&2
false
}
done
[[ "$text" != *"|cwd=$REPO_ROOT"* ]]
[ "$(json_field "$output" seal.rep_env.HOME)" = "$rep_home" ]
[ "$(json_field "$output" seal.rep_env.CODEX_HOME)" = "$rep_home/.codex" ]
[[ "$text" == *"|tmpdir=$(json_field "$output" seal.rep_env.TMPDIR)|"* ]]
[ "$(json_field "$output" seal.auth_copied)" = "true" ]
[[ "$(json_field "$output" seal.config_sanitized)" == *web_search* ]]
[ "$(json_field "$output" seal.platform)" = "$(uname -s)" ]
[ "$(json_field "$output" seal.mechanism)" = "sandbox-exec" ]
# The network seal is recorded, and the whole real CODEX_HOME is denied,
# not only its skills directory.
[ "$(json_field "$output" seal.network.mode)" = "proxy-allowlist" ]
[[ "$(json_field "$output" seal.network.proxy)" == 127.0.0.1:* ]]
[[ "$(json_field "$output" seal.network.hosts)" == *chatgpt.com* ]]
[ "$(json_field "$output" seal.real_codex_home)" = "$fake_codex_home" ]
[[ "$(json_field "$output" seal.denied_read_roots)" == *"$fake_codex_home"* ]]
[[ "$(json_field "$output" seal.launcher_chain)" == *"$producer"* ]]
[[ "$(json_field "$output" seal.config_sha256)" == sha256:* ]]
# The bound text is the exact file bytes, trailing newline included, so
# the digest the verifier recomputes from config_text is the recorded one.
python3 -c '
import hashlib, json, sys
seal = json.loads(sys.argv[1])["seal"]
digest = "sha256:" + hashlib.sha256(seal["config_text"].encode("utf-8")).hexdigest()
assert digest == seal["config_sha256"], (digest, seal["config_sha256"])
assert seal["config_text"].endswith("\n"), "config_text lost its trailing newline"
' "$output"
[[ "$(json_field "$output" seal.dev_write_paths)" == *"/dev/null"* ]]
[[ "$(json_field "$output" seal.denied_read_roots)" == *"$HOME/.agents"* ]]
[[ "$(json_field "$output" seal.denied_link_roots)" == *"$(json_field "$output" seal.dispatch_root)"* ]]
[[ "$(json_field "$output" seal.profile_sha256)" == sha256:* ]]
# The run directory is gone with the harness process.
[ ! -e "$(json_field "$output" seal.run_root)" ]
}
@test "seatbelt profile denies the rep every byte the seal claims to deny" {
[[ "$(uname -s)" == "Darwin" && -x /usr/bin/sandbox-exec ]] \
|| skip "seatbelt (sandbox-exec) is macOS-only"
# The REAL kernel sandbox, not the pass-through seam.
unset PROBE_SEAL_SANDBOX_EXEC
export PATH="${PATH#"$SEATBELT_STUB_DIR:"}"
local checkout_skill main_skill cache_dir custom_codex_home
checkout_skill="$(ls "$REPO_ROOT"/skills/*/SKILL.md | head -n 1)"
[ -s "$checkout_skill" ]
# The MAIN checkout this linked worktree shares: same canonical bytes under
# a path the checkout deny alone does not reach (Codex SEAL-ALT-002).
main_skill="$(dirname "$(git -C "$REPO_ROOT" rev-parse --path-format=absolute --git-common-dir)")/skills/$(basename "$(dirname "$checkout_skill")")/SKILL.md"
[ -e "$main_skill" ] || main_skill="$checkout_skill"
cache_dir="$(getconf DARWIN_USER_CACHE_DIR)"
# A CODEX_HOME outside HOME: only its /skills subdirectory was denied before,
# so its sessions and rollouts (which carry canonical text) stayed readable.
custom_codex_home="$BATS_TEST_TMPDIR/outside-codex-home"
mkdir -p "$custom_codex_home/sessions"
printf '{"token":"fixture"}\n' > "$custom_codex_home/auth.json"
printf 'CANONICAL_ACTION rollout\n' > "$custom_codex_home/sessions/rollout.jsonl"
# The producer runs INSIDE the seal and probes it itself: nesting another
# sandbox-exec here would prove nothing, because seatbelt does not nest.
local producer="$BATS_TEST_TMPDIR/codex-seal-read"
cat > "$producer" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then printf 'codex-cli seal-read-stub\n'; exit 0; fi
cat >/dev/null
run="$(dirname "$(dirname "$PROBE_SEAL_PROFILE_FILE")")"
text=""
probe() {
local label="$1" rc=0
shift
"$@" >/dev/null 2>&1 || rc=$?
text+="$label=$rc|"
}
probe checkout cat "$PROBE_SEAL_CHECKOUT_TARGET"
probe main cat "$PROBE_SEAL_MAIN_TARGET"
probe skills cat "$PROBE_SEAL_SKILLS_TARGET"
probe home cat "$PROBE_SEAL_HOME_TARGET"
probe codexhome cat "$PROBE_SEAL_CODEX_HOME_TARGET"
probe cachedir ls "$PROBE_SEAL_CACHE_TARGET"
probe tmproot ls "$PROBE_SEAL_TMPDIR"
probe dispatchls ls "$run/dispatch"
probe dispatchcat cat "$run/dispatch/control-1.prompt"
probe dispatchstat stat -f %z "$run/dispatch/control-1.prompt"
probe proxylog cat "$run/dispatch/network.log"
probe steal mv "$run/dispatch/control-1.prompt" ./stolen
probe hardlink ln "$PROBE_SEAL_CHECKOUT_TARGET" ./linked
probe clone cp -c "$PROBE_SEAL_CHECKOUT_TARGET" ./cloned
ln -s "$PROBE_SEAL_CHECKOUT_TARGET" ./symlinked 2>/dev/null
probe symlink cat ./symlinked
probe write bash -c 'printf ok > ./written'
probe readback cat ./written
probe profile cat "$PROBE_SEAL_PROFILE_FILE"
# No direct egress: --noproxy keeps this off the harness proxy, so it is the
# kernel that refuses it and no CONNECT is attempted. The proxy's own refusal is
# a separate test, because it DEGRADES the rep by design.
probe curldirect /usr/bin/curl -sS -m 6 --noproxy '*' -o /dev/null https://raw.githubusercontent.com/
# Descriptors: only stdio is inherited, and writing to fd 9 fails.
# No harness descriptor is usable: 9 was the open transcript sink, and 3..8
# covered the dispatch handles.
usable=""
for fd in 3 4 5 6 7 8 9; do
if bash -c "printf x >&$fd" 2>/dev/null; then usable="$usable$fd,"; fi
done
text+="usablefds=${usable:-none}|"
printf '{"type":"thread.started","thread_id":"seal-read-%s"}\n' "$$"
printf '{"type":"turn.started"}\n'
printf '{"type":"item.completed","item":{"id":"item-seal","type":"agent_message","text":"%s"}}\n' "$text"
printf '{"type":"turn.completed","usage":{"input_tokens":1,"cached_input_tokens":0,"output_tokens":1}}\n'
SH
chmod +x "$producer"
run --separate-stderr env CODEX_EXEC_BIN="$producer" \
CODEX_HOME="$custom_codex_home" \
PROBE_SEAL_CHECKOUT_TARGET="$checkout_skill" \
PROBE_SEAL_MAIN_TARGET="$main_skill" \
PROBE_SEAL_SKILLS_TARGET="$SKILLS/demo-skill/SKILL.md" \
PROBE_SEAL_HOME_TARGET="$HOME/.codex/config.toml" \
PROBE_SEAL_CODEX_HOME_TARGET="$custom_codex_home/sessions/rollout.jsonl" \
PROBE_SEAL_CACHE_TARGET="$cache_dir" \
PROBE_SEAL_TMPDIR="${TMPDIR:-/tmp}" \
bash "$HARNESS" --probe demo --live --reps 2 --fixtures fixtures-seal-read
[ "$status" -eq 0 ]
[ "$(json_field "$output" seal.sandbox_exec)" = "/usr/bin/sandbox-exec" ]
[ "$(json_field "$output" seal.network.mode)" = "proxy-allowlist" ]
local text
text="$(transcript_message "$PROBE_DIR/fixtures-seal-read" treatment-2)"
# Denied: every root the seal names (the checkout, the shared git root, the
# skills dir, the operator home, a CODEX_HOME outside it, the Darwin cache
# dir and the temp root), the dispatch contents and listing, both laundering
# paths, and both network paths.
local label
for label in checkout main skills home codexhome cachedir tmproot dispatchls \
dispatchcat proxylog steal hardlink clone symlink curldirect; do
[[ "$text" == *"|$label=0|"* ]] && {
echo "seal allowed $label: $text" >&2
false
}
[[ "$text" == *"$label="[1-9]* ]]
done
# Allowed: metadata on dispatch (node stats its stdio at startup), writing
# and reading back inside the workspace, and the profile itself.
[[ "$text" == *"|dispatchstat=0|"* ]]
[[ "$text" == *"|write=0|"* ]]
[[ "$text" == *"|readback=0|"* ]]
[[ "$text" == *"|profile=0|"* ]]
# Only stdio reached the producer: FD 9 was the open transcript sink and the
# dispatch handles sat just above it.
[[ "$text" == *"|usablefds=none|"* ]]
# Nothing this rep did reached the proxy at all.
[[ "$stderr" != *"network-egress"* ]]
}
@test "a rep that reaches for a host off the allowlist is refused and DEGRADED" {
[[ "$(uname -s)" == "Darwin" && -x /usr/bin/sandbox-exec ]] \
|| skip "the network seal needs the real seatbelt"
unset PROBE_SEAL_SANDBOX_EXEC
export PATH="${PATH#"$SEATBELT_STUB_DIR:"}"
# Fable M2V-01: `curl https://raw.githubusercontent.com/.../skills/premortem/SKILL.md`
# inside the profile returned the canonical bytes, because the outer profile
# was `(allow default)` and codex's own sandbox is bypassed inside it. The
# rep's only egress is now the harness proxy, and the proxy allows only the
# bound hosts.
local producer="$BATS_TEST_TMPDIR/codex-egress"
cat > "$producer" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then printf 'codex-cli egress-stub\n'; exit 0; fi
cat >/dev/null
/usr/bin/curl -sS -m 8 -o /dev/null \
https://raw.githubusercontent.com/boshu2/agentops/main/skills/premortem/SKILL.md \
>/dev/null 2>&1 || true
printf '{"type":"thread.started","thread_id":"egress-%s"}\n' "$$"
printf '{"type":"turn.started"}\n'
printf '{"type":"item.completed","item":{"id":"item-egress","type":"agent_message","text":"ABSENT"}}\n'
printf '{"type":"turn.completed","usage":{"input_tokens":1,"cached_input_tokens":0,"output_tokens":1}}\n'
SH
chmod +x "$producer"
run --separate-stderr env CODEX_EXEC_BIN="$producer" \
bash "$HARNESS" --probe demo --live --reps 2 --fixtures fixtures-egress
# Refused egress degrades the rep, so the capture is incomplete: nonzero,
# no scorecard, no fixture set.
[ "$status" -ne 0 ]
[ -z "$output" ]
[[ "$stderr" == *"network-egress"* ]]
[[ "$stderr" == *"raw.githubusercontent.com:443"* ]]
[ ! -e "$PROBE_DIR/fixtures-egress" ]
[ -z "$(find "$PROBE_DIR" -mindepth 1 -maxdepth 1 -name '.fixtures-egress.capture.*')" ]
}
@test "a forked child is reaped and a rep that edits its config is DEGRADED" {
[[ "$(uname -s)" == "Darwin" && -x /usr/bin/sandbox-exec ]] \
|| skip "process-group reaping is exercised against the real seal"
unset PROBE_SEAL_SANDBOX_EXEC
export PATH="${PATH#"$SEATBELT_STUB_DIR:"}"
# A producer that forks a child outliving the turn: the survivor would see
# the NEXT rep's workspace, which is the whole point of resetting one. It
# records its own pid so the test can prove the child is actually gone; the
# earlier version of this test passed while four `/bin/sleep 45` processes
# were still running with ppid 1, because GNU timeout had setpgid'd itself
# and the reap was signalling a group the rep was never in.
local producer="$BATS_TEST_TMPDIR/codex-survivor"
cat > "$producer" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then printf 'codex-cli survivor-stub\n'; exit 0; fi
cat >/dev/null
/bin/sleep 45 &
printf 'SURVIVOR_PID=%s\n' "$!" >&2
printf 'trust_level = "trusted"\n' >> "$CODEX_HOME/config.toml"
printf '{"type":"thread.started","thread_id":"survivor-%s"}\n' "$$"
printf '{"type":"turn.started"}\n'
printf '{"type":"item.completed","item":{"id":"item-survivor","type":"agent_message","text":"ABSENT"}}\n'
printf '{"type":"turn.completed","usage":{"input_tokens":1,"cached_input_tokens":0,"output_tokens":1}}\n'
SH
chmod +x "$producer"
run --separate-stderr env CODEX_EXEC_BIN="$producer" \
bash "$HARNESS" --probe demo --live --reps 2 --fixtures fixtures-survivor
[ "$status" -ne 0 ]
[ -z "$output" ]
# The rep is degraded for BOTH reasons the harness can see.
[[ "$stderr" == *"config-mutated"* ]]
[[ "$stderr" == *"rep-survivor"* ]]
[ ! -e "$PROBE_DIR/fixtures-survivor" ]
# Nothing the producer forked is still running.
local pids pid
pids="$(printf '%s\n' "$stderr" | sed -n 's/.*SURVIVOR_PID=\([0-9][0-9]*\).*/\1/p')"
[ -n "$pids" ]
for pid in $pids; do
! kill -0 "$pid" 2>/dev/null
[ -z "$(ps -o pid= -p "$pid" 2>/dev/null)" ]
done
}
@test "a child that ignores TERM still leaves the rep DEGRADED and dies" {
[[ "$(uname -s)" == "Darwin" && -x /usr/bin/sandbox-exec ]] \
|| skip "process-group reaping is exercised against the real seal"
unset PROBE_SEAL_SANDBOX_EXEC
export PATH="${PATH#"$SEATBELT_STUB_DIR:"}"
# TERM alone is not a reap: a child that traps it has to be escalated to
# KILL, and the rep is degraded either way because it outlived its rep.
local producer="$BATS_TEST_TMPDIR/codex-stubborn"
cat > "$producer" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then printf 'codex-cli stubborn-stub\n'; exit 0; fi
cat >/dev/null
bash -c 'trap "" TERM; sleep 45' &
printf 'SURVIVOR_PID=%s\n' "$!" >&2
printf '{"type":"thread.started","thread_id":"stubborn-%s"}\n' "$$"
printf '{"type":"turn.started"}\n'
printf '{"type":"item.completed","item":{"id":"item-stubborn","type":"agent_message","text":"ABSENT"}}\n'
printf '{"type":"turn.completed","usage":{"input_tokens":1,"cached_input_tokens":0,"output_tokens":1}}\n'
SH
chmod +x "$producer"
run --separate-stderr env CODEX_EXEC_BIN="$producer" \
bash "$HARNESS" --probe demo --live --reps 2 --fixtures fixtures-stubborn
[ "$status" -ne 0 ]
[[ "$stderr" == *"rep-survivor"* ]]
[ ! -e "$PROBE_DIR/fixtures-stubborn" ]
# Nothing the producer forked is still running.
local pids pid
pids="$(printf '%s\n' "$stderr" | sed -n 's/.*SURVIVOR_PID=\([0-9][0-9]*\).*/\1/p')"
[ -n "$pids" ]
for pid in $pids; do
! kill -0 "$pid" 2>/dev/null
[ -z "$(ps -o pid= -p "$pid" 2>/dev/null)" ]
done
}
@test "PROBE_SEAL=none runs unsealed, prints the ineligible notice, and records seal_mode=none" {
local producer="$BATS_TEST_TMPDIR/codex-seal-none"
local seal_copy="$BATS_TEST_TMPDIR/seal-none.json"
cat > "$producer" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then printf 'codex-cli seal-none-stub\n'; exit 0; fi
cat >/dev/null
if [[ ! -e "$PROBE_STUB_SEAL_COPY" ]]; then
cp "$SKILL_PROBES_DIR"/demo/.fixtures-none.capture.*/seal.json "$PROBE_STUB_SEAL_COPY"
fi
printf '{"type":"thread.started","thread_id":"seal-none-%s"}\n' "$$"
printf '{"type":"turn.started"}\n'
printf '{"type":"item.completed","item":{"id":"item-seal","type":"agent_message","text":"home=%s"}}\n' "$HOME"
printf '{"type":"turn.completed","usage":{"input_tokens":1,"cached_input_tokens":0,"output_tokens":1}}\n'
SH
chmod +x "$producer"
run --separate-stderr env PROBE_SEAL=none CODEX_EXEC_BIN="$producer" \
PROBE_STUB_SEAL_COPY="$seal_copy" \
bash "$HARNESS" --probe demo --live --reps 2 --fixtures fixtures-none
[ "$status" -eq 0 ]
[[ "$stderr" == *"seal: none (coverage-ineligible)"* ]]
[ -f "$seal_copy" ]
[ "$(json_field "$(cat "$seal_copy")" seal_mode)" = "none" ]
[ "$(json_field "$(cat "$seal_copy")" coverage_eligible)" = "false" ]
[ "$(json_field "$output" seal.seal_mode)" = "none" ]
[ "$(json_field "$output" seal.coverage_eligible)" = "false" ]
[ "$(transcript_message "$PROBE_DIR/fixtures-none" control-1)" = "home=$HOME" ]
}
@test "live dispatch refuses to run unsealed when no seatbelt binary is usable" {
local producer="$BATS_TEST_TMPDIR/codex-never-run"
local count="$BATS_TEST_TMPDIR/never-run-count"
cat > "$producer" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then printf 'codex-cli never-run-stub\n'; exit 0; fi
printf 'invoked\n' >> "$PROBE_STUB_COUNT"
cat >/dev/null
exit 70
SH
chmod +x "$producer"
# (a) an unusable explicit binary refuses rather than falling back to PATH.
run --separate-stderr env PROBE_SEAL_SANDBOX_EXEC="$BATS_TEST_TMPDIR/absent-sandbox-exec" \
CODEX_EXEC_BIN="$producer" PROBE_STUB_COUNT="$count" \
bash "$HARNESS" --probe demo --live --reps 2 --fixtures fixtures-unsealed
[ "$status" -eq 2 ]
[ -z "$output" ]
[[ "$stderr" == *"PROBE_SEAL_SANDBOX_EXEC is not executable"* ]]
[ ! -e "$count" ]
[ ! -e "$PROBE_DIR/fixtures-unsealed" ]
[ -z "$(find "$PROBE_DIR" -mindepth 1 -maxdepth 1 -name '.fixtures-unsealed.capture.*')" ]
}
@test "live dispatch refuses when no seatbelt binary exists at all" {
[[ -x /usr/bin/sandbox-exec ]] \
&& skip "the system seatbelt is present, so PATH cannot hide it"
local producer="$BATS_TEST_TMPDIR/codex-never-run"
local count="$BATS_TEST_TMPDIR/never-run-count"
cat > "$producer" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then printf 'codex-cli never-run-stub\n'; exit 0; fi
printf 'invoked\n' >> "$PROBE_STUB_COUNT"
cat >/dev/null
exit 70
SH
chmod +x "$producer"
local unsealed_path
unsealed_path="$(path_without_seatbelt)"
run --separate-stderr env -u PROBE_SEAL_SANDBOX_EXEC PATH="$unsealed_path" \
CODEX_EXEC_BIN="$producer" PROBE_STUB_COUNT="$count" \
bash "$HARNESS" --probe demo --live --reps 2 --fixtures fixtures-unsealed2
[ "$status" -eq 2 ]
[[ "$stderr" == *"sandbox-exec"* ]]
[[ "$stderr" == *"PROBE_SEAL=none"* ]]
[ ! -e "$count" ]
}
@test "the seal runs the system seatbelt, not a sandbox-exec earlier on PATH" {
[[ "$(uname -s)" == "Darwin" && -x /usr/bin/sandbox-exec ]] \
|| skip "the PATH-shadow hazard is macOS-only"
# A stub named sandbox-exec sits FIRST on PATH and records that it ran. The
# harness must take /usr/bin/sandbox-exec by absolute path instead: before
# this fix the wrap invoked the bare name while the record claimed the
# system binary, so a shadowing stub would have silently disabled the seal.
local marker="$BATS_TEST_TMPDIR/shadow-marker"
local shadow="$BATS_TEST_TMPDIR/shadow-bin"
mkdir -p "$shadow"
cat > "$shadow/sandbox-exec" <<'SH'
#!/usr/bin/env bash
printf 'shadow ran\n' >> "$PROBE_STUB_SHADOW_MARKER"
while [[ $# -gt 0 ]]; do
case "$1" in -p|-f|-n|-D) shift 2;; *) break;; esac
done
exec "$@"
SH
chmod +x "$shadow/sandbox-exec"
local producer="$BATS_TEST_TMPDIR/codex-shadow"
cat > "$producer" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then printf 'codex-cli shadow-stub\n'; exit 0; fi
cat >/dev/null
printf '{"type":"thread.started","thread_id":"shadow-%s"}\n' "$$"
printf '{"type":"turn.started"}\n'
printf '{"type":"item.completed","item":{"id":"item-shadow","type":"agent_message","text":"ABSENT"}}\n'
printf '{"type":"turn.completed","usage":{"input_tokens":1,"cached_input_tokens":0,"output_tokens":1}}\n'
SH
chmod +x "$producer"
run --separate-stderr env -u PROBE_SEAL_SANDBOX_EXEC PATH="$shadow:$PATH" \
PROBE_STUB_SHADOW_MARKER="$marker" CODEX_EXEC_BIN="$producer" \
bash "$HARNESS" --probe demo --live --reps 2 --fixtures fixtures-shadow
[ "$status" -eq 0 ]
[ "$(json_field "$output" seal.sandbox_exec)" = "/usr/bin/sandbox-exec" ]
[ "$(python3 -c '
import json, sys
print(json.loads(sys.argv[1])["seal"]["wrap"][0])
' "$output")" = "/usr/bin/sandbox-exec" ]
[ ! -e "$marker" ]
}
# probe_input_field DIR NAME FIELD -> one field of the transcript's bound
# probe-input event (the first JSONL line).
probe_input_field() {
python3 - "$1/$2.txt" "$3" <<'PY'
import json, sys
event = json.loads(open(sys.argv[1], encoding="utf-8").readline())
print(event.get(sys.argv[2], "MISSING"))
PY
}
# workspace_covered SCORECARD PATH -> succeeds when PATH (literal or realpath)
# sits under one of the scorecard's seal.writable_roots.
workspace_covered() {
python3 - "$1" "$2" <<'PY'
import json, os, sys
roots = json.loads(sys.argv[1])["seal"]["writable_roots"]
candidates = {sys.argv[2], os.path.realpath(sys.argv[2])}
covered = any(
path == root or path.startswith(root.rstrip("/") + "/")
for path in candidates
for root in roots
)
raise SystemExit(0 if covered else 1)
PY
}
@test "each live rep starts from an emptied workspace and receives the prompt on stdin only" {
# 2026-09-03 defect: every rep's prompt file was written into the ONE shared
# live workspace, so a control rep ran `rg --files`, saw treatment-1.prompt,
# read it, and held the canonical SKILL.md bytes. The prompt is delivered on
# stdin; nothing a rep can list from its cwd may carry it or any sibling.
# The second pass made the workspace ONE path, reset between reps, so the
# seatbelt profile is constant and the contract binds one profile digest.
local producer="$BATS_TEST_TMPDIR/codex-workspace"
cat > "$producer" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then printf 'codex-cli workspace-stub\n'; exit 0; fi
prompt="$(cat)"
if [[ "$prompt" == *QUESTION* ]]; then stdin=stdin-prompt; else stdin=no-stdin-prompt; fi
listing="$(ls -A1 . | tr '\n' ',')"
sibling=none
for candidate in ./*.prompt ./*.jsonl ./*.stderr ./*.txt ./*.json; do
[[ -e "$candidate" ]] || continue
sibling="read:${candidate#./}:$(wc -c < "$candidate" | tr -d ' ')"
break
done
carried=none
[[ -e ./carried-over ]] && carried=LEFTOVER
[[ -e "$HOME/.codex/carried-over" ]] && carried="$carried,HOME-LEFTOVER"
printf 'rep\n' > ./carried-over
printf 'rep\n' > "$HOME/.codex/carried-over"
text="cwd=$PWD|entries=${listing:-empty}|$stdin|carried=$carried|sibling=$sibling"
printf '{"type":"thread.started","thread_id":"workspace-%s"}\n' "$$"
printf '{"type":"turn.started"}\n'
printf '{"type":"item.completed","item":{"id":"item-workspace","type":"agent_message","text":"%s"}}\n' "$text"
printf '{"type":"turn.completed","usage":{"input_tokens":1,"cached_input_tokens":0,"output_tokens":1}}\n'
SH
chmod +x "$producer"
run --separate-stderr env CODEX_EXEC_BIN="$producer" \
bash "$HARNESS" --probe demo --live --reps 2 --fixtures fixtures-workspace
[ "$status" -eq 0 ]
[ "$(json_field "$output" seal.seal_mode)" = "seatbelt" ]
local name text cwd first=""
for name in control-1 treatment-1 control-2 treatment-2; do
text="$(transcript_message "$PROBE_DIR/fixtures-workspace" "$name")"
cwd="${text#cwd=}"; cwd="${cwd%%|*}"
[ -n "$cwd" ]
[[ "$cwd" != "$REPO_ROOT"/* ]]
[[ "$cwd" != "$PROBE_DIR"/* ]]
# (a) the prompt arrives on stdin; the cwd holds no prompt file and no
# other rep's artifacts.
[[ "$text" == *"|entries=empty|"* ]]
[[ "$text" == *"|stdin-prompt|"* ]]
[[ "$text" == *"|sibling=none" ]]
# (b) nothing the previous rep wrote in the workspace or the scratch
# CODEX_HOME survived the reset.
[[ "$text" == *"|carried=none|"* ]]
# (c) one path for the whole capture, so the profile stays constant.
[ -z "$first" ] || [ "$cwd" = "$first" ]
first="$cwd"
# (d) the seal's writable roots cover the workspace, and the bound
# probe-input event records where the rep ran and that it was reset.
workspace_covered "$output" "$cwd"
[ "$(probe_input_field "$PROBE_DIR/fixtures-workspace" "$name" workspace)" = "$cwd" ]
[ "$(probe_input_field "$PROBE_DIR/fixtures-workspace" "$name" workspace_reset)" = "True" ]
done
[ "$(json_field "$output" seal.workspace_root)" = "$first" ]
[ "$(json_field "$output" verdict)" = "INERT" ]
run python3 "$META_TOOL" verify \
--fixture-dir "$PROBE_DIR/fixtures-workspace" \
--probe-dir "$PROBE_DIR" --skills-dir "$SKILLS" --probe demo
[ "$status" -eq 0 ]
}
@test "a group that will not die returns the fatal code, not a degraded rep" {
# The reap has three outcomes and only two used to matter. A process that
# outlived KILL returned the same "something survived" as one that died to
# TERM, so the schedule continued past a group nobody could prove empty.
# The two reap functions are sliced out of the SHIPPED harness, so this
# tests the bytes that run, without adding a seam to the harness for it.
local slice="$BATS_TEST_TMPDIR/reap.sh"
sed -n '/^rep_group_members() {/,/^}/p;/^reap_rep_group() {/,/^}/p' \
"$HARNESS" > "$slice"
[ -s "$slice" ]
grep -q 'reap_rep_group()' "$slice"
local psdir="$BATS_TEST_TMPDIR/ps-immortal"
mkdir -p "$psdir"
# Always reports one member of the queried group, whatever it is: the
# observable shape of a process that survives TERM and KILL.
cat > "$psdir/ps" <<'SH'
#!/usr/bin/env bash
printf '%s 4242\n' "$PROBE_STUB_PGID"
SH
chmod +x "$psdir/ps"
run env PATH="$psdir:$PATH" PROBE_STUB_PGID=31337 bash -c '
set -uo pipefail
. "'"$slice"'"
reap_rep_group 31337
echo "rc=$?"
'
[ "$status" -eq 0 ]
# 2 is the fatal code the dispatcher turns into an aborted capture; 1 would
# only degrade the rep.
[[ "$output" == *"rc=2"* ]]
# And the caller treats 2 as fatal: the dispatcher exits before the reset.
grep -q 'could not prove the .* process group empty; capture aborted' "$HARNESS"
}
@test "an unreadable process table aborts the capture rather than reading as empty" {
[[ "$(uname -s)" == "Darwin" && -x /usr/bin/sandbox-exec ]] \
|| skip "process-group reaping is exercised against the real seal"
unset PROBE_SEAL_SANDBOX_EXEC
export PATH="${PATH#"$SEATBELT_STUB_DIR:"}"
# Piping a failed `ps` into awk printed 0, so "could not look" and "nothing
# there" were the same answer.
local psdir="$BATS_TEST_TMPDIR/ps-broken"
mkdir -p "$psdir"
printf '#!/usr/bin/env bash\nexit 1\n' > "$psdir/ps"
chmod +x "$psdir/ps"
local producer="$BATS_TEST_TMPDIR/codex-psfail"
cat > "$producer" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then printf 'codex-cli psfail-stub\n'; exit 0; fi
cat >/dev/null
printf '{"type":"thread.started","thread_id":"psfail-%s"}\n' "$$"
printf '{"type":"turn.started"}\n'
printf '{"type":"item.completed","item":{"id":"i","type":"agent_message","text":"ABSENT"}}\n'
printf '{"type":"turn.completed","usage":{"input_tokens":1,"cached_input_tokens":0,"output_tokens":1}}\n'
SH
chmod +x "$producer"
run --separate-stderr env PATH="$psdir:$PATH" CODEX_EXEC_BIN="$producer" \
bash "$HARNESS" --probe demo --live --reps 2 --fixtures fixtures-psfail
[ "$status" -ne 0 ]
[[ "$stderr" == *"capture aborted"* ]]
[ ! -e "$PROBE_DIR/fixtures-psfail" ]
}
@test "a child that calls setsid is caught by the run-root residue check" {
[[ "$(uname -s)" == "Darwin" && -x /usr/bin/sandbox-exec ]] \
|| skip "session escape is exercised against the real seal"
command -v lsof >/dev/null 2>&1 || skip "lsof is required for the residue check"
unset PROBE_SEAL_SANDBOX_EXEC
export PATH="${PATH#"$SEATBELT_STUB_DIR:"}"
# A setsid() child leaves the rep's process group entirely, so the group
# reap cannot see it: four such survivors once outlived a capture with only
# `config-mutated` firing. Anything still holding the run root is the same
# defect by another route.
local producer="$BATS_TEST_TMPDIR/codex-setsid"
cat > "$producer" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then printf 'codex-cli setsid-stub\n'; exit 0; fi
cat >/dev/null
python3 -c '
import os, sys, time
if os.fork() == 0:
os.setsid()
os.chdir(sys.argv[1])
sys.stderr.write("SETSID_PID=%d\n" % os.getpid())
sys.stderr.flush()
time.sleep(45)
' "$PWD" &
sleep 0.5
printf '{"type":"thread.started","thread_id":"setsid-%s"}\n' "$$"
printf '{"type":"turn.started"}\n'
printf '{"type":"item.completed","item":{"id":"i","type":"agent_message","text":"ABSENT"}}\n'
printf '{"type":"turn.completed","usage":{"input_tokens":1,"cached_input_tokens":0,"output_tokens":1}}\n'
SH
chmod +x "$producer"
run --separate-stderr env CODEX_EXEC_BIN="$producer" \
bash "$HARNESS" --probe demo --live --reps 2 --fixtures fixtures-setsid
[ "$status" -ne 0 ]
[[ "$stderr" == *"rep-survivor"* ]]
[[ "$stderr" == *"session escape"* ]]
[ ! -e "$PROBE_DIR/fixtures-setsid" ]
local pids pid
pids="$(printf '%s\n' "$stderr" | sed -n 's/.*SETSID_PID=\([0-9][0-9]*\).*/\1/p')"
[ -n "$pids" ]
for pid in $pids; do
[ -z "$(ps -o pid= -p "$pid" 2>/dev/null)" ]
done
}
@test "the rep's environment is exactly the seal's env_allowlist" {
[[ "$(uname -s)" == "Darwin" && -x /usr/bin/sandbox-exec ]] \
|| skip "the env boundary is exercised against the real seal"
unset PROBE_SEAL_SANDBOX_EXEC
export PATH="${PATH#"$SEATBELT_STUB_DIR:"}"
# Emptying the environment from inside the dispatch subshell could not clear
# bash's readonly exports (SHELLOPTS, BASHOPTS, UID, EUID, PPID): they
# reached the producer undeclared. `env -i` is the boundary now.
export SHELLOPTS BASHOPTS 2>/dev/null || true
local producer="$BATS_TEST_TMPDIR/codex-env"
cat > "$producer" <<'SH'
#!/usr/bin/env bash
if [[ "${1:-}" == "--version" ]]; then printf 'codex-cli env-stub\n'; exit 0; fi
cat >/dev/null
names="$(env | sed 's/=.*//' | sort -u | tr '\n' ' ')"
printf '{"type":"thread.started","thread_id":"env-%s"}\n' "$$"
printf '{"type":"turn.started"}\n'
printf '{"type":"item.completed","item":{"id":"i","type":"agent_message","text":"env=%s"}}\n' "$names"
printf '{"type":"turn.completed","usage":{"input_tokens":1,"cached_input_tokens":0,"output_tokens":1}}\n'
SH
chmod +x "$producer"
run --separate-stderr env CODEX_EXEC_BIN="$producer" \
bash "$HARNESS" --probe demo --live --reps 2 --fixtures fixtures-env
[ "$status" -eq 0 ]
local text names declared
text="$(transcript_message "$PROBE_DIR/fixtures-env" control-1)"
names="${text#env=}"
declared="$(python3 -c '
import json, sys
print(" ".join(sorted(json.loads(sys.argv[1])["seal"]["env_allowlist"])))
' "$output")"
# Set equality: nothing the rep sees is undeclared, and nothing declared is
# missing.
run python3 -c '
import sys
seen = set(sys.argv[1].split())
declared = set(sys.argv[2].split())
extra = sorted(seen - declared)
missing = sorted(declared - seen)
print("extra:", extra, "missing:", missing)
raise SystemExit(0 if not extra and not missing else 1)
' "$names" "$declared"
[ "$status" -eq 0 ]
# The readonly exports that used to survive are gone.
[[ "$names" != *SHELLOPTS* ]]
[[ "$names" != *BASHOPTS* ]]
}
Reference in New Issue View Git Blame Copy Permalink