skills/boshu2__agentops
1
0
Fork 0
mirror of https://github.com/boshu2/agentops.git synced 2026-09-14 15:08:13 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
main
boshu2__agentops/.gitignore
T
Bo 1cebfee94d Measure one skill: sealed probe dispatch and the first current ledger row (#1101) 
## Measure one skill: filesystem-sealed probe dispatch and the first
current ledger row

Provenance: the instrument-panel roadmap
(`docs/plans/2026-08-26-instrument-panel-roadmap.md`) closed out with
zero skills measured because control-arm reps read
`skills/<skill>/SKILL.md` from the checkout and the operator's skill
roots mid-run. Its named successor was filesystem-sealed dispatch. This
PR builds the seal, then uses it, and the instrument caught one more
leak on the way.

**The seal.** Codex's own seatbelt does not nest inside an outer
`sandbox-exec` profile, so a rep now runs under an outer profile with
Codex's sandbox bypassed for exactly that reason: `file-read*` denied on
the checkout and every skill root (`~/.agents`, `~/.claude/skills`,
`~/.gemini/skills`, `~/.codex/skills`), `file-write*` denied everywhere
except the rep's workspace, a scratch HOME, and the temp roots. HOME and
CODEX_HOME are scratch (auth symlinked). Dispatch refuses without
`sandbox-exec` unless `PROBE_SEAL=none` is set, which marks the run
coverage-ineligible. The seal is recorded in the capture contract
(`agentops-skill-probe-capture.v3`, a `seal` block), coverage
eligibility requires `seal.mode == seatbelt`, and `verify-scorecard`
re-checks that the denied roots include the repository and the four
skill roots. Existing 08-26 sets become `legacy-unsealed` and
ineligible, which changes nothing they were counting.

**The leak the seal did not cover.** The first sealed capture showed the
xhigh control-2 rep listing the shared live workspace, finding
`treatment-1.prompt`, and reading it; the treatment prompt embeds the
skill. So every rep now gets its own empty workspace, prompts reach a
rep on stdin only, raw JSONL and stderr live in a harness-private
dispatch directory the rep can only stat, and a `sibling-prompt-read`
trap degrades any rep whose successful command names a prompt, contract,
seal, or stage file or whose output lists a sibling's artifacts.
Replaying the contaminated set under the new trap turns its REGRESSIVE
into INERT with the contaminated rep degraded; that set is superseded
and not committed.

**The second pass, after the judges read the first.** A Fable validator
passed the first sealed capture with four findings and a Codex read
failed it with seven, all real: the whole per-user temp tree stayed
readable (hundreds of stale probe directories held prompt files with the
canonical skill bytes), sibling checkouts and the real `~/.codex`
sessions were readable because seatbelt matches the traversed path and
the skill roots are symlinks into the main checkout, the dispatch
directory was writable so a rep could rename an earlier prompt into its
workspace, the contract trusted asserted seal metadata (a hand-made
Linux record was coverage-eligible), the only real-Darwin test nested
seatbelt and proved nothing, and the operator's `config.toml` was
symlinked in whole, so every rep started the operator's MCP servers. The
seal is now one run directory removed on exit (`home/ ws/ tmp/
dispatch/`, reset per rep); reads are denied under the real HOME, the
real TMPDIR, `/tmp`, the checkout, the git common directory, and each
skill root's resolved target; `dispatch/` is write-denied with
metadata-only reads; `file-link` and `file-clone` are denied on every
denied root; the rep's config is rebuilt from top-level scalars only and
`auth.json` is copied, not linked. The contract binds platform,
mechanism, wrapper, the denied and allowed roots, the rep environment,
and the sanitization, and coverage refuses any record that omits them;
the Darwin test performs its reads inside the seal. One recorded hole
remains by necessity: the codex launcher under `~/.local/bin` and
`~/dev/dotfiles/bin` is re-allowed by literal path, and coverage refuses
any allowed path inside the checkout.

**The third pass, after the judges read the second.** Both judges failed
the hardened tip again, and the headline was network: bypassing codex's
own sandbox to nest the outer seal had dropped every inner deny, not
only the filesystem ones, so a rep could fetch the canonical skill from
GitHub while the disk was sealed. A rep now reaches the network only
through a harness-owned CONNECT proxy
(`scripts/lib/probe-connect-proxy.py`) with a host allowlist discovered
from real runs (`chatgpt.com`, `ab.chatgpt.com`, the rotating
`.oaiusercontent.com` suffix, and the API-key hosts), behind a seatbelt
`network*` deny that admits only the proxy; every attempt is logged, a
refused one degrades the rep, and web search is disabled in the rep's
config. The profile is now rendered from the bound seal block by one
function, so `verify-scorecard` reconstructs it and requires the digest
to match; the recorded launcher chain is the only allowed read under a
denied root; the wrap invokes `/usr/bin/sandbox-exec` by absolute path;
the real CODEX_HOME and the Darwin cache directory are denied and bound;
every non-stdio descriptor is closed and `/dev` writes narrowed to four
devices; each rep runs in its own process group that must be empty
before the next reset; one minimal config is generated per run, bound by
text and digest, and re-parsed after each rep; one guarded trap covers
the run root and any unpublished stage; and the coverage gate prints an
eligibility row for every ledger row that names a scorecard. What stays
open is stated in the README: allow-default still governs what the
profile does not deny.

**The fourth pass, after the judges read the third.** Both judges failed
the third tip on properties the record carried but the verifier did not
check. GNU `timeout` sets its own process group and ran outside the
seal, so the per-rep reap was vacuous (four sleeping survivors after the
test passed); the seal is now the outermost process, timeout runs inside
it with `--foreground`, its absolute path is bound, and the reap counts
survivors before signalling. The verifier now pins, not merely records:
the egress host set (a module constant), port 443 only, no unix sockets,
a loopback proxy, the generated config text equal to the renderer's
output, the real CODEX_HOME and cache root inside the denied roots,
writable roots under the run root, the four device paths, the
environment allowlist, the launcher chain's digest against the producer
identity, and it refuses unknown record fields. The proxy logs an
attempt before dialing, captures the rep at accept time, refuses names
that resolve to private or loopback ranges, and its source joins the
evaluator hash set; the per-rep egress log is published with the fixture
set. Withdrawn ledger rows get their own eligibility reason. The README
lists which seal fields the verifier pins and which it only records; the
cross-capture aggregate is recounted to 4 of 14. Still open and
disclosed: Mach IPC and Apple Events beyond the filesystem and network
seals, exact-byte environment sealing beyond the allowlist, and cleanup
ownership by inode.

**The fifth pass, after the judges read the fourth.** Both judges found
the seal, proxy, reap ordering, transcripts, and scorecards sound and
failed the tip on claims the tree did not hold: the egress log the docs
called published was caught by the repository's `*.log` ignore rule, a
`setsid()` child escaped the process-group reap, and one ledger sentence
said no rep ran a command when a control rep had run `rg` over its empty
workspace. The log is now tracked and required, strictly parsed, with
both allowed and refused counts recomputed per rep; the reap is fatal to
the capture when it cannot prove emptiness and an `lsof` sweep of the
run root catches a session escape; the rep launches through a real `env
-i` boundary; the timeout budget, the launcher chain's adjacency, every
required root in both read and link denies, the evaluator match, and the
trust-table value are pinned by the verifier; and every sentence in the
ledger, README, and runbook names the check that enforces it. The
remaining gap is stated: a child that both escapes the session and holds
nothing under the run root is not detected.

**The sixth pass, after CI read the fifth.** The first CI run of this PR
failed on Linux: the launcher-chain pin walked the capturing Mac's
filesystem, so on the runner both sets read as unsealed and the coverage
headline fell to zero. A pin that holds on one host is not a pin. The
chain is now bound as structure in the seal record (each link's path,
kind, symlink target, and the final file's digest, with the invoked path
as the head), the verifier checks that record on any host, and it
cross-checks the live filesystem only where the head exists, reporting
which happened. A gate-level test runs the coverage gate twice, once
with the chain made absent, and requires byte-identical output.

**The measurement.** `premortem-plan-shape-t2` (two seeded structural
defects, band [1,4]) captured seven times on 2026-09-03, once under each
seal; the first six sets were deleted with their scorecards when a judge
or CI broke that seal. The rows that count are the seventh capture: low
INERT (control 0/2, treatment 0/2) and xhigh INERT (control 1/2,
treatment 1/2); no rep ran a command, no egress was refused, both
per-rep egress logs are tracked and verified, and both scorecards verify
with every pinned field, the launcher chain checked as a record and
cross-checked on the capturing host. The superseded captures read, at
low, BEHAVIORAL 1/1, INERT, INERT, BEHAVIORAL 1/2, INERT, BEHAVIORAL 1/2
and, at xhigh, INERT, BEHAVIORAL 2/2, INERT, INERT, INERT (control 1/2),
INERT. The reversals across captures are recorded as an unresolved N=2
observation, not variance; across all seven the treatment arm put the
marks in band in 7 of 26 usable reps and the control arm in 2 of 28, and
the ledger marks that aggregate as disclosure, not a row. Headroom is
SEPARATED at both levels. This is the first current, manifest-backed
ledger row; it records response-shape behavior change, never quality
uplift, and what it records today is INERT at both levels.

**Evidence on the tip.** Probe, exec-lib, metadata, coverage, headroom,
and adapter bats suites green with the Darwin seal, survivor,
session-escape, and environment-boundary tests running;
`verify-scorecard` on both scorecards; `check-skill-probe-coverage.sh`
reports measured 1 of 12 with the 2026-08-26 row labelled
`verdict-withdrawn`; regen check clean; full gate green with a
HEAD-built binary; CI's bats command green; Go bar, lint, security, and
the gate-tightening ratchet green.

**Judgment, stated plainly.** The caller elected Fable 5.1 as the
binding validator. Its fifth-round verdict on the pre-CI tip was PASS
with no verdict-driving findings; its read of the portability delta
failed on one test-only point (the coverage test had dropped the
`measured` pin that CI tripped on) which this tip restores. The
cross-family Codex reads of the same tips are FAIL. They closed the
network seal, the environment boundary, the fatal reap, the data roots,
the config binding, and the documentation, and they keep these open,
recorded here as the next pass rather than hidden: the evaluator
identity check still accepts the pre-proxy four-file shape for legacy
sets; an inherited absolute `CODEX_EXEC_TIMEOUT_BIN` is trusted without
a probe; a refusal logged under a rep label outside the expected set is
not rejected; an empty top-level `[projects]` table passes the drift
check; the `lsof` sweep treats a failing `lsof` as a clean result; and
the launcher chain, now bound as a record that any host can check, is
corroborated against the filesystem only on the capturing host, so a
record whose head does not exist there is taken on its word (the docs
say so; Codex wants the live walk required at capture and the invoked
head bound into the producer identity). Fable also noted a test that
mis-resolves when TMPDIR sits under `/private/tmp`, a `2026-09-04` date
in one refusal message, and that the low control replies used a `S2:
FINDING <reason>` shape the discriminator scores as "plan not walked"
rather than by band (outcome-invariant). Each of those changes a harness
or verifier file and would orphan the two rows again, so they are filed
as the follow-up instead of an eighth capture today. Across six judging
rounds the open set shrank only from round four on; the convergence
law's stop signal was reached at round three and the run continued on
the caller's instruction to land the instrument. The two ledger rows are
honest about what they measure, and the seal they ran under is the one
this tree can verify on any host.
2026-09-04 01:40:41 +00:00

200 lines
3.8 KiB
Plaintext
Raw Permalink Blame History

# graphify knowledge-graph output — one-shot audit artifact, not a tracked
# repo surface (34MB graph.json + chunk JSONs, stale within days on a hot repo).
# Re-run on demand with the /graphify skill; the standing query surface is `ao inject`.
graphify-out/
# macOS
.DS_Store
.AppleDouble
.LSOverride
# Editor directories and files
.vscode/
.idea/
*.swp
*.swo
*~
# Node modules (if you add tooling)
node_modules/
npm-debug.log*
# Build outputs
dist/
build/
*.log
# ...except the skill-probe egress logs. They are bound evidence: every rep binds
# the digest of its own lines, and the verifier recomputes both counts from this
# file, so a set whose log the tree ignores has a digest with nothing to check.
!evals/skill-probes/**/network.log
# Python
.venv/
venv/
env/
__pycache__/
*.py[cod]
*$py.class
*.egg-info/
.pytest_cache/
.coverage
.coverage.*
htmlcov/
.mypy_cache/
.ruff_cache/
coverage.xml
# Generated reports
reports/
/release-notes.md
wiki/
# Agent runtime state
# Repo-root .agents/ is local/private runtime state for Codex, Claude,
# AgentOps, and other agents. It must never be tracked in git — except the
# AgentOps project config, which pins repo-wide settings (e.g. tracker=br)
# and must survive fresh clones.
/.agents/*
!/.agents/ao/
/.agents/ao/*
!/.agents/ao/config.yaml
tests/claude-code/.agents/
# Allow test fixtures under .agents/ paths
!cli/cmd/ao/testdata/**/.agents/
!cli/cmd/ao/testdata/**/.agents/**
# Runtime state
state.json
.runtime/
.claude/memory.md
# MkDocs build output + local venv
_site/
.venv-docs/
# Local temp artifacts (generated during skill self-tests / hooks)
.tmp/
skills/.tmp/
.x/
cli/.x/
hook-output.txt
skills/vibe/README.md
# Go binaries (built locally)
cli/ao
cli/bin/ao
cli/bin/witness-crosscheck
cli/bin/probe-headroom
cli/cmd/ao/ao
cli/cmd/ao/ao-test-bin
cli/coverage.out
cli/coverage.html
# Go test binaries
*.test
# CLI local state (generated by ao runs in cli/ workdir)
cli/.agents/
# Agent Farm temp files
.farm.meta
.witness.pid
.witness.heartbeat
.witness/
# Gas Town (added by gt)
.claude/
.logs/
.beads/*
!.beads/identity.toml
# Session artifacts (generated by agents)
HANDOFF_SUMMARY.md
# Dolt database files (added by bd init)
.dolt/
*.db
.coverdata/
# Coverage artifacts
cli/*cov*.out
cli/*cov*.html
cli/cmd/ao/.tmp
MEMORY.md
# Beads / Dolt files (added by bd init)
.beads-credential-key
# Local agent-runtime scratch written by `gc` at a repository root.
# The managed deployment template under deploy/gc is product source.
.gc/
city.toml
!deploy/gc/city.toml
city.toml.bak
.doctor/
# Release evidence/output dir generated by the release CI (pre-publish-evidence
# SBOM + security-gate summary). Must be ignored so GoReleaser's --clean dirty-state
# check passes in the publish job.
release-artifacts/
ag-1aav-primitive-selection/
ag-mjlg-materialize/
ag-hlng-regen-docs/
ag-vb3q-authoring/
ag-dxqy/
ag-lkxx/
ag-z8c4/
ag-sjz8-gut-poison/
egpu1-schema-v2/
ag-b4p9-prepublish/
ag-okfu-judgment/
ag-maaa-r3/
ag-32xk-autosnap/
wt-tcf0-1/
wt-ag-469h/
wt-mm6q/
ag-2zf8/
# bd worktree
ag-60jd/
# bd worktree
wt-soc-xn5s/
# bd worktree
ag-dclv/
# Beads / Dolt files (added by bd init)
.beads/proxieddb/
# bv (beads viewer) local config and caches
.bv/
# bd worktree
ag-ni1b-agy-consolidated/
# Local orchestration/session artifacts.
.ntm/
test_budget.env
thread_output.md
# Local bd/git worktrees kept inside the checkout; reconciled by cp-snx.
wt-ag-28dp/
wt-ag-ddrb/
wt-ag-ni1b/
wt-ag-y4ac/
wt-gofmt/
# bd worktree
wt-ag-pj51/
# bd worktree
wt-ag-if7p/
# br tracker ledger — PRIVATE nested repo (boshu2/agentops-beads); never commit to the public repo
_beads/
# foreign nested worktree (recon F5)
wt-ag-qidx/
wt-*/
# build binary (compiled, not source) + run-output dumps
bin/ao
outputs/
# Provenance ledger advisory-lock sidecar (cross-process append lock; never committed)
docs/provenance/*.lock
Reference in New Issue View Git Blame Copy Permalink