mirror of
https://github.com/boshu2/agentops.git
synced 2026-09-14 15:08:13 +08:00
main
10 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Bo
|
8061085c89 |
Ship native evidence helpers and fresh-family review defaults (#1110)
AO now performs intent snapshots, subject manifests, strict evidence verification, atomic verdict storage, and orphan inspection through the Go binary. The command handler keeps verification separate from presentation so it meets the existing complexity limit. These operations preserve the existing evidence formats, require explicit protected storage where applicable, and run outside a checkout without Python. The unchanged Python implementation remains a developer oracle; agents still provide semantic judgment. Codex and Claude skills now default to a fresh reviewer from the author’s model family. Callers can explicitly request cross-model review or pin its model. Reviewer adapters use a finite caller timeout or remaining deadline instead of a fixed ten-minute default, while retaining output limits and abnormal-termination cleanup. Validation: Go build, vet, tests and race/shuffle tests; 1,334 shell tests; aggregate runner; regeneration check; 72 full-mode gates. Independent checks exercised 84 storage-boundary rejections and 21 evidence operations with an empty PATH. Both canonical and generated RPI reference suites pass all 48 tests after updating the migrated oracle import without weakening assertions. Change-sensitive checks explicitly compare the final committed candidate with the original PR base. Linux, Windows, installer, security, and required summary checks are green. |
||
|
Bo
|
baa24e1b04 |
Bound headless execution and require acceptance-linked repair progress (#1107)
Headless invocations could run without a finite bound, and repair loops could count changed digests or finding counts as progress. The existing adapter now enforces positive finite deadlines, an optional inherited absolute deadline, bounded output capture and cleanup of its owned process group, refusing execution when protection is unavailable. The existing goal, RPI and validation skills now require acceptance-linked progress, distinguish newly discovered failures from regressions, and bound Andon recovery. Review effort follows the effect on acceptance and enforcement. These contracts do not claim native aggregate goal-budget or pause enforcement. Validation on the clean main baseline: - 1,317 Bats cases, zero failures, two skips; 46 RPI reference cases. - Go build, vet, tests and race/shuffle tests; local aggregate runner. - Generated projections current; all 72 AO gates pass. - Fresh independent Astra and Fable 5.1 xhigh judgments passed the exact clean source. Earlier uncommitted CDLC adoption, private work/evidence, and unrelated Go changes are excluded from this PR. |
||
|
Bo
|
1cebfee94d |
Measure one skill: sealed probe dispatch and the first current ledger row (#1101)
## Measure one skill: filesystem-sealed probe dispatch and the first current ledger row Provenance: the instrument-panel roadmap (`docs/plans/2026-08-26-instrument-panel-roadmap.md`) closed out with zero skills measured because control-arm reps read `skills/<skill>/SKILL.md` from the checkout and the operator's skill roots mid-run. Its named successor was filesystem-sealed dispatch. This PR builds the seal, then uses it, and the instrument caught one more leak on the way. **The seal.** Codex's own seatbelt does not nest inside an outer `sandbox-exec` profile, so a rep now runs under an outer profile with Codex's sandbox bypassed for exactly that reason: `file-read*` denied on the checkout and every skill root (`~/.agents`, `~/.claude/skills`, `~/.gemini/skills`, `~/.codex/skills`), `file-write*` denied everywhere except the rep's workspace, a scratch HOME, and the temp roots. HOME and CODEX_HOME are scratch (auth symlinked). Dispatch refuses without `sandbox-exec` unless `PROBE_SEAL=none` is set, which marks the run coverage-ineligible. The seal is recorded in the capture contract (`agentops-skill-probe-capture.v3`, a `seal` block), coverage eligibility requires `seal.mode == seatbelt`, and `verify-scorecard` re-checks that the denied roots include the repository and the four skill roots. Existing 08-26 sets become `legacy-unsealed` and ineligible, which changes nothing they were counting. **The leak the seal did not cover.** The first sealed capture showed the xhigh control-2 rep listing the shared live workspace, finding `treatment-1.prompt`, and reading it; the treatment prompt embeds the skill. So every rep now gets its own empty workspace, prompts reach a rep on stdin only, raw JSONL and stderr live in a harness-private dispatch directory the rep can only stat, and a `sibling-prompt-read` trap degrades any rep whose successful command names a prompt, contract, seal, or stage file or whose output lists a sibling's artifacts. Replaying the contaminated set under the new trap turns its REGRESSIVE into INERT with the contaminated rep degraded; that set is superseded and not committed. **The second pass, after the judges read the first.** A Fable validator passed the first sealed capture with four findings and a Codex read failed it with seven, all real: the whole per-user temp tree stayed readable (hundreds of stale probe directories held prompt files with the canonical skill bytes), sibling checkouts and the real `~/.codex` sessions were readable because seatbelt matches the traversed path and the skill roots are symlinks into the main checkout, the dispatch directory was writable so a rep could rename an earlier prompt into its workspace, the contract trusted asserted seal metadata (a hand-made Linux record was coverage-eligible), the only real-Darwin test nested seatbelt and proved nothing, and the operator's `config.toml` was symlinked in whole, so every rep started the operator's MCP servers. The seal is now one run directory removed on exit (`home/ ws/ tmp/ dispatch/`, reset per rep); reads are denied under the real HOME, the real TMPDIR, `/tmp`, the checkout, the git common directory, and each skill root's resolved target; `dispatch/` is write-denied with metadata-only reads; `file-link` and `file-clone` are denied on every denied root; the rep's config is rebuilt from top-level scalars only and `auth.json` is copied, not linked. The contract binds platform, mechanism, wrapper, the denied and allowed roots, the rep environment, and the sanitization, and coverage refuses any record that omits them; the Darwin test performs its reads inside the seal. One recorded hole remains by necessity: the codex launcher under `~/.local/bin` and `~/dev/dotfiles/bin` is re-allowed by literal path, and coverage refuses any allowed path inside the checkout. **The third pass, after the judges read the second.** Both judges failed the hardened tip again, and the headline was network: bypassing codex's own sandbox to nest the outer seal had dropped every inner deny, not only the filesystem ones, so a rep could fetch the canonical skill from GitHub while the disk was sealed. A rep now reaches the network only through a harness-owned CONNECT proxy (`scripts/lib/probe-connect-proxy.py`) with a host allowlist discovered from real runs (`chatgpt.com`, `ab.chatgpt.com`, the rotating `.oaiusercontent.com` suffix, and the API-key hosts), behind a seatbelt `network*` deny that admits only the proxy; every attempt is logged, a refused one degrades the rep, and web search is disabled in the rep's config. The profile is now rendered from the bound seal block by one function, so `verify-scorecard` reconstructs it and requires the digest to match; the recorded launcher chain is the only allowed read under a denied root; the wrap invokes `/usr/bin/sandbox-exec` by absolute path; the real CODEX_HOME and the Darwin cache directory are denied and bound; every non-stdio descriptor is closed and `/dev` writes narrowed to four devices; each rep runs in its own process group that must be empty before the next reset; one minimal config is generated per run, bound by text and digest, and re-parsed after each rep; one guarded trap covers the run root and any unpublished stage; and the coverage gate prints an eligibility row for every ledger row that names a scorecard. What stays open is stated in the README: allow-default still governs what the profile does not deny. **The fourth pass, after the judges read the third.** Both judges failed the third tip on properties the record carried but the verifier did not check. GNU `timeout` sets its own process group and ran outside the seal, so the per-rep reap was vacuous (four sleeping survivors after the test passed); the seal is now the outermost process, timeout runs inside it with `--foreground`, its absolute path is bound, and the reap counts survivors before signalling. The verifier now pins, not merely records: the egress host set (a module constant), port 443 only, no unix sockets, a loopback proxy, the generated config text equal to the renderer's output, the real CODEX_HOME and cache root inside the denied roots, writable roots under the run root, the four device paths, the environment allowlist, the launcher chain's digest against the producer identity, and it refuses unknown record fields. The proxy logs an attempt before dialing, captures the rep at accept time, refuses names that resolve to private or loopback ranges, and its source joins the evaluator hash set; the per-rep egress log is published with the fixture set. Withdrawn ledger rows get their own eligibility reason. The README lists which seal fields the verifier pins and which it only records; the cross-capture aggregate is recounted to 4 of 14. Still open and disclosed: Mach IPC and Apple Events beyond the filesystem and network seals, exact-byte environment sealing beyond the allowlist, and cleanup ownership by inode. **The fifth pass, after the judges read the fourth.** Both judges found the seal, proxy, reap ordering, transcripts, and scorecards sound and failed the tip on claims the tree did not hold: the egress log the docs called published was caught by the repository's `*.log` ignore rule, a `setsid()` child escaped the process-group reap, and one ledger sentence said no rep ran a command when a control rep had run `rg` over its empty workspace. The log is now tracked and required, strictly parsed, with both allowed and refused counts recomputed per rep; the reap is fatal to the capture when it cannot prove emptiness and an `lsof` sweep of the run root catches a session escape; the rep launches through a real `env -i` boundary; the timeout budget, the launcher chain's adjacency, every required root in both read and link denies, the evaluator match, and the trust-table value are pinned by the verifier; and every sentence in the ledger, README, and runbook names the check that enforces it. The remaining gap is stated: a child that both escapes the session and holds nothing under the run root is not detected. **The sixth pass, after CI read the fifth.** The first CI run of this PR failed on Linux: the launcher-chain pin walked the capturing Mac's filesystem, so on the runner both sets read as unsealed and the coverage headline fell to zero. A pin that holds on one host is not a pin. The chain is now bound as structure in the seal record (each link's path, kind, symlink target, and the final file's digest, with the invoked path as the head), the verifier checks that record on any host, and it cross-checks the live filesystem only where the head exists, reporting which happened. A gate-level test runs the coverage gate twice, once with the chain made absent, and requires byte-identical output. **The measurement.** `premortem-plan-shape-t2` (two seeded structural defects, band [1,4]) captured seven times on 2026-09-03, once under each seal; the first six sets were deleted with their scorecards when a judge or CI broke that seal. The rows that count are the seventh capture: low INERT (control 0/2, treatment 0/2) and xhigh INERT (control 1/2, treatment 1/2); no rep ran a command, no egress was refused, both per-rep egress logs are tracked and verified, and both scorecards verify with every pinned field, the launcher chain checked as a record and cross-checked on the capturing host. The superseded captures read, at low, BEHAVIORAL 1/1, INERT, INERT, BEHAVIORAL 1/2, INERT, BEHAVIORAL 1/2 and, at xhigh, INERT, BEHAVIORAL 2/2, INERT, INERT, INERT (control 1/2), INERT. The reversals across captures are recorded as an unresolved N=2 observation, not variance; across all seven the treatment arm put the marks in band in 7 of 26 usable reps and the control arm in 2 of 28, and the ledger marks that aggregate as disclosure, not a row. Headroom is SEPARATED at both levels. This is the first current, manifest-backed ledger row; it records response-shape behavior change, never quality uplift, and what it records today is INERT at both levels. **Evidence on the tip.** Probe, exec-lib, metadata, coverage, headroom, and adapter bats suites green with the Darwin seal, survivor, session-escape, and environment-boundary tests running; `verify-scorecard` on both scorecards; `check-skill-probe-coverage.sh` reports measured 1 of 12 with the 2026-08-26 row labelled `verdict-withdrawn`; regen check clean; full gate green with a HEAD-built binary; CI's bats command green; Go bar, lint, security, and the gate-tightening ratchet green. **Judgment, stated plainly.** The caller elected Fable 5.1 as the binding validator. Its fifth-round verdict on the pre-CI tip was PASS with no verdict-driving findings; its read of the portability delta failed on one test-only point (the coverage test had dropped the `measured` pin that CI tripped on) which this tip restores. The cross-family Codex reads of the same tips are FAIL. They closed the network seal, the environment boundary, the fatal reap, the data roots, the config binding, and the documentation, and they keep these open, recorded here as the next pass rather than hidden: the evaluator identity check still accepts the pre-proxy four-file shape for legacy sets; an inherited absolute `CODEX_EXEC_TIMEOUT_BIN` is trusted without a probe; a refusal logged under a rep label outside the expected set is not rejected; an empty top-level `[projects]` table passes the drift check; the `lsof` sweep treats a failing `lsof` as a clean result; and the launcher chain, now bound as a record that any host can check, is corroborated against the filesystem only on the capturing host, so a record whose head does not exist there is taken on its word (the docs say so; Codex wants the live walk required at capture and the invoked head bound into the producer identity). Fable also noted a test that mis-resolves when TMPDIR sits under `/private/tmp`, a `2026-09-04` date in one refusal message, and that the low control replies used a `S2: FINDING <reason>` shape the discriminator scores as "plan not walked" rather than by band (outcome-invariant). Each of those changes a harness or verifier file and would orphan the two rows again, so they are filed as the follow-up instead of an eighth capture today. Across six judging rounds the open set shrank only from round four on; the convergence law's stop signal was reached at round three and the run continued on the caller's instruction to land the instrument. The two ledger rows are honest about what they measure, and the seal they ran under is the one this tree can verify on any host. |
||
|
Bo
|
10f0277bdb |
Legible membrane, Train 2: what a stranger meets (#1100)
## Legible membrane, Train 2: what a stranger meets Provenance: the 2026-09-02 field audit of this repo against mattpocock/skills, compound-engineering, and the jsm corpus, findings F5 through F9. This train is sized by a consumer inventory built with `rg` on the tip before any lane was written; the promoted-set directory move the audit proposed is deferred because that inventory shows skill-builder backing two blocking gates, swarm pinned by the cathedral gate and a routing golden, using-gc required by Go code, and `ao skills link` unable to install a second root. That inventory is the plan for a later train. **What changes.** - **Archival sweep by consumer disposition.** 172 audit snapshots, 29 pawl receipts, the `evals/workbench` and `evals/membrane` trees with their two bats consumers, four stray scratch docs, four retired eval contracts, and nine caller-less `scripts/check-*.sh` are deleted; git history is the archive. Every machine list that referenced them is pruned (evidence-grounding baseline, preamble grandfather, broken-links allowlist, `.gitattributes`, `.gitignore`, two eval fixtures, the workflow-coverage deferred list). `docs/audits/manifests/` and `.agents/ao/config.yaml` survive because they have live readers. About 48,000 lines. - **Three skills retired.** `goals` (alias of fitness), `shared` (tombstone), and `scope` (folded into plan step 3 as five write-scope checks). Consumers edited; the probe denominator exclusion for goals pruned; Codex package and golden count pins updated. - **Negative routing** on research, codebase-recon, reverse-engineer, premortem, one-way-door, and council, all within the 180-char budget, with a teardown golden (`rq-08`). One wording was changed after the router's prefix stemming showed "repository teardown" leaking into the wrong skill. - **Every promoted skill answers "It's working if"** with observable tells in backticks, and carries a paste-ready `## Prompt` with a concrete subject. Two fictional `ao` subcommands a draft prompt named were caught by the body-ref validator and replaced with real commands. - **Doctrine diet on the core five.** rpi, plan, implement, validate, and anti-ceremony drop from about 5,100 words to 3,600 (bodies from 4,700 to 3,150) by moving the shared ownership boundary, dated incidents, and mechanics tables into step-loaded references (`skills/rpi/references/boundaries.md`, `skills/validate/references/mechanics.md`, `skills/plan/references/ground-truth-routing.md`). Every cathedral canary and every skill validator grep survives unchanged. - **ADR-0018** records the goals, shared, and scope retirement; the cathedral gate tombstone and the routing goldens cite it instead of ADR-0017. - **Router and twins.** `ao skills find` holds a description's "Not for X; that is <sibling>." sentence out of its haystack, so premortem no longer ranks first for "is this live decision reversible" (golden `rq-10` pins the reciprocal of `rq-02`); a penalty variant was tried and reverted because it suppressed skills the caller named outright. A declared trigger phrase of two or more words quoted whole in the query now earns the name weight once, so "check this change" lands on validate rather than on reality-check's name token; a live-catalog test pins seven such queries. Single-quoted YAML descriptions unescape `''`. The Codex catalog keeps the exclusion sentence, and a closing `>` no longer turns `<run-id>/codebase-recon.json` into an invocation. - **Residue the judges found.** handoff, learn, and status open a `## Contract` heading after their tells; validate's prompt names its helper at `skills/validate/scripts/validate.py`; the explicit-skill prompt catalog names only live skills (five stale prompts replaced by nine, floor 20 restored, TESTING.md names the suite); the corpus-delta receipt binds the runner's path and SHA-256 and labels a `live_agent` claim as an unverified caller declaration; the probe README and ledger describe the 12-skill denominator; SKILL-API counts 30 of 54. **Evidence on the tip.** Regen check clean; full gate green with a HEAD-built binary; CI's bats command green; Go build/vet/test green; lint clean; security gate quick PASS; docs-build warnings did not rise. Fresh validation by Fable 5.1 (caller-elected) and a cross-family read by Codex, both recorded in the PR thread. --------- Co-authored-by: Bo <bofuller55@gmail.com> |
||
|
Bo
|
e556834d3d |
Legible membrane, Train 1: Codex projection, executable entry points, green run-all, honest runtime docs (#1097)
## Legible membrane, Train 1: fix what ships Intent source: `docs/plans/2026-09-02-legible-membrane-plan.md` (added in this PR). Provenance: the 2026-09-02 field audit of this repo against mattpocock/skills, compound-engineering, and the jsm corpus. Three defects a stranger hits today, all verified on `main` before the change: 1. **Codex projection truncated 51 of 56 skill descriptions mid-clause** (`scripts/codex-sync.sh` capped prose at 44 chars, so the Codex router read "Freshly judge whether a finished change is Triggers: …"). Now: first sentence of the source prose plus the full `Triggers:` clause, abbreviation- and quote-aware; the per-catalog bound is computed live as "Codex prose average may not exceed Claude's" (cross-multiplied, floor-free) with a 180-char hard ceiling. `using-flywheel` is exempted from runtime-phrase rewriting so its cross-runtime text projects verbatim. The dormant `ao codex ensure-start` emitter and the `# /x`→`# $x` title rewrite are removed. A literal oracle pins five twins and a rule oracle covers all 56. 2. **23 shebang-bearing shell entry points were committed non-executable**, including `scripts/regen-all.sh` and three gate scripts. Now `100755`, with an advisory gate `shell.exec-bits` (fast+full, fail-closed on enumeration error, reads the index blob, skips symlinks by stated policy). 3. **`tests/run-all.sh` was red on `main`**: the GOALS validator asserted a pre-08-25 file shape, and validate's description exceeded the 180-char budget. The validator now parses only the `## Gates` block (stops at any heading, like production), requires cited script paths to exist, takes its path via argv, and fails closed on tokenization; four negative fixtures. validate's description is 177 chars and gains the reality-check negative. `AGENTS.md` quotes CI's authoritative bats and Go commands. README and `docs/install-day2-ops.md` replace "No other runtime is required" with a runtime table derived by reading each skill's procedure (HARD / OPTIONAL / conditional), covered by a bats test that grounds every HARD row in an invocation line. Evidence on the tip: full gate 71/71 (HEAD binary), CI's bats command green across `tests/scripts/*.bats`, `tests/run-all.sh` green, Go build/vet/test green, golangci-lint clean, security gate quick PASS, `scripts/regen-all.sh --check` clean. Each lane was validated by a fresh context; the integrated train had two cross-family (Codex) review rounds and every finding is closed on this tip. Out of scope (successor intent): promoted set / `skills-internal/`, the process-artifact sweep, "It's working if" blocks, routing clusters, doctrine diet. One pre-existing drift noted for a follow-up: `packs/agentops-executor/agents/validator/skills/validate/SKILL.md` carries a stale third description that no gate binds. |
||
|
Bo
|
f3c6d0ecf2 |
Converge retained WIP and harden evidence boundaries (#1065)
Summary:
- lands the audited current WIP lanes and excludes stale/process-only
material
- hardens prune path confinement, probe-v3 evidence binding,
codebase-recon identity, handoff/release/reverse-engineer behavior, and
Codex prompt handling
- truth-labels static skill scoring and regenerates all owning
projections
Validation:
- fresh independent PASS on commit
|
||
|
boshu
|
4823077621 | refactor: cut AgentOps to a single-pass evidence loop | ||
|
Bo
|
1d4c003e05 |
Cut plan: gate registry diet, legacy bash gate removal, CI collapse, pawl warm-service removal (#906)
## Summary - **Wave 1** — remove the `always.wiring-closure` meta-gate, demote `docs.skill-refs` to Full tier, delete 8 orphan scripts. - **Wave 2** — delete the legacy bash pre-push gate (`pre-push-gate.sh`, 2,279 lines) and its parity machinery; collapse `validate.yml` from 18 jobs to 5. - **Wave 3** — remove the pawl warm NTM standing service; the cold review spine (pawl-review / pawl-verdict / pre-push check) is untouched. - **Wave 4** — gate registry diet: cut 24 doctrine/meta check IDs and ~70 backing scripts/workflows/bats; trim nightly knowledge-cycle job. Net: gate registry 112 → 83 checks (incl. main's additions), ~24k lines of scripts/CI/bats deleted. Routine one-file fast gate now runs in ~19s. ## Test plan - [x] `go build` / `go vet` / gates+goals+ports package tests green - [x] `ao gate check --fast --scope range:origin/main..HEAD` — 66 passed, 0 failed, 1 skipped (199 changed files) - [x] `validate-ci-policy-parity.sh` PASS; validate.yml / nightly.yml parse - [x] `regen-changed-scope --check` green; registry.json regenerated - [x] Grepped tree for dangling references to every deleted script/check ID Made with [Cursor](https://cursor.com) --------- Co-authored-by: boshu <241868352+boshu2@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
boshu
|
b4c74afd02 |
feat(pawl): reviewer adapter contract — codex byte-compatible, agy cold, local-mlx eval-only; folds age-iian timeout re-snapshot (age-rk3r.1)
Post-refutation fixes (cross-family pawl caught two defects in the first cut): - DEFECT 1 (fail-open): the agy/local-mlx genuine-run marker 'VERDICT:' appears in the packet itself, so a run that merely cat/echoed the packet classified GENUINE and the last-verdict parser could extract a CONFIRMED from echoed diff-context lines. Fixed: sentinel-wrapped agy packets (random-nonce boundary lines) + a marker-independent packet-line containment check (reviewer_packet_echoed) classify packet echo as ECHO. - DEFECT 2 (wrong family certified): the verdict write hardcoded --refuter codex:...; REVIEWER=agy now certifies the canonical 'gemini' family (roster: agy|google->gemini), codex stays byte-identical, local-mlx gets a deliberately off-roster label so an opted-in eval verdict can never pass the prod roster check. |
||
|
boshu
|
55186e5460 | refactor(scripts): codex-exec lib — one fail-closed hardened runner, migrate all non-pawl callers (age-gate-the-ungated-egwt.8) |