mirror of
https://github.com/boshu2/agentops.git
synced 2026-09-14 15:08:13 +08:00
main
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Bo
|
e1fae0dae6 |
Make the engineering harness lean and add optional topic memory (#1116)
RPI now owns the authorized outcome through finish, with Plan and Memory
loaded only when useful. Known defects get direct repair, and evidence
can change the approach under unchanged acceptance. Fresh exact-content
validation remains required. Memory provides optional recall, mining and
curation of reviewed topic pages; specialists and the fixed-dispatch
adapter remain optional.
The change reconciles current documentation and generated skill
projections. It preserves native budget and permission authority, BD
work ownership, protected external evidence storage and the distinction
between a supported lesson and demonstrated later benefit. It adds no
scheduler, work store, Go command or evidence schema.
Validation: required local Go/build/vet/race checks, aggregate suite,
generated-output check and 72 gates pass. The complete 44-test executor
suite passes; its shared-deadline fixture now tolerates CI scheduling
jitter while still requiring deadline exhaustion and preventing a third
launch. Fresh author-distinct review passed all 112 changed paths with
no findings; all seven exact-head CI checks passed at
|
||
|
Bo
|
57ece9fb7b |
Restore private context routes and verify native judgment receipts (#1112)
Add explicit, recoverable private context routing through `ao config context`, binding native source, owner, task, model and destination to existing policy and external storage. Recovery reads the original Beads maintenance anchor; configuration reports native access enforcement as unattested. Add `ao provenance verify-judgments` to check required review profiles against exact native transcript receipts, independent subject and acceptance, distinct contexts, completion and permitted providers. Requested identity and unreported effort do not count as runtime evidence. The verdict schema is unchanged. Repair the existing cleanup test: a 0.3-second budget could expire during preparation before either fixture process started. A separate controlled-delay test now proves preparation cannot renew that deadline. The running-cleanup case requires parent/child readiness, preserved partial output, the postlaunch cleanup result and both processes stopped within its existing four-second bound. Production timeout behavior is unchanged. Validation: fresh author-distinct review passed the exact 55-path final subject and all T05/T21 acceptance. The complete local Bats run passed (1,333 passed, two existing skips), as did Go build/vet/test/race, all 72 full-mode gates, the aggregate and generated-output checks. Ubuntu/Windows CI, security and both installation jobs passed on the final commit. The final evidence scan found no new orphaned bindings; 73 historical bindings remain preserved. Earlier failed results and private evidence remain outside the PR. |
||
|
Bo
|
8061085c89 |
Ship native evidence helpers and fresh-family review defaults (#1110)
AO now performs intent snapshots, subject manifests, strict evidence verification, atomic verdict storage, and orphan inspection through the Go binary. The command handler keeps verification separate from presentation so it meets the existing complexity limit. These operations preserve the existing evidence formats, require explicit protected storage where applicable, and run outside a checkout without Python. The unchanged Python implementation remains a developer oracle; agents still provide semantic judgment. Codex and Claude skills now default to a fresh reviewer from the author’s model family. Callers can explicitly request cross-model review or pin its model. Reviewer adapters use a finite caller timeout or remaining deadline instead of a fixed ten-minute default, while retaining output limits and abnormal-termination cleanup. Validation: Go build, vet, tests and race/shuffle tests; 1,334 shell tests; aggregate runner; regeneration check; 72 full-mode gates. Independent checks exercised 84 storage-boundary rejections and 21 evidence operations with an empty PATH. Both canonical and generated RPI reference suites pass all 48 tests after updating the migrated oracle import without weakening assertions. Change-sensitive checks explicitly compare the final committed candidate with the original PR base. Linux, Windows, installer, security, and required summary checks are green. |
||
|
Bo
|
baa24e1b04 |
Bound headless execution and require acceptance-linked repair progress (#1107)
Headless invocations could run without a finite bound, and repair loops could count changed digests or finding counts as progress. The existing adapter now enforces positive finite deadlines, an optional inherited absolute deadline, bounded output capture and cleanup of its owned process group, refusing execution when protection is unavailable. The existing goal, RPI and validation skills now require acceptance-linked progress, distinguish newly discovered failures from regressions, and bound Andon recovery. Review effort follows the effect on acceptance and enforcement. These contracts do not claim native aggregate goal-budget or pause enforcement. Validation on the clean main baseline: - 1,317 Bats cases, zero failures, two skips; 46 RPI reference cases. - Go build, vet, tests and race/shuffle tests; local aggregate runner. - Generated projections current; all 72 AO gates pass. - Fresh independent Astra and Fable 5.1 xhigh judgments passed the exact clean source. Earlier uncommitted CDLC adoption, private work/evidence, and unrelated Go changes are excluded from this PR. |
||
|
Bo
|
1cebfee94d |
Measure one skill: sealed probe dispatch and the first current ledger row (#1101)
## Measure one skill: filesystem-sealed probe dispatch and the first current ledger row Provenance: the instrument-panel roadmap (`docs/plans/2026-08-26-instrument-panel-roadmap.md`) closed out with zero skills measured because control-arm reps read `skills/<skill>/SKILL.md` from the checkout and the operator's skill roots mid-run. Its named successor was filesystem-sealed dispatch. This PR builds the seal, then uses it, and the instrument caught one more leak on the way. **The seal.** Codex's own seatbelt does not nest inside an outer `sandbox-exec` profile, so a rep now runs under an outer profile with Codex's sandbox bypassed for exactly that reason: `file-read*` denied on the checkout and every skill root (`~/.agents`, `~/.claude/skills`, `~/.gemini/skills`, `~/.codex/skills`), `file-write*` denied everywhere except the rep's workspace, a scratch HOME, and the temp roots. HOME and CODEX_HOME are scratch (auth symlinked). Dispatch refuses without `sandbox-exec` unless `PROBE_SEAL=none` is set, which marks the run coverage-ineligible. The seal is recorded in the capture contract (`agentops-skill-probe-capture.v3`, a `seal` block), coverage eligibility requires `seal.mode == seatbelt`, and `verify-scorecard` re-checks that the denied roots include the repository and the four skill roots. Existing 08-26 sets become `legacy-unsealed` and ineligible, which changes nothing they were counting. **The leak the seal did not cover.** The first sealed capture showed the xhigh control-2 rep listing the shared live workspace, finding `treatment-1.prompt`, and reading it; the treatment prompt embeds the skill. So every rep now gets its own empty workspace, prompts reach a rep on stdin only, raw JSONL and stderr live in a harness-private dispatch directory the rep can only stat, and a `sibling-prompt-read` trap degrades any rep whose successful command names a prompt, contract, seal, or stage file or whose output lists a sibling's artifacts. Replaying the contaminated set under the new trap turns its REGRESSIVE into INERT with the contaminated rep degraded; that set is superseded and not committed. **The second pass, after the judges read the first.** A Fable validator passed the first sealed capture with four findings and a Codex read failed it with seven, all real: the whole per-user temp tree stayed readable (hundreds of stale probe directories held prompt files with the canonical skill bytes), sibling checkouts and the real `~/.codex` sessions were readable because seatbelt matches the traversed path and the skill roots are symlinks into the main checkout, the dispatch directory was writable so a rep could rename an earlier prompt into its workspace, the contract trusted asserted seal metadata (a hand-made Linux record was coverage-eligible), the only real-Darwin test nested seatbelt and proved nothing, and the operator's `config.toml` was symlinked in whole, so every rep started the operator's MCP servers. The seal is now one run directory removed on exit (`home/ ws/ tmp/ dispatch/`, reset per rep); reads are denied under the real HOME, the real TMPDIR, `/tmp`, the checkout, the git common directory, and each skill root's resolved target; `dispatch/` is write-denied with metadata-only reads; `file-link` and `file-clone` are denied on every denied root; the rep's config is rebuilt from top-level scalars only and `auth.json` is copied, not linked. The contract binds platform, mechanism, wrapper, the denied and allowed roots, the rep environment, and the sanitization, and coverage refuses any record that omits them; the Darwin test performs its reads inside the seal. One recorded hole remains by necessity: the codex launcher under `~/.local/bin` and `~/dev/dotfiles/bin` is re-allowed by literal path, and coverage refuses any allowed path inside the checkout. **The third pass, after the judges read the second.** Both judges failed the hardened tip again, and the headline was network: bypassing codex's own sandbox to nest the outer seal had dropped every inner deny, not only the filesystem ones, so a rep could fetch the canonical skill from GitHub while the disk was sealed. A rep now reaches the network only through a harness-owned CONNECT proxy (`scripts/lib/probe-connect-proxy.py`) with a host allowlist discovered from real runs (`chatgpt.com`, `ab.chatgpt.com`, the rotating `.oaiusercontent.com` suffix, and the API-key hosts), behind a seatbelt `network*` deny that admits only the proxy; every attempt is logged, a refused one degrades the rep, and web search is disabled in the rep's config. The profile is now rendered from the bound seal block by one function, so `verify-scorecard` reconstructs it and requires the digest to match; the recorded launcher chain is the only allowed read under a denied root; the wrap invokes `/usr/bin/sandbox-exec` by absolute path; the real CODEX_HOME and the Darwin cache directory are denied and bound; every non-stdio descriptor is closed and `/dev` writes narrowed to four devices; each rep runs in its own process group that must be empty before the next reset; one minimal config is generated per run, bound by text and digest, and re-parsed after each rep; one guarded trap covers the run root and any unpublished stage; and the coverage gate prints an eligibility row for every ledger row that names a scorecard. What stays open is stated in the README: allow-default still governs what the profile does not deny. **The fourth pass, after the judges read the third.** Both judges failed the third tip on properties the record carried but the verifier did not check. GNU `timeout` sets its own process group and ran outside the seal, so the per-rep reap was vacuous (four sleeping survivors after the test passed); the seal is now the outermost process, timeout runs inside it with `--foreground`, its absolute path is bound, and the reap counts survivors before signalling. The verifier now pins, not merely records: the egress host set (a module constant), port 443 only, no unix sockets, a loopback proxy, the generated config text equal to the renderer's output, the real CODEX_HOME and cache root inside the denied roots, writable roots under the run root, the four device paths, the environment allowlist, the launcher chain's digest against the producer identity, and it refuses unknown record fields. The proxy logs an attempt before dialing, captures the rep at accept time, refuses names that resolve to private or loopback ranges, and its source joins the evaluator hash set; the per-rep egress log is published with the fixture set. Withdrawn ledger rows get their own eligibility reason. The README lists which seal fields the verifier pins and which it only records; the cross-capture aggregate is recounted to 4 of 14. Still open and disclosed: Mach IPC and Apple Events beyond the filesystem and network seals, exact-byte environment sealing beyond the allowlist, and cleanup ownership by inode. **The fifth pass, after the judges read the fourth.** Both judges found the seal, proxy, reap ordering, transcripts, and scorecards sound and failed the tip on claims the tree did not hold: the egress log the docs called published was caught by the repository's `*.log` ignore rule, a `setsid()` child escaped the process-group reap, and one ledger sentence said no rep ran a command when a control rep had run `rg` over its empty workspace. The log is now tracked and required, strictly parsed, with both allowed and refused counts recomputed per rep; the reap is fatal to the capture when it cannot prove emptiness and an `lsof` sweep of the run root catches a session escape; the rep launches through a real `env -i` boundary; the timeout budget, the launcher chain's adjacency, every required root in both read and link denies, the evaluator match, and the trust-table value are pinned by the verifier; and every sentence in the ledger, README, and runbook names the check that enforces it. The remaining gap is stated: a child that both escapes the session and holds nothing under the run root is not detected. **The sixth pass, after CI read the fifth.** The first CI run of this PR failed on Linux: the launcher-chain pin walked the capturing Mac's filesystem, so on the runner both sets read as unsealed and the coverage headline fell to zero. A pin that holds on one host is not a pin. The chain is now bound as structure in the seal record (each link's path, kind, symlink target, and the final file's digest, with the invoked path as the head), the verifier checks that record on any host, and it cross-checks the live filesystem only where the head exists, reporting which happened. A gate-level test runs the coverage gate twice, once with the chain made absent, and requires byte-identical output. **The measurement.** `premortem-plan-shape-t2` (two seeded structural defects, band [1,4]) captured seven times on 2026-09-03, once under each seal; the first six sets were deleted with their scorecards when a judge or CI broke that seal. The rows that count are the seventh capture: low INERT (control 0/2, treatment 0/2) and xhigh INERT (control 1/2, treatment 1/2); no rep ran a command, no egress was refused, both per-rep egress logs are tracked and verified, and both scorecards verify with every pinned field, the launcher chain checked as a record and cross-checked on the capturing host. The superseded captures read, at low, BEHAVIORAL 1/1, INERT, INERT, BEHAVIORAL 1/2, INERT, BEHAVIORAL 1/2 and, at xhigh, INERT, BEHAVIORAL 2/2, INERT, INERT, INERT (control 1/2), INERT. The reversals across captures are recorded as an unresolved N=2 observation, not variance; across all seven the treatment arm put the marks in band in 7 of 26 usable reps and the control arm in 2 of 28, and the ledger marks that aggregate as disclosure, not a row. Headroom is SEPARATED at both levels. This is the first current, manifest-backed ledger row; it records response-shape behavior change, never quality uplift, and what it records today is INERT at both levels. **Evidence on the tip.** Probe, exec-lib, metadata, coverage, headroom, and adapter bats suites green with the Darwin seal, survivor, session-escape, and environment-boundary tests running; `verify-scorecard` on both scorecards; `check-skill-probe-coverage.sh` reports measured 1 of 12 with the 2026-08-26 row labelled `verdict-withdrawn`; regen check clean; full gate green with a HEAD-built binary; CI's bats command green; Go bar, lint, security, and the gate-tightening ratchet green. **Judgment, stated plainly.** The caller elected Fable 5.1 as the binding validator. Its fifth-round verdict on the pre-CI tip was PASS with no verdict-driving findings; its read of the portability delta failed on one test-only point (the coverage test had dropped the `measured` pin that CI tripped on) which this tip restores. The cross-family Codex reads of the same tips are FAIL. They closed the network seal, the environment boundary, the fatal reap, the data roots, the config binding, and the documentation, and they keep these open, recorded here as the next pass rather than hidden: the evaluator identity check still accepts the pre-proxy four-file shape for legacy sets; an inherited absolute `CODEX_EXEC_TIMEOUT_BIN` is trusted without a probe; a refusal logged under a rep label outside the expected set is not rejected; an empty top-level `[projects]` table passes the drift check; the `lsof` sweep treats a failing `lsof` as a clean result; and the launcher chain, now bound as a record that any host can check, is corroborated against the filesystem only on the capturing host, so a record whose head does not exist there is taken on its word (the docs say so; Codex wants the live walk required at capture and the invoked head bound into the producer identity). Fable also noted a test that mis-resolves when TMPDIR sits under `/private/tmp`, a `2026-09-04` date in one refusal message, and that the low control replies used a `S2: FINDING <reason>` shape the discriminator scores as "plan not walked" rather than by band (outcome-invariant). Each of those changes a harness or verifier file and would orphan the two rows again, so they are filed as the follow-up instead of an eighth capture today. Across six judging rounds the open set shrank only from round four on; the convergence law's stop signal was reached at round three and the run continued on the caller's instruction to land the instrument. The two ledger rows are honest about what they measure, and the seal they ran under is the one this tree can verify on any host. |
||
|
Bo
|
f3c6d0ecf2 |
Converge retained WIP and harden evidence boundaries (#1065)
Summary:
- lands the audited current WIP lanes and excludes stale/process-only
material
- hardens prune path confinement, probe-v3 evidence binding,
codebase-recon identity, handoff/release/reverse-engineer behavior, and
Codex prompt handling
- truth-labels static skill scoring and regenerates all owning
projections
Validation:
- fresh independent PASS on commit
|
||
|
boshu
|
4823077621 | refactor: cut AgentOps to a single-pass evidence loop | ||
|
Bo
|
7e06834615 |
fix(ci): make the validate.yml backstop green — fix all four pre-existing red check classes (#908)
## Why Every PR since #906 has run red on the same 5 checks (correctness ubuntu+windows, go-gate-shadow, security, summary) — #906 and #907 both merged with identical red sets. A permanently-red backstop is worse than none: a real regression is invisible in the noise. This PR fixes every class and makes green the expected state again. ## What (three commits) **Wave 1 — the four red check classes** (each verified by running the CI job's own command): - **correctness/bats (138 CI-only failures, ~6 env mismatches)**: pawl verdict-edge fail-closed fires without a trusted `ao` on PATH → `PAWL_EDGE_FAIL_OPEN=1` on the bats step (the 3 fail-closed regression suites opt back out); gtimeout probe crashing setup() on Linux; gawk \\" bracket-expression warning corrupting jq assertions; ripgrep missing on runners; teardown/skip-path exit codes; taxonomy fixture missing the now-required observations field. - **go-gate-shadow (8 full-gate failures → 0)**: #906 landed without a heal/regen pass — pre/post-mortem pointer skills missing `hexagonal_role`; codex twins/hashes drift; shrink-only grandfather lists; evolve SKILL.md referencing 3 deleted scripts; @covered-by/allowlist conflicts; 3 go.lint findings. - **security (BLOCKED_HIGH → PASS)**: 3 false positives suppressed per-finding with justification (gosec G122 on the dev-time arch checker; 2 semgrep mis-traces). - **correctness/windows**: tests set HOME but Go os.UserHomeDir() reads USERPROFILE on Windows → cross-platform setHome helper. **Wave 2 — 18 residual bats failures across 13 files**, each first proven identical on pristine origin/main (differential run) before fixing. Content drift (README golden-path literal, discovery references, bootstrap↔install-bd wiring, pawl-review reachability via council, rpi helper-rung tokens), validator regressions (validate-codex-rpi-contract resolver routing, em-loop labels, goal-design fixture), and stale tests updated only where provably behind a deliberate change (catalog schema_version pin 1→2, SHA-pinned action regex, retired perf scenario, drained dangler pin 2→0, tagged-ao test build). **Commit 3**: the `ao skills unlink` train that landed on main mid-work re-reddened security with the same semgrep value-struct FP — suppressed like its skills_link sibling. ## Verification (rebased onto current main tip) - `ao gate check --full`: **92/93 pass, 0 fail** (1 expected AP7 skip) - `security-gate.sh --mode quick`: **PASS** - `go vet` clean; **11,878 Go tests green** (149 packages) - All 13 wave-2 bats suites **0 not-ok**; CI-sim (no-ao PATH) bats clean on wave-1 suites - Windows fix verified via GOOS=windows build+vet (needs the windows-latest job for final confirmation) **Acceptance: this PR's own validate.yml run going green.** Bead: age-htrqp --------- Co-authored-by: boshu <241868352+boshu2@users.noreply.github.com> |
||
|
boshu
|
55186e5460 | refactor(scripts): codex-exec lib — one fail-closed hardened runner, migrate all non-pawl callers (age-gate-the-ungated-egwt.8) |