Companion to e16da42a. The agentops-eval-advisory job calls
scripts/eval-agentops.sh --fast directly, which has its own copy of the
strict baseline-audit logic — untouched by the pre-push-gate.sh / new CI
job softening in e16da42a. Result: advisory job still failed on the 54
missing_compare_baselines under the no-tracked-.agents/ policy from
3f1566fd, drowning out the 5 real eval-suite regressions in the noise.
Mirror the same drift-only logic into eval-agentops.sh:
- stale_suite_hashes>0 → record_failure (real drift)
- policy_mismatch_count>0 → record_warning (substrate-info)
- both 0 → unchanged success line
The 5 real suite regressions (distribution-install-update,
hook-manifest-runtime-contracts, pre-push-gate-governance,
push-worktree-closeout, release-security-gates) remain visible as
suite-level FAIL records. They are real signal worth a separate
investigation but unrelated to the substrate inconsistency this change
addresses.
Verified locally: shellcheck pre-push-gate-style → ok; bash -n → ok.
The required agentops-eval-baseline-audit CI job from 9fe8ad4b would have
failed every fresh clone after upstream commit 3f1566fd ("stop tracking
agents runtime state") removed .agents/evals/baselines/ from git. Without
tracked baselines, all 54 mode=compare suites report missing_compare_baselines
→ policy_mismatch_count=54 → gate fails on a clean checkout. The drift signal
the validation FAIL actually wanted (item 2: "drift undetected") is
stale_suite_hashes — which only triggers when a *promoted* baseline's
recorded suite SHA stops matching the current suite definition.
Changes:
- .github/workflows/validate.yml: agentops-eval-baseline-audit fails only
on stale_suite_hashes>0; policy_mismatch_count is echoed for visibility.
- scripts/pre-push-gate.sh block 24d: same softening; missing-compare-
baselines becomes warn(), not fail(). Mirror keeps local and CI in sync.
- AGENTS.md: CI table description updated to reflect drift-only semantics
and document the substrate reason.
Verified locally:
- python yaml.safe_load: ok
- validate-ci-policy-parity: PASS (34 jobs; 4 non-blocking)
- shellcheck pre-push-gate.sh: ok
- ao eval baseline-audit on this checkout: stale=0 mismatch=54 — gate
would now pass on fresh clone
The "missing baseline" signal is preserved as a warn line both locally and
in CI logs. Re-tracking baselines vs flipping suites to mode=none is a
substrate decision separate from this fix.
The pre-push gate that 12d00c4f just wired immediately fired on its own
push, blocking it because validate-ci-policy-parity.sh requires the AGENTS
CI table to enumerate every required job in summary.needs. Adds the two
new blocking jobs to AGENTS.md so the parity check passes.
Verified: scripts/validate-ci-policy-parity.sh → PASS (34 jobs; 4 non-blocking).
The 2026-05-03 overnight /evolve session shipped a baseline-audit pre-push
gate (commit d5d38ceb) but post-mortem council verdict was FAIL: gate had
0% enforcement (core.hooksPath was dangling), CI ran the eval suite with
continue-on-error: true (drift undetected), and eval-agentops.sh's per-suite
loop could promote baselines for suites that had just regressed. This commit
lands all three critical fixes from .agents/learnings/2026-05-03-overnight-evolve-post-mortem.md (L2, L4) and phase-3-summary-2026-05-03-overnight-evolve.md
items 1-3.
Item 1 — wire pre-push baseline-audit gate to a real git hook:
- scripts/check-pre-push-gate-wired.sh: asserts .githooks/pre-push exists,
is executable, and references scripts/pre-push-gate.sh; verifies the
baseline-audit block exists with a non-zero exit path; --dry-run-smoke
runs `git push --dry-run` against a sandbox bare remote and asserts
the hook actually fires.
- install-dev-hooks.sh already sets core.hooksPath=.githooks but had not
been run on this checkout; ran it to repair the local config.
Item 2 — promote ao eval baseline-audit to required CI:
- .github/workflows/validate.yml: new agentops-eval-baseline-audit job
(required, gated through summary needs/checks) builds ao, runs
`ao eval baseline-audit --root evals/agentops-core --json`, and fails
on stale_suite_hashes>0 OR policy_mismatch_count>0. Mirrors
pre-push-gate.sh block 24d.
- Also adds pre-push-gate-wired job (required) that runs
check-pre-push-gate-wired.sh --dry-run-smoke on every PR.
Item 3 — eval-agentops.sh two-phase promotion:
- Per-suite loop now queues promotions via queue_promotion() instead of
executing them. After the loop, run_promote_queue() aborts if
failures>0 OR working tree is dirty outside .agents/ and .beads/.
Override: EVAL_AGENTOPS_ALLOW_DIRTY=1.
Verified locally:
- check-pre-push-gate-wired.sh --dry-run-smoke --check-activation: 7/7 ok
- jq parsing of ao baseline-audit output: stale=0 mismatch=0
- eval-agentops.sh --fast --advisory: still runs (caught a real
regression on agentops-core.beads-issue-tracking; with the new
two-phase logic, --promote-baseline would now refuse to overwrite
that suite's baseline)
- python yaml.safe_load on validate.yml: 36 jobs, both new jobs in
summary.needs
Refs: .agents/council/2026-05-03-validate-overnight-evolve-claude-skeptic.md
.agents/learnings/2026-05-03-overnight-evolve-post-mortem.md (L2, L4)
.agents/rpi/phase-3-summary-2026-05-03-overnight-evolve.md
Migrates 5 Go test files from manual HOME save/restore to t.Setenv("HOME", ...),
which serializes through testing's package-private mutex — required for
go test -race -shuffle=on -count=N determinism.
Files migrated:
- cli/cmd/ao/doctor_test.go
- cli/cmd/ao/hooks_test.go
- cli/cmd/ao/inject_test.go
- cli/internal/harvest/walker_test.go
- cli/internal/paths/paths_test.go
New lint scripts/check-test-home-isolation.sh (kill switch:
CHECK_TEST_HOME_ISOLATION_DISABLED=1) detects:
- os.Setenv("HOME", ...) inside test functions (banned)
- *_test.go files reaching into $HOME paths without t.Setenv or TestMain isolation
BATS hook lifecycle race fixed in tests/hooks/{helpers/test_helper.bash,
hook-output-schema.bats, hook-stdin-contracts.bats, test-hooks.bats}.
Pre-push gate (scripts/pre-push-gate.sh) wires the new lint as a
diff-conditional check; opt-out via AGENTOPS_PREPUSH_SKIP_TEST_HOME_ISO=1
(symmetric to soc-h53j prepush_skip_flag pattern).
Paired BATS stubs added per f-2026-04-27-002:
- tests/scripts/pre-push-gate.bats: 5 new tests (36-40) pass
- tests/scripts/check-agents-write-surfaces.bats: extended
Acceptance:
- scripts/check-test-home-isolation.sh -> PASS (0 offenders)
- bats tests/scripts/pre-push-gate.bats -> 40/40 pass
- go test -race -count=1 ./... -> PASS
Time-deferred acceptance (observational): full -race -shuffle=on -count=10
gate satisfies on next CI nightly run. Local -race -count=3 sample run was
backgrounded; if a real flake surfaces in CI, reopen with the failing test
ID for targeted fix.
Covers items 4, 15, 16 from .agents/research/2026-05-03-top-50-ci-failures.md.
soc-y1bk
Codify the policy distinction between pre-merge blocking and post-merge advisory
CI jobs so warn-only red stops training agents to ignore CI. Covers items 2, 3,
33, 34, 38, 40 from .agents/research/2026-05-03-top-50-ci-failures.md.
Changes:
- .github/workflows/validate.yml:
- Rename advisory display names from "(warn-only)" to "(advisory)" for
agentops-eval-advisory, security-toolchain-gate, doctor-check, and
check-test-staleness.
- Mark swarm-evidence as advisory: add `name: swarm-evidence (advisory)`,
`continue-on-error: true`, and remove from summary fail-condition.
- security-toolchain-gate install step now uses 3-attempt
exponential-backoff retry per item 40 (covers transient trivy/hadolint
network timeouts).
- AGENTS.md:
- Add "Advisory Job Triage SLAs" subsection with per-job SLA + escalation
rule (eval-advisory: 7d release-blocking when stale,
security-toolchain-gate: 14d, doctor-check: 30d, check-test-staleness:
info-only, swarm-evidence: info-only).
- Update inline non-blocking list and swarm-evidence row to reflect new
advisory status. validate-ci-policy-parity.sh now reports
38 jobs / 5 non-blocking (was 4).
- Document the retrieval-bench warn-then-fail ratchet promotion criterion
(14 consecutive nightlies of nightly_p_at_5 >= baseline_p_at_5).
- docs/CI-CD.md:
- Update "Soft Gates" table to include swarm-evidence and triage SLA column.
- Add "Retrieval-bench ratchet" subsection with promotion criterion,
baseline source (.agents/rpi/baseline-precision.json), 30-day staleness
fallback to 0.30, and explicit observational-only note (the
14-consecutive-nightly counter is not yet wired into automation; promotion
remains a manual decision).
- .agents/rpi/baseline-precision.json (new, force-tracked via .gitignore
re-include): seeds baseline_p_at_5 = 0.30 with metadata documenting owner,
promotion rule, fallback, and stale_after_days = 30.
- .gitignore: add re-include for .agents/rpi/baseline-precision.json and
remove the redundant late `.agents/` directory rule that was overriding
the earlier child-pattern re-includes.
H2 BRANCH-PROTECTION AUDIT (per soc-z7qq acceptance criteria):
$ gh api repos/boshu2/agentops/branches/main/protection \
--jq '.required_status_checks.contexts'
-> {"message":"Branch not protected","status":"404"}
Result: main is NOT protected on github.com/boshu2/agentops, so the
display-name renames carry zero required-check impact. No manual update
command needed. If branch protection is later enabled and any of the
renamed jobs (agentops-eval-advisory, security-toolchain-gate, doctor-check,
check-test-staleness, swarm-evidence) become required contexts, the GitHub
check name now ends in " (advisory)" — the workflow job ID is unchanged,
so required_status_checks.contexts must reference the job ID, not the
display name. The CI policy parity script keys on job IDs, not display
names, so no parity drift was introduced.
The 14-consecutive-nightlies ratchet is intentionally observational. A
follow-up bead is needed to wire the consecutive-green counter into
automation before promoting retrieval-bench to blocking.
Validation:
- python3 -c "import yaml; yaml.safe_load(open('.github/workflows/validate.yml'))"
-> OK
- bash scripts/validate-ci-policy-parity.sh
-> CI_POLICY_PARITY: PASS (38 jobs; 5 non-blocking)
- bash scripts/pre-push-gate.sh --fast
-> pre-push gate (fast): passed (40 skipped)
Items 35 (compile), 36 (dream-cycle), 37 (Athena follow-up) from
.agents/research/2026-05-03-top-50-ci-failures.md all share the same compile
substrate and exhibited the same 13% nightly red rate when the upstream corpus
had zero citation activity in the measurement window. Three separate jobs
produced three separate artifacts and three independent spurious-red signals
on the same underlying dormant-corpus condition.
Consolidate into a single `knowledge-cycle` job that:
- Calls a new `scripts/nightly-knowledge-cycle.sh` precondition substrate
before any compile/dream/Athena stage. When
`total_citations_in_window == 0 && total_artifacts > 0` (read from
`ao metrics report --json` .citations_this_period and .total_artifacts), the
cycle SKIPs every downstream stage with reason `corpus-dormant` rather than
failing — implements `f-2026-04-30-002`.
- Records each stage (compile-mine, compile-defrag, compile-health,
dream-cycle, athena) into one consolidated `triage.json`/`triage.md` and
uploads a single `nightly-knowledge-cycle` artifact in place of the three
prior `compile-report`, `dream-cycle-report`, and (referenced) Athena
artifacts.
- Replaces the prior `Nightly dream cycle status` issue with a single
`Nightly knowledge-cycle status` issue.
- Updates the summary job's `needs:` and fail-condition to track only the new
`knowledge-cycle` job.
The 5-consecutive-nightlies acceptance criterion (green-or-SKIPPED with no
spurious red) is time-deferred — this commit lands the substrate; the
observation gate closes after 5 nightly runs.
Updates:
- scripts/nightly-knowledge-cycle.sh: new shared substrate (precondition,
record-stage, read-decision subcommands)
- .github/workflows/nightly.yml: replace `compile` + `dream-cycle` jobs with
one `knowledge-cycle` job; consolidate artifact + issue creation; update
summary needs/failset
- AGENTS.md: add Nightly Workflow Jobs subsection documenting the deduped
knowledge-cycle job, the corpus-dormant precondition, and the
NIGHTLY_KNOWLEDGE_CYCLE_FORCE escape hatch
Validation:
- python3 yaml.safe_load: OK on .github/workflows/nightly.yml
- shellcheck scripts/nightly-knowledge-cycle.sh: PASS
- bash scripts/pre-push-gate.sh --fast: PASS
- substrate smoke (RUN path): decision=RUN, citations=2642, artifacts=526
- substrate smoke (SKIP path with mocked dormant corpus): decision=SKIP,
read-decision exit code 78
Refs: bd soc-2xmg; .agents/plans/2026-05-03-ci-failures-1-40-handling.md
§nightly-knowledge-cycle-dedupe; .agents/findings/f-2026-04-30-002.md;
.agents/research/2026-05-03-top-50-ci-failures.md items 35, 36, 37
(cross-ref item 39 Static Validation as DEFER)
Reduce 30% windows-smoke flake rate toward H1 <5% target by hardening
tests/windows/test-windows-smoke.ps1:
- Set-StrictMode -Version Latest (catches uninitialized vars and missing
properties at runtime instead of producing silent empty strings that
cascade into cryptic later failures).
- All path construction routed through new Join-PathSegments helper that
uses Join-Path; removed embedded backslash separators in literal path
arguments. Honors host DirectorySeparatorChar in every case.
- Pin nested PowerShell host: prefer current pwsh, fall back to
powershell.exe. Eliminates 5.1 vs 7.x execution-policy / parser drift
when CI runs under `shell: pwsh`.
- Header documents POSIX-symlink-dependent checks as explicitly out of
scope; canonical symlink audit lives in Linux-only plugin-load-test.
- Wrap nested invocations in -NoProfile to avoid host profile drift.
- Helper Remove-IfExists hardens cleanup against partial install dirs.
Time-deferred acceptance gate (<1 failure across 20 consecutive nightly
windows-smoke runs) satisfies later via observation; this commit lands
the hardening and unblocks the observation cycle.
Per .agents/plans/2026-05-03-ci-failures-1-40-handling.md
§windows-smoke-stabilize and item 6 of the top-50 CI failures research.
The aggregated codex-runtime-sections job ran 7 substeps sequentially in one
job, so the first failure cascaded and PR check lists showed only one badge for
all 7 sections. Split into 7 modular jobs so each fails/passes independently:
- validate-codex-runtime-sections
- validate-codex-generated-artifacts
- validate-codex-backbone-prompts
- validate-codex-override-coverage
- validate-codex-rpi-contract
- validate-codex-lifecycle-guards
- validate-headless-runtime-skills
R4 cost mitigation: each new job uses the same `actions/checkout@v6` pattern
already used by every other job in validate.yml. Wall time delta vs the
aggregated job is negative (parallel execution; longest substep, not the sum)
so a composite-action wrapper would not reduce billing minutes meaningfully —
deferred. Total summary.needs grew from 32 to 38 entries.
Updates:
- .github/workflows/validate.yml: replace 1 aggregated job with 7 modular jobs;
expand summary.needs and summary fail-condition to cover all 7
- AGENTS.md: replace single codex-runtime-sections row with 7 new rows
(alphabetically sorted); update blocking-policy paragraph
Validation:
- python3 yaml.safe_load: OK (40 jobs total, 38 in summary.needs)
- scripts/validate-ci-policy-parity.sh: PASS (38 jobs; 4 non-blocking)
- scripts/pre-push-gate.sh --fast: PASS
Refs: bd soc-ltp2; .agents/plans/2026-05-03-ci-failures-1-40-handling.md
§codex-runtime-split; .agents/research/2026-05-03-top-50-ci-failures.md item 19
Two-part eval-substrate hardening from harvested council finding #5:
1. cli/internal/eval/engine_test.go: TestRunSuiteRerunIsDeterministic runs
an artifact_check suite twice and asserts JSON equality of the
normalized RunRecord (RunID, StartedAt/CompletedAt, and per-case
DurationMS stripped). Exposes any nondeterminism leaking into
deterministic-tier suite results so promoted baselines remain
reproducible.
2. scripts/pre-push-gate.sh: new "24d. AgentOps eval baseline-audit"
block runs `ao eval baseline-audit --root evals/agentops-core --json`
when eval files changed (or PRE_PUSH_RUN_EVAL=1) and FAILs the gate
when stale_suite_hashes > 0 or policy_mismatch_count > 0. Catches
the drift class that was silent in this evolve session: 54 baselines
went stale through normal suite churn between 2026-04-25 and
2026-05-03 with nothing flagging it pre-push.
Also reconciles cycle-14 codex parity: skills-codex/pre-mortem/.agentops-
generated.json + skills-codex/.agentops-manifest.json source_hash bumped
to current skills/pre-mortem/ tree hash so validate-codex-generated-
artifacts.sh upstream scope passes.
Prior cycle-16 commit promoted via .agents/evolve/scripts/run-eval-baseline.sh
which passed absolute suite paths to ao eval run, so the promoted records
stored absolute "/home/boful/dev/personal/agentops/..." in suite.path and
baseline_path. The 2026-04-25 baselines used repo-relative paths, and that
is the portable form we want preserved.
Fix: run-eval-baseline.sh and promote-baselines.sh now cd to repo root and
pass relative suite paths / baseline outputs. Re-promoted from a clean
fix-pass; final audit unchanged (57 suites, 54 baselines, 0 stale, 0
policy mismatches).
Bulk catch-up: every baseline's expected_sha was stale relative to its
suite definition (last fresh promotion was 2026-04-25 on a now-stale set
of suite hashes). All 57 suites currently pass at aggregate 1.0; the 54
non-opt-out suites are now baselined against current suite SHAs.
3 suites with baseline_policy.mode=none (hook-context-value,
lid-primitives-demo, remote-compute-contracts) correctly retain no
promoted baseline.
Final audit: 57 suites, 54 baselines, 0 stale_suite_hashes, 0 policy
mismatches. Promoted from clean mid-baseline records.
Updated --baseline and --runtime artifact_contains expectations in
eval-runner-engine-control-plane.json to match current Cobra column
alignment. The recent LID baseline-A/B PR added --baseline-mode and
--delta-out to `ao eval run`, widening the help text alignment column
and breaking exact substring matches.
Eval signal: agentops-core.eval-runner-engine-control-plane went from
fail to pass (aggregate 1.0). cli/docs/COMMANDS.md is current; only the
eval expectations were stale.
Followup: artifact_contains is brittle to alignment shifts. Consider a
regex or normalized-whitespace matcher mode for help-text checks.
* chore(nightly): bootstrap 2026-05-03-v2 baseline + blocker matrix
Refresh baseline-goals.json with warm-cache measurement (cold-cache
go-cli-tests flake from 06:23 PM resolves under warm cache; not a
regression).
Update blocker-matrix.json: PR #212 was merged into main as b7d1181
between the 03:30 baseline check and nightly start; only PR #221
(triage) remains open and touches no failing-goal fix files.
Persist .agents/dream/probe-results.jsonl: 2/3 morning packets probed
stale (operator-environment install/reinstall tasks from
post-mortem-d73fc3bf, not in-repo code changes).
* refactor(cli): split runQuickstart into mode-specific helpers
runQuickstart sat at cyclomatic complexity 21, one above the 20 cap
enforced by the go-complexity-ceiling goal for cli/.
Extract runQuickstartDryRun, runQuickstartMinimal, runQuickstartFull,
ensureProjectClaudeMd, and finalizeQuickstartFull. The top-level
function now dispatches on dry-run / minimal / full and stays at
CC ~5. Each helper is single-purpose and below 18.
go-complexity-ceiling: fail (21 runQuickstart) -> pass.
Existing TestQuickstart_runQuickstart_minimal,
TestQuickstart_runQuickstart_fullNoBeads, and
TestQuickstart_runQuickstart_createsClaudeMd all continue to pass.
* chore(nightly): record cycle 1 audit trail (fitness, history, attempts)
* fix(hooks): swap awk-based int_ge for bash native -ge in eval-verdict-compiler
The threshold-breach branch (harmful_count >= 3 AND utility < 0.3)
calls int_ge, which routed integer comparison through awk. The awk
expression `print (a+0)>=(b+0)?1:0` is BSD/GNU-awk fragile and emits a
syntax error on macOS BSD awk; the branch still closes correctly but
the verdict log gets polluted with awk diagnostics on every breach.
Replace with bash native `[ "$1" -ge "$2" ] 2>/dev/null`. Both args
are integers (HARMFUL_THRESHOLD defaults to 3; new_harmful is
$((old_harmful + 1))) so -ge applies cleanly. The 2>/dev/null
preserves the silent-on-non-numeric semantics the awk version had via
the `+0` coercion.
Closes wave-1-residual|awk-portability-int-ge from post-mortem-finding.
Sync embedded copy in cli/embedded/hooks/.
* chore(nightly): record cycle 2 audit trail
* refactor(cli): consolidate atomicWriteFile onto quest.AtomicWriteFileWithPerm
cli/internal/search/util.go carried its own AtomicWriteFile copy from
before cli/internal/types/quest/atomic.go landed in l12. Of the 5
ad-hoc copies post-mortem agentops-mpp called out, 4 already moved to
quest; the final one in search/ had a single caller (the
atomicWriteFile thin wrapper in cli/cmd/ao/inject.go).
- Delete search.AtomicWriteFile.
- Point cli/cmd/ao/inject.go's wrapper at quest.AtomicWriteFileWithPerm.
- Update TestAtomicWriteFile_BadDir -> TestAtomicWriteFile_CreatesParentDirs:
quest's implementation MkdirAll-s the parent chain (a stricter durability
contract: fewer lost writes from missing parents), so the previous
no-create assertion no longer holds. New test exercises the documented
behavior.
Closes wave-1-residual|writeFileAtomic-dedup from post-mortem-finding.
Full `go test ./...` green; goals fitness unchanged at 97.27.
* chore(nightly): record cycle 3 audit trail
* chore(triage): stale-audit — wave-1-residual doc-index already shipped
docs/documentation-index.md:219 already contains the Eval Verdict
Pipeline Contract entry that the post-mortem-finding asked for. The
work shipped before this nightly; only the lifecycle fields needed to
catch up.
Stale-audit count for nightly 2026-05-03-v2: 1 (this commit).
* test(eval-verdict-compiler): add regressed/breach/idempotency assertions
Wave 1.5 plan rev 2 called for 4 missing assertions. Add 3 of them
(crash-replay deferred — needs more setup than simple watermark
mtime work):
(a) regressed-mutates-downward: regressed verdict drives EMA(0.7,0.3)
on (0.5, 0.1) -> 0.380000 and harmful_count 0 -> 1.
(b) breach-queues-retire: harmful_count>=3 AND utility<0.3 queues
a verdict-driven retire candidate to .agents/rpi/next-work.jsonl.
This is the branch the int_ge bash-native rewrite from
wave-1-residual|awk-portability-int-ge protects from BSD-awk
syntax noise; pairs with the cycle 2 fix in this nightly. Test
runs the hook from inside $tmp so ROOT lands in $tmp, keeping
the queued entry out of the actual repo's queue.
(c) idempotency-via-watermark: a second run after a successful first
pass leaves the learning bit-identical; new_manifests_available
skips manifests older than processed.jsonl.
Fixtures: manifest-regressed.json, manifest-regressed-breach.json,
learning-near-breach.md (harmful=2, utility=0.36 — one regressed
verdict away from breach).
Test count: 5 functional + 2 reward + 1 shellcheck (was 7) ->
9 functional + 1 shellcheck (10 markers when shellcheck installed,
9 when not). Plan target was 9; delivered 10.
Closes wave-1-residual|test-eval-verdict-compiler-9-markers.
* chore(nightly): record cycle 4 audit trail
* chore(nightly): persist final-goals snapshot for 2026-05-03-v2
* fix(test-fixture): wrap math in code block to satisfy markdownlint MD037
The literal '0.7 * 0.36' etc. tripped no-space-in-emphasis. Wrap the
EMA derivation in a fenced code block; fixture content is documentary
only (the test reads frontmatter, not body).
---------
Co-authored-by: Claude <noreply@anthropic.com>
Boden Fuller