Migrates 5 Go test files from manual HOME save/restore to t.Setenv("HOME", ...),
which serializes through testing's package-private mutex — required for
go test -race -shuffle=on -count=N determinism.
Files migrated:
- cli/cmd/ao/doctor_test.go
- cli/cmd/ao/hooks_test.go
- cli/cmd/ao/inject_test.go
- cli/internal/harvest/walker_test.go
- cli/internal/paths/paths_test.go
New lint scripts/check-test-home-isolation.sh (kill switch:
CHECK_TEST_HOME_ISOLATION_DISABLED=1) detects:
- os.Setenv("HOME", ...) inside test functions (banned)
- *_test.go files reaching into $HOME paths without t.Setenv or TestMain isolation
BATS hook lifecycle race fixed in tests/hooks/{helpers/test_helper.bash,
hook-output-schema.bats, hook-stdin-contracts.bats, test-hooks.bats}.
Pre-push gate (scripts/pre-push-gate.sh) wires the new lint as a
diff-conditional check; opt-out via AGENTOPS_PREPUSH_SKIP_TEST_HOME_ISO=1
(symmetric to soc-h53j prepush_skip_flag pattern).
Paired BATS stubs added per f-2026-04-27-002:
- tests/scripts/pre-push-gate.bats: 5 new tests (36-40) pass
- tests/scripts/check-agents-write-surfaces.bats: extended
Acceptance:
- scripts/check-test-home-isolation.sh -> PASS (0 offenders)
- bats tests/scripts/pre-push-gate.bats -> 40/40 pass
- go test -race -count=1 ./... -> PASS
Time-deferred acceptance (observational): full -race -shuffle=on -count=10
gate satisfies on next CI nightly run. Local -race -count=3 sample run was
backgrounded; if a real flake surfaces in CI, reopen with the failing test
ID for targeted fix.
Covers items 4, 15, 16 from .agents/research/2026-05-03-top-50-ci-failures.md.
soc-y1bk
Codify the policy distinction between pre-merge blocking and post-merge advisory
CI jobs so warn-only red stops training agents to ignore CI. Covers items 2, 3,
33, 34, 38, 40 from .agents/research/2026-05-03-top-50-ci-failures.md.
Changes:
- .github/workflows/validate.yml:
- Rename advisory display names from "(warn-only)" to "(advisory)" for
agentops-eval-advisory, security-toolchain-gate, doctor-check, and
check-test-staleness.
- Mark swarm-evidence as advisory: add `name: swarm-evidence (advisory)`,
`continue-on-error: true`, and remove from summary fail-condition.
- security-toolchain-gate install step now uses 3-attempt
exponential-backoff retry per item 40 (covers transient trivy/hadolint
network timeouts).
- AGENTS.md:
- Add "Advisory Job Triage SLAs" subsection with per-job SLA + escalation
rule (eval-advisory: 7d release-blocking when stale,
security-toolchain-gate: 14d, doctor-check: 30d, check-test-staleness:
info-only, swarm-evidence: info-only).
- Update inline non-blocking list and swarm-evidence row to reflect new
advisory status. validate-ci-policy-parity.sh now reports
38 jobs / 5 non-blocking (was 4).
- Document the retrieval-bench warn-then-fail ratchet promotion criterion
(14 consecutive nightlies of nightly_p_at_5 >= baseline_p_at_5).
- docs/CI-CD.md:
- Update "Soft Gates" table to include swarm-evidence and triage SLA column.
- Add "Retrieval-bench ratchet" subsection with promotion criterion,
baseline source (.agents/rpi/baseline-precision.json), 30-day staleness
fallback to 0.30, and explicit observational-only note (the
14-consecutive-nightly counter is not yet wired into automation; promotion
remains a manual decision).
- .agents/rpi/baseline-precision.json (new, force-tracked via .gitignore
re-include): seeds baseline_p_at_5 = 0.30 with metadata documenting owner,
promotion rule, fallback, and stale_after_days = 30.
- .gitignore: add re-include for .agents/rpi/baseline-precision.json and
remove the redundant late `.agents/` directory rule that was overriding
the earlier child-pattern re-includes.
H2 BRANCH-PROTECTION AUDIT (per soc-z7qq acceptance criteria):
$ gh api repos/boshu2/agentops/branches/main/protection \
--jq '.required_status_checks.contexts'
-> {"message":"Branch not protected","status":"404"}
Result: main is NOT protected on github.com/boshu2/agentops, so the
display-name renames carry zero required-check impact. No manual update
command needed. If branch protection is later enabled and any of the
renamed jobs (agentops-eval-advisory, security-toolchain-gate, doctor-check,
check-test-staleness, swarm-evidence) become required contexts, the GitHub
check name now ends in " (advisory)" — the workflow job ID is unchanged,
so required_status_checks.contexts must reference the job ID, not the
display name. The CI policy parity script keys on job IDs, not display
names, so no parity drift was introduced.
The 14-consecutive-nightlies ratchet is intentionally observational. A
follow-up bead is needed to wire the consecutive-green counter into
automation before promoting retrieval-bench to blocking.
Validation:
- python3 -c "import yaml; yaml.safe_load(open('.github/workflows/validate.yml'))"
-> OK
- bash scripts/validate-ci-policy-parity.sh
-> CI_POLICY_PARITY: PASS (38 jobs; 5 non-blocking)
- bash scripts/pre-push-gate.sh --fast
-> pre-push gate (fast): passed (40 skipped)
Items 35 (compile), 36 (dream-cycle), 37 (Athena follow-up) from
.agents/research/2026-05-03-top-50-ci-failures.md all share the same compile
substrate and exhibited the same 13% nightly red rate when the upstream corpus
had zero citation activity in the measurement window. Three separate jobs
produced three separate artifacts and three independent spurious-red signals
on the same underlying dormant-corpus condition.
Consolidate into a single `knowledge-cycle` job that:
- Calls a new `scripts/nightly-knowledge-cycle.sh` precondition substrate
before any compile/dream/Athena stage. When
`total_citations_in_window == 0 && total_artifacts > 0` (read from
`ao metrics report --json` .citations_this_period and .total_artifacts), the
cycle SKIPs every downstream stage with reason `corpus-dormant` rather than
failing — implements `f-2026-04-30-002`.
- Records each stage (compile-mine, compile-defrag, compile-health,
dream-cycle, athena) into one consolidated `triage.json`/`triage.md` and
uploads a single `nightly-knowledge-cycle` artifact in place of the three
prior `compile-report`, `dream-cycle-report`, and (referenced) Athena
artifacts.
- Replaces the prior `Nightly dream cycle status` issue with a single
`Nightly knowledge-cycle status` issue.
- Updates the summary job's `needs:` and fail-condition to track only the new
`knowledge-cycle` job.
The 5-consecutive-nightlies acceptance criterion (green-or-SKIPPED with no
spurious red) is time-deferred — this commit lands the substrate; the
observation gate closes after 5 nightly runs.
Updates:
- scripts/nightly-knowledge-cycle.sh: new shared substrate (precondition,
record-stage, read-decision subcommands)
- .github/workflows/nightly.yml: replace `compile` + `dream-cycle` jobs with
one `knowledge-cycle` job; consolidate artifact + issue creation; update
summary needs/failset
- AGENTS.md: add Nightly Workflow Jobs subsection documenting the deduped
knowledge-cycle job, the corpus-dormant precondition, and the
NIGHTLY_KNOWLEDGE_CYCLE_FORCE escape hatch
Validation:
- python3 yaml.safe_load: OK on .github/workflows/nightly.yml
- shellcheck scripts/nightly-knowledge-cycle.sh: PASS
- bash scripts/pre-push-gate.sh --fast: PASS
- substrate smoke (RUN path): decision=RUN, citations=2642, artifacts=526
- substrate smoke (SKIP path with mocked dormant corpus): decision=SKIP,
read-decision exit code 78
Refs: bd soc-2xmg; .agents/plans/2026-05-03-ci-failures-1-40-handling.md
§nightly-knowledge-cycle-dedupe; .agents/findings/f-2026-04-30-002.md;
.agents/research/2026-05-03-top-50-ci-failures.md items 35, 36, 37
(cross-ref item 39 Static Validation as DEFER)
Reduce 30% windows-smoke flake rate toward H1 <5% target by hardening
tests/windows/test-windows-smoke.ps1:
- Set-StrictMode -Version Latest (catches uninitialized vars and missing
properties at runtime instead of producing silent empty strings that
cascade into cryptic later failures).
- All path construction routed through new Join-PathSegments helper that
uses Join-Path; removed embedded backslash separators in literal path
arguments. Honors host DirectorySeparatorChar in every case.
- Pin nested PowerShell host: prefer current pwsh, fall back to
powershell.exe. Eliminates 5.1 vs 7.x execution-policy / parser drift
when CI runs under `shell: pwsh`.
- Header documents POSIX-symlink-dependent checks as explicitly out of
scope; canonical symlink audit lives in Linux-only plugin-load-test.
- Wrap nested invocations in -NoProfile to avoid host profile drift.
- Helper Remove-IfExists hardens cleanup against partial install dirs.
Time-deferred acceptance gate (<1 failure across 20 consecutive nightly
windows-smoke runs) satisfies later via observation; this commit lands
the hardening and unblocks the observation cycle.
Per .agents/plans/2026-05-03-ci-failures-1-40-handling.md
§windows-smoke-stabilize and item 6 of the top-50 CI failures research.
The aggregated codex-runtime-sections job ran 7 substeps sequentially in one
job, so the first failure cascaded and PR check lists showed only one badge for
all 7 sections. Split into 7 modular jobs so each fails/passes independently:
- validate-codex-runtime-sections
- validate-codex-generated-artifacts
- validate-codex-backbone-prompts
- validate-codex-override-coverage
- validate-codex-rpi-contract
- validate-codex-lifecycle-guards
- validate-headless-runtime-skills
R4 cost mitigation: each new job uses the same `actions/checkout@v6` pattern
already used by every other job in validate.yml. Wall time delta vs the
aggregated job is negative (parallel execution; longest substep, not the sum)
so a composite-action wrapper would not reduce billing minutes meaningfully —
deferred. Total summary.needs grew from 32 to 38 entries.
Updates:
- .github/workflows/validate.yml: replace 1 aggregated job with 7 modular jobs;
expand summary.needs and summary fail-condition to cover all 7
- AGENTS.md: replace single codex-runtime-sections row with 7 new rows
(alphabetically sorted); update blocking-policy paragraph
Validation:
- python3 yaml.safe_load: OK (40 jobs total, 38 in summary.needs)
- scripts/validate-ci-policy-parity.sh: PASS (38 jobs; 4 non-blocking)
- scripts/pre-push-gate.sh --fast: PASS
Refs: bd soc-ltp2; .agents/plans/2026-05-03-ci-failures-1-40-handling.md
§codex-runtime-split; .agents/research/2026-05-03-top-50-ci-failures.md item 19
Two-part eval-substrate hardening from harvested council finding #5:
1. cli/internal/eval/engine_test.go: TestRunSuiteRerunIsDeterministic runs
an artifact_check suite twice and asserts JSON equality of the
normalized RunRecord (RunID, StartedAt/CompletedAt, and per-case
DurationMS stripped). Exposes any nondeterminism leaking into
deterministic-tier suite results so promoted baselines remain
reproducible.
2. scripts/pre-push-gate.sh: new "24d. AgentOps eval baseline-audit"
block runs `ao eval baseline-audit --root evals/agentops-core --json`
when eval files changed (or PRE_PUSH_RUN_EVAL=1) and FAILs the gate
when stale_suite_hashes > 0 or policy_mismatch_count > 0. Catches
the drift class that was silent in this evolve session: 54 baselines
went stale through normal suite churn between 2026-04-25 and
2026-05-03 with nothing flagging it pre-push.
Also reconciles cycle-14 codex parity: skills-codex/pre-mortem/.agentops-
generated.json + skills-codex/.agentops-manifest.json source_hash bumped
to current skills/pre-mortem/ tree hash so validate-codex-generated-
artifacts.sh upstream scope passes.
Prior cycle-16 commit promoted via .agents/evolve/scripts/run-eval-baseline.sh
which passed absolute suite paths to ao eval run, so the promoted records
stored absolute "/home/boful/dev/personal/agentops/..." in suite.path and
baseline_path. The 2026-04-25 baselines used repo-relative paths, and that
is the portable form we want preserved.
Fix: run-eval-baseline.sh and promote-baselines.sh now cd to repo root and
pass relative suite paths / baseline outputs. Re-promoted from a clean
fix-pass; final audit unchanged (57 suites, 54 baselines, 0 stale, 0
policy mismatches).
Bulk catch-up: every baseline's expected_sha was stale relative to its
suite definition (last fresh promotion was 2026-04-25 on a now-stale set
of suite hashes). All 57 suites currently pass at aggregate 1.0; the 54
non-opt-out suites are now baselined against current suite SHAs.
3 suites with baseline_policy.mode=none (hook-context-value,
lid-primitives-demo, remote-compute-contracts) correctly retain no
promoted baseline.
Final audit: 57 suites, 54 baselines, 0 stale_suite_hashes, 0 policy
mismatches. Promoted from clean mid-baseline records.
Updated --baseline and --runtime artifact_contains expectations in
eval-runner-engine-control-plane.json to match current Cobra column
alignment. The recent LID baseline-A/B PR added --baseline-mode and
--delta-out to `ao eval run`, widening the help text alignment column
and breaking exact substring matches.
Eval signal: agentops-core.eval-runner-engine-control-plane went from
fail to pass (aggregate 1.0). cli/docs/COMMANDS.md is current; only the
eval expectations were stale.
Followup: artifact_contains is brittle to alignment shifts. Consider a
regex or normalized-whitespace matcher mode for help-text checks.
* chore(nightly): bootstrap 2026-05-03-v2 baseline + blocker matrix
Refresh baseline-goals.json with warm-cache measurement (cold-cache
go-cli-tests flake from 06:23 PM resolves under warm cache; not a
regression).
Update blocker-matrix.json: PR #212 was merged into main as b7d1181
between the 03:30 baseline check and nightly start; only PR #221
(triage) remains open and touches no failing-goal fix files.
Persist .agents/dream/probe-results.jsonl: 2/3 morning packets probed
stale (operator-environment install/reinstall tasks from
post-mortem-d73fc3bf, not in-repo code changes).
* refactor(cli): split runQuickstart into mode-specific helpers
runQuickstart sat at cyclomatic complexity 21, one above the 20 cap
enforced by the go-complexity-ceiling goal for cli/.
Extract runQuickstartDryRun, runQuickstartMinimal, runQuickstartFull,
ensureProjectClaudeMd, and finalizeQuickstartFull. The top-level
function now dispatches on dry-run / minimal / full and stays at
CC ~5. Each helper is single-purpose and below 18.
go-complexity-ceiling: fail (21 runQuickstart) -> pass.
Existing TestQuickstart_runQuickstart_minimal,
TestQuickstart_runQuickstart_fullNoBeads, and
TestQuickstart_runQuickstart_createsClaudeMd all continue to pass.
* chore(nightly): record cycle 1 audit trail (fitness, history, attempts)
* fix(hooks): swap awk-based int_ge for bash native -ge in eval-verdict-compiler
The threshold-breach branch (harmful_count >= 3 AND utility < 0.3)
calls int_ge, which routed integer comparison through awk. The awk
expression `print (a+0)>=(b+0)?1:0` is BSD/GNU-awk fragile and emits a
syntax error on macOS BSD awk; the branch still closes correctly but
the verdict log gets polluted with awk diagnostics on every breach.
Replace with bash native `[ "$1" -ge "$2" ] 2>/dev/null`. Both args
are integers (HARMFUL_THRESHOLD defaults to 3; new_harmful is
$((old_harmful + 1))) so -ge applies cleanly. The 2>/dev/null
preserves the silent-on-non-numeric semantics the awk version had via
the `+0` coercion.
Closes wave-1-residual|awk-portability-int-ge from post-mortem-finding.
Sync embedded copy in cli/embedded/hooks/.
* chore(nightly): record cycle 2 audit trail
* refactor(cli): consolidate atomicWriteFile onto quest.AtomicWriteFileWithPerm
cli/internal/search/util.go carried its own AtomicWriteFile copy from
before cli/internal/types/quest/atomic.go landed in l12. Of the 5
ad-hoc copies post-mortem agentops-mpp called out, 4 already moved to
quest; the final one in search/ had a single caller (the
atomicWriteFile thin wrapper in cli/cmd/ao/inject.go).
- Delete search.AtomicWriteFile.
- Point cli/cmd/ao/inject.go's wrapper at quest.AtomicWriteFileWithPerm.
- Update TestAtomicWriteFile_BadDir -> TestAtomicWriteFile_CreatesParentDirs:
quest's implementation MkdirAll-s the parent chain (a stricter durability
contract: fewer lost writes from missing parents), so the previous
no-create assertion no longer holds. New test exercises the documented
behavior.
Closes wave-1-residual|writeFileAtomic-dedup from post-mortem-finding.
Full `go test ./...` green; goals fitness unchanged at 97.27.
* chore(nightly): record cycle 3 audit trail
* chore(triage): stale-audit — wave-1-residual doc-index already shipped
docs/documentation-index.md:219 already contains the Eval Verdict
Pipeline Contract entry that the post-mortem-finding asked for. The
work shipped before this nightly; only the lifecycle fields needed to
catch up.
Stale-audit count for nightly 2026-05-03-v2: 1 (this commit).
* test(eval-verdict-compiler): add regressed/breach/idempotency assertions
Wave 1.5 plan rev 2 called for 4 missing assertions. Add 3 of them
(crash-replay deferred — needs more setup than simple watermark
mtime work):
(a) regressed-mutates-downward: regressed verdict drives EMA(0.7,0.3)
on (0.5, 0.1) -> 0.380000 and harmful_count 0 -> 1.
(b) breach-queues-retire: harmful_count>=3 AND utility<0.3 queues
a verdict-driven retire candidate to .agents/rpi/next-work.jsonl.
This is the branch the int_ge bash-native rewrite from
wave-1-residual|awk-portability-int-ge protects from BSD-awk
syntax noise; pairs with the cycle 2 fix in this nightly. Test
runs the hook from inside $tmp so ROOT lands in $tmp, keeping
the queued entry out of the actual repo's queue.
(c) idempotency-via-watermark: a second run after a successful first
pass leaves the learning bit-identical; new_manifests_available
skips manifests older than processed.jsonl.
Fixtures: manifest-regressed.json, manifest-regressed-breach.json,
learning-near-breach.md (harmful=2, utility=0.36 — one regressed
verdict away from breach).
Test count: 5 functional + 2 reward + 1 shellcheck (was 7) ->
9 functional + 1 shellcheck (10 markers when shellcheck installed,
9 when not). Plan target was 9; delivered 10.
Closes wave-1-residual|test-eval-verdict-compiler-9-markers.
* chore(nightly): record cycle 4 audit trail
* chore(nightly): persist final-goals snapshot for 2026-05-03-v2
* fix(test-fixture): wrap math in code block to satisfy markdownlint MD037
The literal '0.7 * 0.36' etc. tripped no-space-in-emphasis. Wrap the
EMA derivation in a fenced code block; fixture content is documentary
only (the test reads frontmatter, not body).
---------
Co-authored-by: Claude <noreply@anthropic.com>
* refactor(cli): split processCitationFeedbackWithOptions and schedule.Load
go-complexity-ceiling (w=6) gate measures cli/ at CC<=20 AND
cli/internal/ at CC<=18. Two violations in main:
- processCitationFeedbackWithOptions in cli/cmd/ao/flywheel_citation_feedback.go @ CC=21
- Load in cli/internal/schedule/parser.go @ CC=19
Split each into per-branch / per-field helpers without changing
behavior. Existing test suites for both packages still pass; refactors
preserve identical error paths and event payloads.
processCitationFeedback path:
- evaluateCitationFeedback dispatches on namespace / artifact-kind
- nonPrimaryNamespaceCitationFeedback handles the audited shadow-namespace branch
- findingArtifactCitationFeedback handles the finding-artifact branch
- learningArtifactCitationFeedback handles the learning-artifact branch
schedule.Load path:
- parseScheduleEntry validates a single entry and returns its template
- validateScheduleEntryCron enforces the effective-period floor
- parseScheduleEntryTimeout / buildScheduleEntryBackpressure / encodeScheduleEntryPayload
cover the remaining per-entry validation
* feat(scripts): WORKTREE_DISPOSITION_ALLOW canonical-branch glob override
Without this override the canonical-branch check hard-fails on every
nightly/* worktree, forcing the routine to skip the gate or chase a
spurious red signal. Implements one-time fix#5 from the nightly
routine prompt: a comma-separated list of glob patterns
(e.g. "nightly/*,triage/*") that the canonical worktree may be on.
The dirty-tree, allowed-worktree, and preserved-refs checks below
still apply unchanged.
Tested:
- bare invocation on nightly/2026-05-03 still FAILs with the new
override hint embedded in the error message
- WORKTREE_DISPOSITION_ALLOW='nightly/*' bypasses the branch check and
proceeds to the dirty-tree / worktree-allow validation
* chore(gitignore): force-track persisted nightly + evolve audit trails
Implements one-time fix#1 from the nightly routine prompt: the persisted
JSON paths used by the evolve loop are 10 KB/day of audit truth, but the
existing .agents/ blanket ignore meant every nightly run had to git add
-f them or watch them silently drop. Switches the .agents/ directory
ignore to .agents/* (children) so per-path !-exceptions can re-include
the audit-trail subpaths:
.agents/rpi/next-work.jsonl
.agents/nightly/<date>/{baseline,final,blocker-matrix,main-ci-baseline}.json
.agents/evolve/{cycle-history.jsonl,fitness-cycle-*.json}
.agents/goals/<id>/attempts.jsonl
.agents/dream/probe-results.jsonl
Existing tracked .agents/ content (council/, brainstorm/, INDEX.md, etc.)
is unaffected because tracked files are not re-evaluated against
.gitignore.
Also persists this run's prelude artifacts so the 2026-05-03 nightly's
baseline + blocker matrix + main-CI inheritance ledger + dream probe
results + cycle-2 fitness snapshot survive into the morning digest and
into the next nightly's anchor comparison.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat(nightly): include current CI/PR/incident context in nightly RPI brief
scripts/nightly-rpi-brief.sh previously only ingested past Nightly PR
bodies and scheduled Nightly run results. The autonomous RPI prompt
issue produced from that brief had no signal about (a) whether the
current Validate workflow is green on main, (b) what PRs are open and
their check-rollup state, (c) open Nightly-failure incidents, or (d)
whether the prompt issue itself already exists. As a result the
$agentops:rpi --auto target list could miss live CI blockers entirely.
This change:
- Adds --validate-workflow flag (default "Validate")
- Captures four new structured artifacts via capture_optional_array
helper that tolerates gh failures: validate-runs.json, open-prs.json,
open-incidents.json, prompt-issues.json
- Threads new fields into summary.json: current_ci, open_prs,
open_incidents, prompt_issue, ranked stabilization_targets
- Adds tests/scripts/nightly-rpi-brief.bats covering the happy path
with mocked gh fixtures (~80 LOC fixtures + assertions)
- Updates docs/CI-CD.md to describe the new inputs and outputs
* chore(knowledge): nightly decay metadata + new f-2026-05-02-006 finding
- Routine decay/reward updates from running ao processes:
.agents/learnings/{2026-04-07,2026-04-19,2026-04-27}*.md
.agents/patterns/pre-tag-ci-validation.md
.agents/findings/f-2026-04-30-002.md (hit_count 2 -> 3, last_cited
2026-05-03)
- New finding f-2026-05-02-006 in registry: release helpers must filter
semver tags. Source: 2026-05-02 full-changelog post-mortem.
- RPI state files (execution-packet, ranked-packet, next-work) +
.agents/ao/last-processed brought current.
skills/pre-mortem/SKILL.md had the same reference link on lines 212
and 223. heal --strict reports DUPLICATE_REF and the skill-integrity
gate fails on every PR until this is fixed. Remove the line-223 dupe
(line 212 is in the logical position after mandatory-checks).
* feat(eval): plumb DisableHooks toggle for LID baseline-A/B (W1.1)
Wire SuiteEnvironment.DisableHooks (suite-declared) and
LiveRuntimeOptions.OverrideDisableHooks (caller-toggled) through
liveEnvironmentRecord, liveRuntimeEnv, and liveRuntimePrompt via the
new effectiveDisableHooks helper. When effective:
- liveEnvironmentRecord populates the existing HooksDisabled placeholder
- liveRuntimeEnv appends AGENTOPS_HOOKS_DISABLED=1 (already honored at
hooks/session-start.sh:7)
- liveRuntimePrompt appends a "do not load skills" constraint sentence
Round-trip JSON snapshot tests for SuiteEnvironment + EnvironmentRecord
per f-2026-04-26-001. Internal helpers for the prompt/env wiring.
Issue: soc-x5yz / W1.1
Plan: .agents/plans/2026-05-02-lid-primitives-eval-integration.md
* feat(eval): --baseline-mode flag + DeltaScorecard runner (W1.2)
Add `ao eval run --baseline-mode={skill-on,skill-off,both}`. When `both`,
runs the suite twice (skill-on, then skill-off via OverrideDisableHooks)
and synthesises a DeltaScorecard with per-case pass/fail delta + aggregate
score delta. cobra ValidArgs reject invalid mode values. New `--delta-out`
writes the scorecard JSON. Per-leg run records get `-skill-on`/`-skill-off`
suffixed paths and run-ids when caller supplies --out / --run-id, so the
two leg artifacts are preserved alongside the scorecard.
Wires OverrideDisableHooks through RunOptions → environmentRecord so the
deterministic SIL path mirrors the live path's HooksDisabled provenance
(W1.1 wired the live path; this commit completes the deterministic side).
Tests: AllBaselineModes/IsValidBaselineMode, deltaSign triad, computeDelta
with mixed pass/fail/missing cases, appendBaselineSuffix path-mangling.
Issue: soc-x5yz / W1.2
Plan: .agents/plans/2026-05-02-lid-primitives-eval-integration.md
* feat(eval): wire DisableHooks into deterministic engine + LID demo fixture (W1.3)
Engine: when OverrideDisableHooks is set, mutate the local suite copy so
runtimeRecord, environmentRecord, and the case command env all see the same
effective DisableHooks state. Inject AGENTOPS_HOOKS_DISABLED=1 into the case
command env (parallels the live-runtime path from W1.1) so SIL fixtures can
gate behavior on the kill-switch.
Demo fixture: evals/agentops-core/lid-primitives-demo.json — two shell cases
that produce the smallest measurable delta proving the baseline-A/B wiring
works end-to-end. Smoke run reports +0.5 aggregate delta:
ao eval run evals/agentops-core/lid-primitives-demo.json --baseline-mode=both
→ skill-on=1.0000 (pass) skill-off=0.5000 (fail) delta=+0.5000 cases=2
Issue: soc-x5yz / W1.3
Plan: .agents/plans/2026-05-02-lid-primitives-eval-integration.md
* docs(eval): contract + code-map for LID baseline-A/B (W1.4)
- docs/contracts/eval-baseline-ab.md: --baseline-mode semantics, what
skill suppression actually disables (and what it doesn't), DeltaScorecard
schema, when to use which mode, SIL vs HIL/VIL scope (HIL/VIL deferred),
backward-compat guarantees, regression guards.
- docs/code-map/eval-lid-primitives.md: file inventory of every Wave 1
surface, test coverage map, live smoke recipe, recipe for adding new
baseline-A/B-eligible suites, open follow-ups (Wave 2 anchor namespace,
HIL/VIL extension, release-readiness gate promotion).
Hook audit confirmed only hooks/session-start.sh:7 auto-honors
AGENTOPS_HOOKS_DISABLED today; future skill-loading hooks must honor it
or future baseline-A/B measurements will under-report.
Issue: soc-x5yz / W1.4
Plan: .agents/plans/2026-05-02-lid-primitives-eval-integration.md
* fix(daemon): Wave A durability — fsync prop, orphan-tmp sweep, snapshot dir-fsync
Three audit follow-ups from .agents/research/2026-05-02-bug-daemon-audit-and-chaos.md:
- W-A-04 (soc-58q5.1): AppendLedgerEvent fsync error propagation. Audit was
STALE — code already returned wrapped Sync error at store.go:211. Added
L2 regression test TestAppendLedgerEvent_SyncErrorPropagates via a
test-only ledgerSyncFn function-variable seam. Caught by reverting line
211 and observing the new test fail.
- W-A-09 (soc-58q5.2): rotateLedger orphan-tmp cleanup. Sweeps
ledger.*.jsonl.gz.tmp files older than 5 min at rotation start (grace
prevents racing concurrent in-flight gzip work). Cleanup failures log at
warn and do not fail rotation. L2 test TestRotation_CleansOrphanedGzTmp
pre-seeds stale + fresh tmps via os.Chtimes, asserts only stale removed.
- W-A-21 (soc-58q5.3): WriteProjectionSnapshot syncDir after rename.
Adds private syncDir helper; calls it after os.Rename(tmp, dst).
syncDir failure DOES fail the function (durability contract). L2 test
TestWriteProjectionSnapshot_SyncsParentDir confirms happy-path remains
green; injection-based negative test deferred (would require interface
threading that costs more than defense-in-depth value).
go test -race -count=1 ./internal/daemon/... green (26.7s).
* fix(ci): close out PR #212 CI failures (W1.x parity + pre-mortem ref link)
- skills/pre-mortem/SKILL.md: add the missing scope-predicate-positive-
negative-cases.md to References (fixes go-build TestSkillContract_
ReferencesLinkedInSKILLMD failure that's been carrying forward from
commit 12fa13b4).
- docs/documentation-index.md: register docs/contracts/eval-baseline-ab.md
(fixes contract-compatibility-gate orphan-contract failure).
- cli/docs/COMMANDS.md: regenerate to capture --baseline-mode and
--delta-out flags added in W1.2 (fixes cli-docs-parity diff failure).
- AGENTS.md: add the hook-output-schema-lint job introduced by 12fa13b4
to the CI jobs table (fixes validate-ci-policy-parity drift).
Local: 11899 tests pass. heal-skill --strict clean. validate-ci-policy-
parity PASS.
Issue: soc-x5yz / W1 CI closure
Cherry-picked from feat/lid-baseline-ab-wave1 commit cdb683df. The
scope-predicate-positive-negative-cases.md reference file was added in
commit 12fa13b4 (mt-olympus 2026-05-02 validation pass) but never linked
from skills/pre-mortem/SKILL.md, breaking the
TestSkillContract_ReferencesLinkedInSKILLMD contract test.
Single-file pick (LID-eval-related changes from cdb683df not included).
Verifies cd cli && go test -count=1 ./cmd/ao/ -run TestSkillContract green.
Three audit follow-ups (final secondary batch):
- S-08 (soc-58q5.16): Reject input.JobID == scheduleSentinelJobID in
SubmitJob. Auto-generated IDs (job_ prefix) and 'schedule_*' user
IDs are unaffected; only the exact sentinel string is rejected.
Test TestSubmitJob_RejectsReservedJobID pins both the rejection
and that 'schedule_*' prefix is allowed.
- S-13 (soc-58q5.19): checkMutationRequestScope godoc documents the
present-then-check pattern as deliberate. Loopback (127.0.0.1) bind
is the security boundary; Origin/Sec-Fetch-Site checks are
defense-in-depth for non-default deployments. Browsers reliably send
these headers for the realistic XSS-from-malicious-site threat;
non-browser callers (curl, ao CLI) are gated by the mutation token.
Doc-only.
- S-14 (soc-58q5.20): writeJSON now logs encoding errors at warn level
via stdlib log.Printf. Single change-point covers all ~50 callsites.
Test TestWriteJSON_LogsEncodingError captures log.Writer and asserts
the error is logged with status code and json.Encoder error detail.
go test -race -count=1 ./internal/daemon/... green (50.2s).
Five audit follow-ups (independent, no inter-deps):
- S-06 (soc-58q5.15): jobLeaseExpired godoc + 6 callsite annotations
marking it as advisory lock-free; authoritative serialization is in
Store.AppendLedgerEvent. Comment-only, no test.
- S-11 (soc-58q5.17): /v1/plans/diff applyReadOnlyEventsLimit parity
with /v1/events. Test TestPlansDiff_AppliesEventsLimit covers
limit=N, default behavior, and 400 on bad limit. Note: neither
endpoint enforces a default cap when ?limit= is omitted; both still
return full ledger — that's a follow-up.
- S-12 (soc-58q5.18): constantTimeTokenEqual. Doc-only. Caller analysis
showed empty-token short-circuit is degenerate (already observable
via 403); non-empty path uses subtle.ConstantTimeCompare which is
timing-safe for equal-length inputs. 22-line godoc captures the
rationale so future security review doesn't re-litigate.
- S-20 (soc-58q5.21): Removed 'if ctx == nil { ctx = Background() }'
mask in plans_executor.go, dream_executor.go, rpi_executor.go.
Caller audit (13 callsites): all pass non-nil ctx (Background, parent,
WithCancel/WithTimeout). Future regressions panic loudly at first
ctx.Err()/ctx.Done() use rather than silently dropping cancellation.
- S-24 (soc-58q5.22): ledger_health LatestSnapshotAge clamped to >= 0
on clock skew (RebuiltAt parsing as future). Test
TestLedgerHealth_ClampsNegativeAge.
go test -race -count=1 ./internal/daemon/... green (56.1s).
Two audit follow-ups (Wave C sub-wave 2):
- W-C-16 (soc-58q5.11): wiki_jobs.go provider required validation.
STALE — code at wiki_jobs.go:107-109 already returns "provider is
required" via Validate(), reachable from both ToJobSpec() and
WikiForgeJobSpecFromPayload(). Added regression test
TestWikiJobs_RejectsEmptyProvider with 4 subtests covering each
validation entrypoint (Validate, ToJobSpec, FromPayload,
Queue.SubmitJob). No production change.
- W-C-19 (soc-58q5.14): dream_executor.go logfile 0o600. Tightened
overnight.log open mode 0o644 -> 0o600. Test
TestDreamExecutor_LogfilePermsAreUserOnly. (W-C-18's MkdirAll/symlink
check preserved.)
go test -race -count=1 ./internal/daemon/... green.
Three audit follow-ups + one defense-in-depth extension (Wave C sub-wave 1):
- W-C-15 (soc-58q5.10): wiki_jobs.go expandedSources CWD containment.
validateWikiForgeSourcePathsContainment helper does filepath.Abs +
filepath.Clean (catches '..' lexically) + EvalSymlinks (catches
symlink traversal when path exists). Repo root canonicalized once via
EvalSymlinks so symlinked tmp dirs (macOS /tmp -> /private/tmp) don't
false-positive. Called twice in runClaimedWikiForgeJob: once on raw
spec.SourcePaths, once on expandedSources (defense in depth). Failure
surfaces as FailureRequestRejected (Retryable=false) BEFORE worker
spawn. Tests: TestWikiJobs_RejectsTraversalSourcePaths (6 subtests
through real queue+runner) + TestValidateWikiForgeSourcePathsContainment_Unit.
- W-C-15 extension (out of original bd scope, in same threat surface):
wiki_executor.go also calls expandWikiForgeSourcePaths and was the
newer/preferred code path. Added the same validate-twice pattern.
Updated 3 pre-existing executor tests to put source paths inside
store.root rather than sibling tempdirs (matches real production
shape where source paths are repo-relative).
- W-C-17 (soc-58q5.12): wiki_executor.go RunJob ctx.Err early-exit.
Added ctx.Err() guard at top of RunJob (returns context.Canceled
before payload parse / source expansion / worker spawn). Per-iteration
check inside source loop for finer-grained cancellation between worker
invocations. Test TestWikiExecutor_RespectsCtxCancelBeforeRun.
- W-C-18 (soc-58q5.13): dream_executor.go output_dir hardening.
os.Lstat symlink rejection before MkdirAll; perms 0o755 -> 0o700
(user-only). Tests TestDreamExecutor_RejectsSymlinkOutputDir +
TestDreamExecutor_OutputDirPermsAreUserOnly.
go test -race -count=1 ./internal/daemon/... green (27.6s).
Two audit follow-ups (Wave B sub-wave 2):
- W-B-10 (soc-58q5.6): refreshSchedules nextTick recompute on Cron change.
Pre-fix: when an existing schedule's Cron field updates, refreshSchedules
did wholesale-replace and recomputed nextTick using the passed-in 'now'
parameter (stale; breaks clock-injection contract per pre-mortem Gap-5).
Also: invalid Cron on update would delete the running schedule entirely.
Fix: in-place reparse + s.clock.Now() for nextTick recompute; invalid
Cron on update keeps prior cadence (logs WARN; typo protection). Tests:
TestRefreshSchedules_RecomputesNextTickOnCronChange (passes stale now to
prove s.clock.Now() is used) + TestRefreshSchedules_KeepsPreviousCadenceOnInvalidCronUpdate.
- W-B-25 (soc-58q5.9): projections ArtifactRefs deep-copy. Found one
asymmetric copy site in initStateFromSnapshot — Artifacts deep-copied,
ArtifactRefs aliased to the source snapshot's underlying map. Added
symmetric deep-copy. ArtifactRef has only scalar fields, so per-key
value copy is sufficient (no recursive clone needed). Test
TestProjections_ArtifactRefsAreCopied mutates source after copy and
asserts destination is unchanged.
go test -race -count=1 ./internal/daemon/... green (28.2s).
Four audit follow-ups (Wave B sub-wave 1):
- W-B-05 (soc-58q5.4): SubmitJob concurrent-IdempotencyKey race. Audit was
RIGHT and the bug is REAL — pre-mortem agent's reproducer test produced
3 distinct JobIDs from 50× concurrent submits with same IdempotencyKey.
Root cause: store-level dedup was by EventID only, not IdempotencyKey.
Fix: extend Store.AppendLedgerEvent to dedup JobAccepted events by
IdempotencyKey under s.mu (the only lock that actually serializes).
appendQueueEvent now returns the durable event so SubmitJob can resolve
the post-dedup JobID. Pre-check in SubmitJob retained as fast path with
corrected godoc. TestSubmitJob_DedupRelyOnStoreLock now PASSES.
- W-B-07 (soc-58q5.5): fireOne single queue snapshot. Refactored
derivedQueueState as a pure function; added queueStateOnce wrapper
documenting single-snapshot intent. Tests: TestFireOne_QueueSnapshotConsistent
+ TestFireOne_DerivedQueueStateIsPure.
- W-B-22 (soc-58q5.7): RebuildProjections error contract. Caller audit
found 3 production + 8 test callers all correct (audit was partially
stale). Added explicit godoc on both Store.RebuildProjections and the
package-level function: callers MUST check err first; on error the set
is zero-valued and field-access will panic. Test
TestRebuildProjections_ErrorReturnsUnusableSet pins both via subtests.
- W-B-23 (soc-58q5.8): ReconcileRPIJobs ctx.Err early-exit. Loop now
checks ctx.Err() at top of each iteration; returns partial report +
context.Canceled on cancel. Test TestReconcileRPIJobs_RespectsCtxCancel
pins immediate-cancel behavior.
go test -race -count=1 ./internal/daemon/... green (32.4s).
Boden Fuller