boshu2__agentops

skills/boshu2__agentops

Fork 0

mirror of https://github.com/boshu2/agentops.git synced 2026-09-14 15:08:13 +08:00

Commit Graph

Author SHA1 Message Date

Author	SHA1	Message	Date
Boden Fuller	1e7453a7e1	docs(code-map): reconcile A3 audit drift findings Closes soc-r8sm, soc-k8qd, soc-otcj. Why - A3 audit (cycle 36) identified ~6 real drift items in docs/code-map/.md. The three follow-up beads spawned at the end of that cycle are the doc-hygiene closure. What - docs/code-map/agentopsd-codebase-map.md: prefix-fix sweep — bare `internal/<pkg>` references now use `cli/internal/<pkg>` consistently; bare `cmd/ao/...` references now use `cli/cmd/ ao/...`. ~32 line changes. Closes soc-r8sm. - docs/code-map/agentopsd-codebase-map.md: the bad reference to `cli/cmd/ao/eval_finding_coverage.go` no longer appears after the prefix sweep (it was never in a current form). Closes soc-k8qd. - docs/code-map/eval-lid-primitives.md: two stale `.agents/` refs softened to operator-local pointers — `.agents/plans/ 2026-05-02-…md` (now: "operator-local plan archive") and `.agents/council/2026-05-02-…md` (now: "verdict artifact lives operator-local"). Maps shouldn't cite gitignored paths as if they're cross-fleet portable. Closes soc-otcj. Verification - pre-push --fast: green. Companion: A3 audit soc-cd32 (cycle 36).	2026-05-11 19:12:13 -04:00
Bo	b7d11812bd	feat(eval): LID baseline-A/B primitive — Wave 1 (soc-x5yz) (#212 ) * feat(eval): plumb DisableHooks toggle for LID baseline-A/B (W1.1) Wire SuiteEnvironment.DisableHooks (suite-declared) and LiveRuntimeOptions.OverrideDisableHooks (caller-toggled) through liveEnvironmentRecord, liveRuntimeEnv, and liveRuntimePrompt via the new effectiveDisableHooks helper. When effective: - liveEnvironmentRecord populates the existing HooksDisabled placeholder - liveRuntimeEnv appends AGENTOPS_HOOKS_DISABLED=1 (already honored at hooks/session-start.sh:7) - liveRuntimePrompt appends a "do not load skills" constraint sentence Round-trip JSON snapshot tests for SuiteEnvironment + EnvironmentRecord per f-2026-04-26-001. Internal helpers for the prompt/env wiring. Issue: soc-x5yz / W1.1 Plan: .agents/plans/2026-05-02-lid-primitives-eval-integration.md * feat(eval): --baseline-mode flag + DeltaScorecard runner (W1.2) Add `ao eval run --baseline-mode={skill-on,skill-off,both}`. When `both`, runs the suite twice (skill-on, then skill-off via OverrideDisableHooks) and synthesises a DeltaScorecard with per-case pass/fail delta + aggregate score delta. cobra ValidArgs reject invalid mode values. New `--delta-out` writes the scorecard JSON. Per-leg run records get `-skill-on`/`-skill-off` suffixed paths and run-ids when caller supplies --out / --run-id, so the two leg artifacts are preserved alongside the scorecard. Wires OverrideDisableHooks through RunOptions → environmentRecord so the deterministic SIL path mirrors the live path's HooksDisabled provenance (W1.1 wired the live path; this commit completes the deterministic side). Tests: AllBaselineModes/IsValidBaselineMode, deltaSign triad, computeDelta with mixed pass/fail/missing cases, appendBaselineSuffix path-mangling. Issue: soc-x5yz / W1.2 Plan: .agents/plans/2026-05-02-lid-primitives-eval-integration.md * feat(eval): wire DisableHooks into deterministic engine + LID demo fixture (W1.3) Engine: when OverrideDisableHooks is set, mutate the local suite copy so runtimeRecord, environmentRecord, and the case command env all see the same effective DisableHooks state. Inject AGENTOPS_HOOKS_DISABLED=1 into the case command env (parallels the live-runtime path from W1.1) so SIL fixtures can gate behavior on the kill-switch. Demo fixture: evals/agentops-core/lid-primitives-demo.json — two shell cases that produce the smallest measurable delta proving the baseline-A/B wiring works end-to-end. Smoke run reports +0.5 aggregate delta: ao eval run evals/agentops-core/lid-primitives-demo.json --baseline-mode=both → skill-on=1.0000 (pass) skill-off=0.5000 (fail) delta=+0.5000 cases=2 Issue: soc-x5yz / W1.3 Plan: .agents/plans/2026-05-02-lid-primitives-eval-integration.md * docs(eval): contract + code-map for LID baseline-A/B (W1.4) - docs/contracts/eval-baseline-ab.md: --baseline-mode semantics, what skill suppression actually disables (and what it doesn't), DeltaScorecard schema, when to use which mode, SIL vs HIL/VIL scope (HIL/VIL deferred), backward-compat guarantees, regression guards. - docs/code-map/eval-lid-primitives.md: file inventory of every Wave 1 surface, test coverage map, live smoke recipe, recipe for adding new baseline-A/B-eligible suites, open follow-ups (Wave 2 anchor namespace, HIL/VIL extension, release-readiness gate promotion). Hook audit confirmed only hooks/session-start.sh:7 auto-honors AGENTOPS_HOOKS_DISABLED today; future skill-loading hooks must honor it or future baseline-A/B measurements will under-report. Issue: soc-x5yz / W1.4 Plan: .agents/plans/2026-05-02-lid-primitives-eval-integration.md * fix(daemon): Wave A durability — fsync prop, orphan-tmp sweep, snapshot dir-fsync Three audit follow-ups from .agents/research/2026-05-02-bug-daemon-audit-and-chaos.md: - W-A-04 (soc-58q5.1): AppendLedgerEvent fsync error propagation. Audit was STALE — code already returned wrapped Sync error at store.go:211. Added L2 regression test TestAppendLedgerEvent_SyncErrorPropagates via a test-only ledgerSyncFn function-variable seam. Caught by reverting line 211 and observing the new test fail. - W-A-09 (soc-58q5.2): rotateLedger orphan-tmp cleanup. Sweeps ledger..jsonl.gz.tmp files older than 5 min at rotation start (grace prevents racing concurrent in-flight gzip work). Cleanup failures log at warn and do not fail rotation. L2 test TestRotation_CleansOrphanedGzTmp pre-seeds stale + fresh tmps via os.Chtimes, asserts only stale removed. - W-A-21 (soc-58q5.3): WriteProjectionSnapshot syncDir after rename. Adds private syncDir helper; calls it after os.Rename(tmp, dst). syncDir failure DOES fail the function (durability contract). L2 test TestWriteProjectionSnapshot_SyncsParentDir confirms happy-path remains green; injection-based negative test deferred (would require interface threading that costs more than defense-in-depth value). go test -race -count=1 ./internal/daemon/... green (26.7s). fix(ci): close out PR #212 CI failures (W1.x parity + pre-mortem ref link) - skills/pre-mortem/SKILL.md: add the missing scope-predicate-positive- negative-cases.md to References (fixes go-build TestSkillContract_ ReferencesLinkedInSKILLMD failure that's been carrying forward from commit `12fa13b4`). - docs/documentation-index.md: register docs/contracts/eval-baseline-ab.md (fixes contract-compatibility-gate orphan-contract failure). - cli/docs/COMMANDS.md: regenerate to capture --baseline-mode and --delta-out flags added in W1.2 (fixes cli-docs-parity diff failure). - AGENTS.md: add the hook-output-schema-lint job introduced by `12fa13b4` to the CI jobs table (fixes validate-ci-policy-parity drift). Local: 11899 tests pass. heal-skill --strict clean. validate-ci-policy- parity PASS. Issue: soc-x5yz / W1 CI closure	2026-05-02 23:51:32 -04:00

Boden Fuller

1e7453a7e1

docs(code-map): reconcile A3 audit drift findings

Closes soc-r8sm, soc-k8qd, soc-otcj.

Why
- A3 audit (cycle 36) identified ~6 real drift items in
  docs/code-map/*.md. The three follow-up beads spawned at the
  end of that cycle are the doc-hygiene closure.

What
- docs/code-map/agentopsd-codebase-map.md: prefix-fix sweep —
  bare `internal/<pkg>` references now use `cli/internal/<pkg>`
  consistently; bare `cmd/ao/...` references now use `cli/cmd/
  ao/...`. ~32 line changes. Closes soc-r8sm.
- docs/code-map/agentopsd-codebase-map.md: the bad reference to
  `cli/cmd/ao/eval_finding_coverage.go` no longer appears after
  the prefix sweep (it was never in a current form). Closes
  soc-k8qd.
- docs/code-map/eval-lid-primitives.md: two stale `.agents/*`
  refs softened to operator-local pointers — `.agents/plans/
  2026-05-02-…md` (now: "operator-local plan archive") and
  `.agents/council/2026-05-02-…md` (now: "verdict artifact lives
  operator-local"). Maps shouldn't cite gitignored paths as if
  they're cross-fleet portable. Closes soc-otcj.

Verification
- pre-push --fast: green.

Companion: A3 audit soc-cd32 (cycle 36).

2026-05-11 19:12:13 -04:00

b7d11812bd

feat(eval): LID baseline-A/B primitive — Wave 1 (soc-x5yz) (#212 )

* feat(eval): plumb DisableHooks toggle for LID baseline-A/B (W1.1)

Wire SuiteEnvironment.DisableHooks (suite-declared) and
LiveRuntimeOptions.OverrideDisableHooks (caller-toggled) through
liveEnvironmentRecord, liveRuntimeEnv, and liveRuntimePrompt via the
new effectiveDisableHooks helper. When effective:
- liveEnvironmentRecord populates the existing HooksDisabled placeholder
- liveRuntimeEnv appends AGENTOPS_HOOKS_DISABLED=1 (already honored at
  hooks/session-start.sh:7)
- liveRuntimePrompt appends a "do not load skills" constraint sentence

Round-trip JSON snapshot tests for SuiteEnvironment + EnvironmentRecord
per f-2026-04-26-001. Internal helpers for the prompt/env wiring.

Issue: soc-x5yz / W1.1
Plan: .agents/plans/2026-05-02-lid-primitives-eval-integration.md

* feat(eval): --baseline-mode flag + DeltaScorecard runner (W1.2)

Add `ao eval run --baseline-mode={skill-on,skill-off,both}`. When `both`,
runs the suite twice (skill-on, then skill-off via OverrideDisableHooks)
and synthesises a DeltaScorecard with per-case pass/fail delta + aggregate
score delta. cobra ValidArgs reject invalid mode values. New `--delta-out`
writes the scorecard JSON. Per-leg run records get `-skill-on`/`-skill-off`
suffixed paths and run-ids when caller supplies --out / --run-id, so the
two leg artifacts are preserved alongside the scorecard.

Wires OverrideDisableHooks through RunOptions → environmentRecord so the
deterministic SIL path mirrors the live path's HooksDisabled provenance
(W1.1 wired the live path; this commit completes the deterministic side).

Tests: AllBaselineModes/IsValidBaselineMode, deltaSign triad, computeDelta
with mixed pass/fail/missing cases, appendBaselineSuffix path-mangling.

Issue: soc-x5yz / W1.2
Plan: .agents/plans/2026-05-02-lid-primitives-eval-integration.md

* feat(eval): wire DisableHooks into deterministic engine + LID demo fixture (W1.3)

Engine: when OverrideDisableHooks is set, mutate the local suite copy so
runtimeRecord, environmentRecord, and the case command env all see the same
effective DisableHooks state. Inject AGENTOPS_HOOKS_DISABLED=1 into the case
command env (parallels the live-runtime path from W1.1) so SIL fixtures can
gate behavior on the kill-switch.

Demo fixture: evals/agentops-core/lid-primitives-demo.json — two shell cases
that produce the smallest measurable delta proving the baseline-A/B wiring
works end-to-end. Smoke run reports +0.5 aggregate delta:

  ao eval run evals/agentops-core/lid-primitives-demo.json --baseline-mode=both
  → skill-on=1.0000 (pass) skill-off=0.5000 (fail) delta=+0.5000 cases=2

Issue: soc-x5yz / W1.3
Plan: .agents/plans/2026-05-02-lid-primitives-eval-integration.md

* docs(eval): contract + code-map for LID baseline-A/B (W1.4)

- docs/contracts/eval-baseline-ab.md: --baseline-mode semantics, what
  skill suppression actually disables (and what it doesn't), DeltaScorecard
  schema, when to use which mode, SIL vs HIL/VIL scope (HIL/VIL deferred),
  backward-compat guarantees, regression guards.
- docs/code-map/eval-lid-primitives.md: file inventory of every Wave 1
  surface, test coverage map, live smoke recipe, recipe for adding new
  baseline-A/B-eligible suites, open follow-ups (Wave 2 anchor namespace,
  HIL/VIL extension, release-readiness gate promotion).

Hook audit confirmed only hooks/session-start.sh:7 auto-honors
AGENTOPS_HOOKS_DISABLED today; future skill-loading hooks must honor it
or future baseline-A/B measurements will under-report.

Issue: soc-x5yz / W1.4
Plan: .agents/plans/2026-05-02-lid-primitives-eval-integration.md

* fix(daemon): Wave A durability — fsync prop, orphan-tmp sweep, snapshot dir-fsync

Three audit follow-ups from .agents/research/2026-05-02-bug-daemon-audit-and-chaos.md:

- W-A-04 (soc-58q5.1): AppendLedgerEvent fsync error propagation. Audit was
  STALE — code already returned wrapped Sync error at store.go:211. Added
  L2 regression test TestAppendLedgerEvent_SyncErrorPropagates via a
  test-only ledgerSyncFn function-variable seam. Caught by reverting line
  211 and observing the new test fail.

- W-A-09 (soc-58q5.2): rotateLedger orphan-tmp cleanup. Sweeps
  ledger.*.jsonl.gz.tmp files older than 5 min at rotation start (grace
  prevents racing concurrent in-flight gzip work). Cleanup failures log at
  warn and do not fail rotation. L2 test TestRotation_CleansOrphanedGzTmp
  pre-seeds stale + fresh tmps via os.Chtimes, asserts only stale removed.

- W-A-21 (soc-58q5.3): WriteProjectionSnapshot syncDir after rename.
  Adds private syncDir helper; calls it after os.Rename(tmp, dst).
  syncDir failure DOES fail the function (durability contract). L2 test
  TestWriteProjectionSnapshot_SyncsParentDir confirms happy-path remains
  green; injection-based negative test deferred (would require interface
  threading that costs more than defense-in-depth value).

go test -race -count=1 ./internal/daemon/... green (26.7s).

* fix(ci): close out PR #212 CI failures (W1.x parity + pre-mortem ref link)

- skills/pre-mortem/SKILL.md: add the missing scope-predicate-positive-
  negative-cases.md to References (fixes go-build TestSkillContract_
  ReferencesLinkedInSKILLMD failure that's been carrying forward from
  commit 12fa13b4).
- docs/documentation-index.md: register docs/contracts/eval-baseline-ab.md
  (fixes contract-compatibility-gate orphan-contract failure).
- cli/docs/COMMANDS.md: regenerate to capture --baseline-mode and
  --delta-out flags added in W1.2 (fixes cli-docs-parity diff failure).
- AGENTS.md: add the hook-output-schema-lint job introduced by 12fa13b4
  to the CI jobs table (fixes validate-ci-policy-parity drift).

Local: 11899 tests pass. heal-skill --strict clean. validate-ci-policy-
parity PASS.

Issue: soc-x5yz / W1 CI closure

2026-05-02 23:51:32 -04:00

2 Commits