mirror of
https://github.com/boshu2/agentops.git
synced 2026-09-14 15:08:13 +08:00
cca8fc6731
* docs(mkdocs): fix 5 anchor mismatches + 1 unresolvable relative link (soc-2yx9) mkdocs slugify collapses em-dash spacing in headings to a single hyphen, so '## A — B' produces id="a-b", not 'a--b'. Five doc cross-references encoded the visual double-dash and were silently flagged INFO by `mkdocs build --strict`. Fixes (slugs verified against built site/ HTML): - docs/GLOSSARY.md → how-it-works.md#ralph-wiggum-pattern-fresh-context-every-wave - docs/cdlc.md (×2) → the-science.md#part-6-the-convergence-cdlc-as-the-unifying-spine - docs/context-lifecycle.md (×2) → #the-knowledge-ledger-session-to-session-flow - docs/releases/2026-03-21-v2.28.0-notes.md → ../CHANGELOG.md#2280-2026-03-21 Plus one structural fix: - docs/documentation-index.md → ../evals/workbench/ replaced with the github.com URL pattern used by every other extra-docs link in this file. evals/ lives outside docs_dir so mkdocs cannot resolve relative links there. Acceptance: zero INFO/WARN/ERROR mentioning these 5 files in `mkdocs build --strict`. Pre-existing INFO lines on templates/ and L*-*/ README directory links are out of scope (research §U4 / pre-mortem L3). Closes soc-2yx9 from epic soc-xlw8. * fix(goals): anchor cwd to GOALS.md repo root for `ao goals measure` (soc-crzz) Reproduction: from /tmp, `ao goals measure --file <abs>/GOALS.md` returned 21/22 fails with "bash: scripts/check-*.sh: No such file or directory". Goal Check strings are relative paths; bash inherited the caller's cwd. Fix: at RunMeasure entry, derive the repo root from the absolute GoalsFile path and chdir to it for the duration of the measurement; restore the prior cwd via deferred no-op-safe restorer (`withGoalFileCwd`). When GoalsFile is not inside a git repo, the helper returns a no-op so today's behavior is preserved outside repos. Also expose `Paths.RepoRoot` from cli/internal/paths so future callers can detect the resolved repo top without re-shelling git. The field is empty when the resolver was given a directory not inside a repo, mirroring `git rev-parse --show-toplevel` semantics. Smoke (post-fix): `cd /tmp && ao goals measure --file <abs>/GOALS.md --json` returns 22/22 pass, zero "No such file" errors. Tests: - TestRunMeasure_RelativeScripts_FromExternalCwd exercises the bug path end-to-end (fake git repo + relative-script goal + chdir to outside). - TestWithGoalFileCwd_NoRepoFallback verifies the no-repo no-op. - TestWithGoalFileCwd_EmptyPath verifies the empty-input no-op. Per pre-mortem H1 (.agents/council/2026-05-07-pre-mortem-drain-open-next-work-items.md). Closes soc-crzz from epic soc-xlw8. * feat(eval): add stdout_contains_auto_detect ratchet for live test counts (soc-bvhn) Two eval pins drifted silently when tests were added: - evals/agentops-core/hook-lifecycle-behavior.json "Total: 226" - evals/agentops-core/pre-push-gate-governance.json "1..51" The 226-pin only counted tests/hooks/test-hooks.sh and missed the 86-test test-hooks.bats suite entirely; the 1..51-pin was a manually-bumped plan line (commitbd104270was 50→51 after PR #263 added a test). Replace both with a new expectation type stdout_contains_auto_detect that: 1. Re-executes a source-of-truth command (e.g. `bash tests/hooks/test-hooks.sh`) 2. Captures a regex group from its combined stdout+stderr 3. Verifies the case stdout contains the FULL match (drift = fail loud) 4. Honors tolerance_pct (default 0; strict-by-default) 5. Pre-checks the command's binary on PATH so missing `bats` etc. fails with an explicit "not found on PATH" message instead of regex miss 6. Inherits the case's cwd / suite-dir + timeout policy Drift surfaces a `BUMP the pinned value to "Total: N"` one-liner so the operator can react in one edit. Migrated both pins to use the new type. Pre-mortem M3 (.agents/council/2026-05-07-pre-mortem-drain-open-next-work-items.md) patch is enforced indirectly: validExpectationType registers the new kind, and the parser rejects empty command/pattern at load time. Tests: 8 new in expectations_auto_detect_test.go covering pass-on-match, fail-loud-on-drift, missing-binary-fails-loud, pattern-no-match, the bats `1..N` plan-line shape, missing-command rejection, tolerance allowing numeric drift, and registry membership. Closes soc-bvhn from epic soc-xlw8. * feat(reconcile): observation-log aggregator unblocks Wave 1E (soc-ejq2) Wave 1E (soc-f42z9) of the Reconciliation Engine epic cannot promote the factory-claim-ledger validator from advisory to blocking until ≥20 advisory runs are aggregated to .agents/reconcile/observation-log.jsonl. The CI job (`factory-claim-ledger-strict (advisory)` at validate.yml:450-565) already emits per-run observation JSON as a workflow artifact; nothing aggregated them. This builds the missing pull-mode aggregator. scripts/aggregate-observation-log.sh: 1. `gh run list --workflow validate.yml ...` for recent runs 2. `gh run download <run-id> -n factory-claim-ledger-observation` per run (silently skips runs that predate the advisory job) 3. Schema-validates EVERY observation BEFORE dedup (pre-mortem M3 patch: `jq unique_by(.run_id)` silently collapses null-key entries — we reject malformed observations up front with `jq -e all(.run_id and ...)`) 4. Dedups on run_id, atomic write to .agents/reconcile/observation-log.jsonl 5. Backfills merged_anyway and ledger_updated: - pr_number=null (push-to-main): both false. Push-to-main observations satisfy the no-silent-merge criterion by definition. - else: `gh pr view` for state + `git log` for ledger-touch detection Empty-output case (no runs yet): produces valid empty JSONL, exits 0. Idempotent: re-running does not duplicate; verified by L2 test. Skeleton .agents/reconcile/promotion-decision.md (gitignored, not in repo) seeded by the script for operator to fill in false-positive baseline. L1+L2 fixture-driven tests (9/9 pass): empty-input, dedup, schema rejection of null run_id, push-to-main backfill semantics, --dry-run no-write, idempotent byte-identical re-runs. Adds `reconcile` to docs/contracts/agents-write-surfaces.md (prose table and BEGIN/END allowlist) — required for `check-agents-write-surfaces.sh` to pass after the new write surface lands. NOT in scope (deferred to follow-up issues): - Scheduled .github/workflows/aggregate-observations.yml workflow - Operator-filled promotion-decision.md baseline - Wave 1E gate flip itself (soc-f42z9) Per pre-mortem M3 (.agents/council/2026-05-07-pre-mortem-drain-open-next-work-items.md). Closes soc-ejq2 from epic soc-xlw8. * feat(crank): wire CI-policy parity gate into wave acceptance (soc-il9k) Prevents recurrence of c587b361-style manual fixes. Wave 1C of soc-e4ulx added factory-claim-ledger-strict (advisory) to .github/workflows/validate.yml without updating AGENTS.md or `summary.needs:`. Codex-team caught and fixed the drift manually. This PR formalizes that recovery as a conditional crank wave-acceptance gate, so the next wave that touches workflow YAML fails loud at acceptance instead of slipping through. Insertion (canonical and codex-mirrored): - skills/crank/SKILL.md Step 5.5: new "CI-Policy Parity Gate (conditional)" sub-section pointing to the worked example. - skills/crank/references/wave-patterns.md: appended worked example with drift→fix transcript referencing commitc587b361. - skills/swarm/references/local-mode.md (worker prompt template): added section 4 to CONDITIONAL PREFLIGHT CHECKS so workers run the gate proactively, not only at acceptance. - skills-codex/* parity copies + regenerated codex hashes. Trigger (narrow grep — yaml workflow files only): if git diff --name-only HEAD~1 -- | grep -qE '^\.github/workflows/.*\.ya?ml$'; then bash scripts/validate-ci-policy-parity.sh || exit 1 fi CODEOWNERS-only or markdown-only changes do not trip the gate. The validator (scripts/validate-ci-policy-parity.sh, 186 LOC, unchanged) is a read-only dependency. Golden fixture tests/integration/test-ci-policy-parity-wave-gate.sh (9/9 pass): SKILL doc presence, trigger-pattern shape, worked-example linkage, worker-template wiring, codex parity, validator presence, drifted-fixture exits 1, aligned-fixture exits 0, live-repo PASS. Per planning rule finding-2026-05-07-ci-parity-as-wave-acceptance — this PR IS the formalization of that rule (self-referential by design). Closes soc-il9k from epic soc-xlw8. * feat(reconcile): thesis-stability gate + Wave 0 snapshot (soc-mt50) Gates the Reconciliation Engine arc transition from Wave 1 to Wave 2 (soc-r3y8b) by diffing current README/PRODUCT/GOALS hero sections against the snapshot frozen at Wave 0 close (commitab479e26— "docs(positioning): wiki-framing sweep across all surface docs (soc-9xn0)"). scripts/check-thesis-stability.sh: Hero extractor anchored to '^## ' (with trailing space) per pre-mortem M2 patch — code-fence-safe, excludes the H2 boundary line itself: awk 'NR==1, /^## / {if (!/^## /) print}' <file> Per-file diff against the snapshot's fenced ```...``` block. Operator decides accept/re-brainstorm/incidental on FAIL. Exit codes: 0 = no drift, 1 = drift (operator decision required), 2 = precondition error (snapshot missing, source files missing, or snapshot carries a literal `^<!-- WAVE_0_TODO -->$` line — anchored regex prevents prose mentions of the marker from triggering the precondition gate, per pre-mortem L6). --json mode emits {verdict, drifted, decision_template} for tooling. Snapshot artifact (.agents/reconcile/wave-0-thesis-snapshot.md): Frozen verbatim hero blocks for README.md / PRODUCT.md / GOALS.md at closure SHAab479e26. Header documents extraction contract + regen path. Decision template (.agents/reconcile/thesis-stability-decision.md): Operator-fillable record. Three explicit options (accept / re-brainstorm / incidental). Re-validation matrix for Waves 2-4 acceptance criteria on accept-drift path. .gitignore: Adds the snapshot + decision-template paths to the .agents/ allowlist alongside existing exceptions (rpi/next-work.jsonl, nightly/, evolve/, goals/, findings/registry.jsonl). observation-log.jsonl and promotion-decision.md remain runtime state (generated by aggregator and operator fill-in respectively) and stay ignored. Smoke (run during implementation): - bash scripts/check-thesis-stability.sh → exit 0 (no drift today) - bash scripts/check-thesis-stability.sh --help → usage emitted, exit 0 - inject SMOKE-DRIFT into README hero, re-run → exit 1 with diff - revert, re-run → exit 0 Per pre-mortem M2 + L6 (.agents/council/2026-05-07-pre-mortem-drain-open-next-work-items.md). Closes soc-mt50 from epic soc-xlw8. * ci(reconcile): allowlist thesis-stability snapshot + decision artifacts (soc-mt50 follow-up) PR-E (06f416e1) added two static contract artifacts under .agents/reconcile/ that need to be tracked: wave-0-thesis-snapshot.md (frozen at Wave 0 SHAab479e26) and thesis-stability-decision.md (operator-fillable template). The .gitignore exception alone is not sufficient — scripts/check-no-tracked-agents.sh maintains a narrow audit-truth allowlist of paths under .agents/ that may be tracked. Without an allowlist update, the pre-push-gate's "tracked .agents state" check fails on these two new files. Extends ALLOWED_PATHS_REGEX and ALLOWED_REINCLUDES_REGEX to include the two reconcile paths. observation-log.jsonl + promotion-decision.md intentionally remain runtime state (generated by the aggregator and operator fill-in respectively) and are NOT added to the allowlist. Updates the file's header comment to document the new reconcile entry in the audit-truth allowlist surface. Acceptance: - bash scripts/check-no-tracked-agents.sh → "no disallowed tracked repo-root .agents state" - bash scripts/pre-push-gate.sh --fast → exit 0 (46 skipped, 0 failed) Per CLAUDE.md "All exceptions require a coordinated update of .gitignore + this allowlist + CLAUDE.md/PROGRAM.md guidance." (Reconciliation Engine arc context covered in PR-E commit message.) * chore(rpi): mark soc-9xn0 + soc-e4ulx next-work entries consumed (soc-ot2m) Closes the housekeeping tail of epic soc-xlw8. Both unconsumed entries in .agents/rpi/next-work.jsonl flip to consumed=true, claim_status=consumed, consumed_by=soc-xlw8, consumed_at=2026-05-08T09:30:00-04:00. Mapping (10 raw items → 7 work units → 6 PRs; U1 already resolved by Wave 1A pre-session, no PR needed): U2 BATS count audit gap → PR-C f75201f5 (soc-bvhn) U3 ao goals measure cwd → PR-B 04e585e8 (soc-crzz) U4 mkdocs link rot + anchors → PR-A cf584671 (soc-2yx9) U5 observation-log aggregator → PR-D af681e9c (soc-ejq2) U6 thesis-stability gate → PR-E 06f416e1 (+93e4cf05) U7 CI-policy parity in crank → PR-F a2fb6a2f (soc-il9k) Closure note (gitignored, local provenance): .agents/rpi/closure-2026-05-07-next-work-items.md Also adds .agents/findings/registry.jsonl with three reusable findings generated during this session (the file was untracked due to .agents/ gitignore precedence; the audit-truth allowlist already permits it): - finding-2026-05-07-mkdocs-slug-observed-not-asserted - finding-2026-05-07-sed-range-includes-boundary-line - finding-2026-05-07-jq-unique-by-collapses-null-keys Acceptance: - bash scripts/validate-next-work-contract-parity.sh → exit 0 - jq -c . < .agents/rpi/next-work.jsonl → valid JSONL, 2 entries - bash scripts/check-no-tracked-agents.sh → no disallowed paths - bash scripts/pre-push-gate.sh --fast → exit 0 (46 skipped) Per pre-mortem L5 (.agents/council/2026-05-07-pre-mortem-drain-open-next-work-items.md): the validator was already passing pre-PR (the phase-2-summary's claim of broken parity was stale). Acceptance is steady-state ("validator continues to exit 0 after marking entries consumed"), not transition. Closes soc-ot2m and epic soc-xlw8. * chore(reconcile): record Wave 1E promotion-decision baseline (Option B defer) Filled .agents/reconcile/promotion-decision.md from skeleton with computed baseline from observation-log.jsonl: 3 obs / 0 fail / 2 PRs / vacuous FP rate. Sample 3 < 20 threshold and PRs 2 < 3 threshold do not justify promotion to blocking; first-day post-aggregator data is also clustered within ~4h. Selected Option B (defer/extend sampling) with explicit re-evaluation trigger documented (>=20 runs, >=3 PRs, >=1 fail). validate.yml gate stays advisory (continue-on-error: true intact). No behavior change to CI. Re-evaluation tracked under bd soc-nsuq, which blocks Wave 1E parent soc-f42z9. Audit-truth allowlist extended in both .gitignore and scripts/check-no-tracked-agents.sh to track the decision artifact, mirroring the thesis-stability-decision.md allowlist pattern from 93e4cf05. The promotion-decision is committed input to the Wave 1E gate-flip workflow, not runtime state. Refs: soc-xlw8 (RPI follow-up), soc-f42z9 (Wave 1E parent), soc-nsuq (re-evaluation), soc-ejq2 (aggregator that produced the baseline).
162 lines
5.3 KiB
Bash
Executable File
162 lines
5.3 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# check-thesis-stability.sh — Thesis-Stability Gate for the Reconciliation
|
|
# Engine arc (soc-r3y8b, between Wave 1 and Wave 2).
|
|
#
|
|
# Bead: soc-mt50 (PR-E, Wave 1H of epic soc-xlw8)
|
|
# Plan: .agents/plans/2026-05-07-drain-open-next-work-items.md §soc-mt50
|
|
#
|
|
# What it does
|
|
# ------------
|
|
# Diffs the current README.md / PRODUCT.md / GOALS.md hero sections against
|
|
# the snapshot frozen at Wave 0 close (.agents/reconcile/wave-0-thesis-snapshot.md).
|
|
# A diff = thesis drift; the operator must consciously accept it (and
|
|
# re-validate Waves 2-4) before proceeding, OR re-brainstorm Waves 2-4.
|
|
#
|
|
# Hero extraction (per pre-mortem M2):
|
|
# awk 'NR==1, /^## / {if (!/^## /) print}' <file>
|
|
# This anchors to ^## (with the H2 boundary literal) and excludes the
|
|
# boundary line from the captured hero. Code-fence-safe because it matches
|
|
# only line-leading "## ".
|
|
#
|
|
# Exit codes:
|
|
# 0 = no drift (proceed to Wave 2)
|
|
# 1 = drift detected (operator decision required)
|
|
# 2 = precondition error (snapshot missing, source files missing, or
|
|
# snapshot contains a WAVE_0_TODO marker)
|
|
#
|
|
# Usage:
|
|
# bash scripts/check-thesis-stability.sh
|
|
# bash scripts/check-thesis-stability.sh --json
|
|
# bash scripts/check-thesis-stability.sh --help
|
|
|
|
set -euo pipefail
|
|
|
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
SNAPSHOT="${THESIS_STABILITY_SNAPSHOT:-$REPO_ROOT/.agents/reconcile/wave-0-thesis-snapshot.md}"
|
|
DECISION_TEMPLATE="$REPO_ROOT/.agents/reconcile/thesis-stability-decision.md"
|
|
|
|
JSON=0
|
|
case "${1:-}" in
|
|
-h|--help)
|
|
cat <<USAGE
|
|
Usage: bash scripts/check-thesis-stability.sh [--json]
|
|
|
|
Gates the Reconciliation Engine arc transition from Wave 1 to Wave 2 by
|
|
diffing current README/PRODUCT/GOALS hero sections against the Wave 0
|
|
snapshot at $SNAPSHOT.
|
|
|
|
Exit 0 = no drift; 1 = drift (operator decision); 2 = precondition error.
|
|
USAGE
|
|
exit 0
|
|
;;
|
|
--json)
|
|
JSON=1
|
|
;;
|
|
"")
|
|
;;
|
|
*)
|
|
echo "ERROR: unknown argument: $1" >&2
|
|
exit 2
|
|
;;
|
|
esac
|
|
|
|
# Hero extractor — keep the awk pattern in lockstep with the snapshot
|
|
# header's documented contract.
|
|
extract_hero() {
|
|
awk 'NR==1, /^## / {if (!/^## /) print}' "$1"
|
|
}
|
|
|
|
# Snapshot section extractor — pulls the fenced code block under
|
|
# "## <FILE>.md hero".
|
|
extract_snapshot_section() {
|
|
local file="$1"
|
|
awk -v target="## ${file} hero" '
|
|
$0 == target { in_section = 1; next }
|
|
in_section && /^## / { exit }
|
|
in_section && /^```/ { in_block = !in_block; next }
|
|
in_section && in_block { print }
|
|
' "$SNAPSHOT"
|
|
}
|
|
|
|
# Precondition: snapshot exists.
|
|
if [[ ! -f "$SNAPSHOT" ]]; then
|
|
echo "ERROR: snapshot file missing: $SNAPSHOT" >&2
|
|
echo " Run PR-E (soc-mt50) to seed the snapshot from Wave 0 closure SHA." >&2
|
|
exit 2
|
|
fi
|
|
|
|
# Precondition: snapshot is not a TODO stub. The marker must be on its own
|
|
# line to count — backtick-wrapped or quoted prose mentions of the marker
|
|
# are explanatory text, not stubs.
|
|
if grep -qE '^<!-- WAVE_0_TODO -->[[:space:]]*$' "$SNAPSHOT"; then
|
|
echo "ERROR: snapshot contains WAVE_0_TODO marker — gate cannot evaluate." >&2
|
|
echo " Operator must finalize $SNAPSHOT before this gate is meaningful." >&2
|
|
exit 2
|
|
fi
|
|
|
|
# Precondition: source files exist.
|
|
declare -a missing=()
|
|
for f in README.md PRODUCT.md GOALS.md; do
|
|
[[ -f "$REPO_ROOT/$f" ]] || missing+=("$f")
|
|
done
|
|
if (( ${#missing[@]} > 0 )); then
|
|
echo "ERROR: source files missing: ${missing[*]}" >&2
|
|
exit 2
|
|
fi
|
|
|
|
drift_count=0
|
|
declare -a drifted=()
|
|
|
|
for f in README PRODUCT GOALS; do
|
|
src="$REPO_ROOT/${f}.md"
|
|
current="$(extract_hero "$src")"
|
|
pinned="$(extract_snapshot_section "${f}.md")"
|
|
if [[ -z "$pinned" ]]; then
|
|
echo "ERROR: snapshot is missing the '${f}.md hero' section." >&2
|
|
exit 2
|
|
fi
|
|
if [[ "$current" != "$pinned" ]]; then
|
|
drifted+=("${f}.md")
|
|
drift_count=$((drift_count + 1))
|
|
if [[ "$JSON" -eq 0 ]]; then
|
|
echo "DRIFT: ${f}.md hero differs from Wave 0 snapshot" >&2
|
|
diff <(printf '%s\n' "$pinned") <(printf '%s\n' "$current") >&2 || true
|
|
echo "" >&2
|
|
fi
|
|
fi
|
|
done
|
|
|
|
if (( drift_count == 0 )); then
|
|
if [[ "$JSON" -eq 1 ]]; then
|
|
echo '{"verdict":"pass","drifted":[]}'
|
|
else
|
|
echo "thesis-stability: PASS (no drift in README/PRODUCT/GOALS hero sections vs snapshot)"
|
|
fi
|
|
exit 0
|
|
fi
|
|
|
|
if [[ "$JSON" -eq 1 ]]; then
|
|
printf '{"verdict":"fail","drifted":['
|
|
sep=""
|
|
for d in "${drifted[@]}"; do
|
|
printf '%s"%s"' "$sep" "$d"
|
|
sep=","
|
|
done
|
|
printf '],"decision_template":"%s"}\n' "$DECISION_TEMPLATE"
|
|
else
|
|
echo ""
|
|
echo "thesis-stability: FAIL (${drift_count} surface(s) drifted: ${drifted[*]})"
|
|
echo ""
|
|
echo "Operator action required. Two paths:"
|
|
echo " 1. Accept drift: thesis has evolved. Document acceptance in"
|
|
echo " $DECISION_TEMPLATE"
|
|
echo " and re-validate Waves 2-4 acceptance criteria against the new thesis."
|
|
echo " 2. Re-brainstorm Waves 2-4: thesis change invalidates downstream"
|
|
echo " plan; restart from /brainstorm before continuing."
|
|
echo ""
|
|
echo "If the drift is incidental (e.g. typo fix), regenerate the snapshot"
|
|
echo "from a known-good SHA via git and document the rationale in the"
|
|
echo "decision template."
|
|
fi
|
|
exit 1
|