mirror of
https://github.com/boshu2/agentops.git
synced 2026-09-14 15:08:13 +08:00
Boden Fuller
5da522e91a
Two diagnostic gates landing the prevention work from
.agents/research/2026-05-02-{ratchet-manifest-fixture-drift,silent-revert-root-cause}.md
(findings f-2026-05-02-004 and f-2026-05-02-005).
scripts/check-retrieval-manifest-paths.sh — fails when any ground_truth
path in a retrieval-eval manifest does not resolve via `git ls-tree HEAD`.
Wired into pre-push-gate.sh as 19d (always-on in fast lane). Catches the
generated-fixture failure mode that landed via the codex/evolve manifest
(12 of 17 queries pointed at .agents/learnings/*.md files that never
existed). Manifest itself was already pruned upstream (8dff7b1e).
hooks/edit-audit.sh — PostToolUse hook for Edit/Write/MultiEdit that
appends one JSON line per edit to .agents/ao/edit-audit.log:
{ts, tool, file, sha256, git_status, head_sha}. Diagnostic-only,
non-blocking, gated by AGENTOPS_HOOKS_DISABLED and
AGENTOPS_EDIT_AUDIT_DISABLED. When the silent-revert phenomenon
recurs — Edit succeeds, subsequent grep shows reverted, git status clean —
the audit log lets us prove the post-Edit hash differs from what was on
disk at the next tool invocation, with timestamp + head_sha context to
correlate against bd auto-commits or background processes.
Embedded copies synced via `cd cli && make sync-hooks`. The eval-verdict
+ session-end-maintenance embedded copies were already drifted; the sync
run picked them up alongside the new edit-audit.sh.
Smoke-tested:
- positive case: scripts/check-retrieval-manifest-paths.sh on the pruned
manifest → PASS, 4 paths validated
- negative case: deliberately-bad fixture → FAIL, exit 1, clear message
- edit-audit smoke: synthetic Edit JSON over stdin → JSON line appended
- pre-push-gate.sh --fast → ok retrieval manifest paths, all green
77 lines
2.4 KiB
Bash
Executable File
77 lines
2.4 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# check-retrieval-manifest-paths.sh — validate every ground_truth path in a
|
|
# retrieval-eval manifest resolves against `git ls-tree HEAD`.
|
|
#
|
|
# Catches the failure mode where a generated/merged manifest references files
|
|
# that never existed on any branch (see f-2026-05-02-004). Without this gate,
|
|
# such a manifest passes schema validation but silently collapses retrieval
|
|
# quality metrics (any_relevant_at_k → ~0).
|
|
#
|
|
# Usage:
|
|
# scripts/check-retrieval-manifest-paths.sh <manifest.json> [<manifest.json> ...]
|
|
#
|
|
# Exit codes:
|
|
# 0 — every ground_truth path in every manifest resolves at HEAD
|
|
# 1 — one or more paths missing (details on stderr)
|
|
# 2 — usage error (no manifests, jq missing, file unreadable)
|
|
|
|
set -euo pipefail
|
|
|
|
if [[ $# -lt 1 ]]; then
|
|
echo "usage: $0 <manifest.json> [<manifest.json> ...]" >&2
|
|
exit 2
|
|
fi
|
|
|
|
if ! command -v jq >/dev/null 2>&1; then
|
|
echo "FAIL retrieval manifest paths: jq is required" >&2
|
|
exit 2
|
|
fi
|
|
|
|
if ! command -v git >/dev/null 2>&1; then
|
|
echo "FAIL retrieval manifest paths: git is required" >&2
|
|
exit 2
|
|
fi
|
|
|
|
REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
|
|
|
fail=0
|
|
total=0
|
|
for manifest in "$@"; do
|
|
if [[ ! -f "$manifest" ]]; then
|
|
echo "FAIL retrieval manifest paths: $manifest: not a file" >&2
|
|
fail=1
|
|
continue
|
|
fi
|
|
|
|
# Support both canonical {queries:[{ground_truth:[]}]} and legacy
|
|
# [{relevant:[]}] shapes — same coverage as the Go loader.
|
|
if jq -e 'type == "array"' "$manifest" >/dev/null 2>&1; then
|
|
paths="$(jq -r '.[] | (.relevant // [])[]' "$manifest")"
|
|
else
|
|
paths="$(jq -r '.queries[] | (.ground_truth // [])[]' "$manifest")"
|
|
fi
|
|
|
|
if [[ -z "$paths" ]]; then
|
|
echo "WARN retrieval manifest paths: $manifest has zero ground_truth paths" >&2
|
|
continue
|
|
fi
|
|
|
|
while IFS= read -r path; do
|
|
[[ -z "$path" ]] && continue
|
|
total=$((total + 1))
|
|
# Strip ./ prefix if present
|
|
path="${path#./}"
|
|
if ! git -C "$REPO_ROOT" ls-tree --name-only HEAD -- "$path" 2>/dev/null | grep -qFx "$path"; then
|
|
echo "FAIL retrieval manifest paths: $manifest: ground_truth path missing at HEAD: $path" >&2
|
|
fail=1
|
|
fi
|
|
done <<<"$paths"
|
|
done
|
|
|
|
if [[ $fail -ne 0 ]]; then
|
|
exit 1
|
|
fi
|
|
|
|
echo "PASS retrieval manifest paths: $total ground_truth path(s) validated across $# manifest(s)"
|
|
exit 0
|