mirror of
https://github.com/boshu2/agentops.git
synced 2026-09-14 15:08:13 +08:00
Bo
51f5d64474
## What this is The eval program for the skill harness, end to end: SOTA research (two adversarially-verified deep-research passes) → a 12-decision eval architecture → harness completion → the first measured probe wave → skill improvements driven by the measurements. ## Wave-1 results (gpt-5.6-luna, xhigh + low effort, N=2/arm/config, directional) | Probe | Skill | xhigh C→T | low C→T | Verdict | |---|---|---|---|---| | premortem-self-validation | premortem | 0.5→1.0 | 0.0→1.0 | **BEHAVIORAL** | | standards-go-conventions | standards | 0.5→1.0 | 0.0→1.0 | **BEHAVIORAL** | | validate-not-proven | validate | 1.0→1.0 | 1.0→1.0 | INERT (ceiling) | | security-coverage-gap | security | 1.0→1.0 | 1.0→1.0 | INERT (ceiling) | | reality-check-gap | reality-check | 1.0→1.0 | 1.0→1.0 | INERT (ceiling) | | crank-luna | crank | 1.0→1.0 | 1.0→1.0 | INERT (3rd config) | The effect **grows as the producer weakens** — the SkillsBench gradient reproduced locally. Full report: `docs/evals/2026-08-04-probe-wave-1.md`. ## Verification (all in this branch's tree) - Go build + vet + 415 tests green (adapters/eval, eval, gates) - validate-skill-frontmatter 51/51; heal --check clean - shellcheck -S warning clean on all new/modified shell - probe-coverage gate: 4/11 measured (was 0/11); bats suite green - `_stats` vendored: 42 pytest green - Every wave cell replayable: `bash scripts/probe-skill.sh --probe <id> --replay` ## Notes for review - `evals/skill-probes/LEDGER.md` is the new hand-maintained MEASURED ledger — moved out of generated SKILL-TIERS.md because a regen wiped it there (measured results can't live in generated files). - skills/premortem + skills/standards gained front-loaded MEASURED blocks (the exact prelude content the probes proved behavioral); codex twins regenerated on this clean baseline. - The architecture doc is PROPOSED — ratification is the repo owner's call; nothing in it is load-bearing for this PR's code.