* feat(gaia): pre-submission exploit audit + signed attestation (ADR-167)
Motivated by UC Berkeley RDI's April-2026 finding that all 8 major agent
benchmarks were gamed to ~98-100% without solving a single task (GAIA leaked
~98% of answers via public answer DBs + normalization collisions; o3/Claude-3.7
monkey-patch graders in 30%+ of runs per METR).
Signing proves transport-integrity; this audit proves earning-integrity. The
audit report is deterministic ($0, offline, pure-core) and is registered as an
ADR-103 witness fix marker so its sha256 + "clean": true marker are signed INTO
manifest.md.json.
- ADR-167 (v3/docs/adr): threat model, check catalogue mapped to RDI/METR
vectors, manifest-attestation design, honest harness-gap limitations.
- gaia-audit.mjs: 7 pure checks (answer-leakage, no-work, oracle-leakage,
grader-isolation, normalization-collision, voting-disclosure, split-integrity);
CLI exit 0 clean / 1 CRITICAL / 2 usage. Schema-blocked checks return skip
with a harness_gap note, never a false pass.
- Tests: 27 passing; clean + dirty fixtures triggering every vector.
- Wired --audit into /gaia validate; pre-sign gate into /gaia submit
(refuses CRITICAL fail unless --allow-dirty) + gaia-submission skill.
- Bump ruflo-workflows 0.4.0 → 0.5.0.
Co-Authored-By: claude-flow <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_01V84YJJK7RBF9TuUo4eVSW7
* docs(adr): ADR-168 — GAIA harness evidence recording (implements ADR-167 §7 forward contract)
Co-Authored-By: RuFlo <ruv@ruv.net>
* docs(adr): ADR-169 — benchmark reporting integrity standard (strict-EM headlines, view-labeled scaling, disclosed gaps)
Codifies the discipline the FRAMES ablation self-audit measured against the
Berkeley RDI vectors. Completes the integrity family: ADR-167 audits the
submission, ADR-168 records the evidence, ADR-169 governs how numbers are
reported.
Co-Authored-By: RuFlo <ruv@ruv.net>
* feat(gaia): port static source-scan checks (AUD-8/9/10) into gaia-audit
Ports the unique, non-overlapping checks from the reverted #2547 duplicate
gaia-integrity.mjs into gaia-audit.mjs as a new static source-scan family,
matching ADR-167's check-registry / status / harness_gap design:
- AUD-8 answer-key-reads (CRITICAL, fail-closed): scans runner sources +
produced artifacts for answer/gold/solution/ground-truth-shaped paths
referenced outside the sanctioned dataset dir(s).
- AUD-9 dynamic-eval (CRITICAL, fail-closed): static scan of the gaia-bench
runner sources for eval()/new Function()/exec-family calls with a non-literal
first arg. Preserves the #2547 false-positive refinements (multi-line aware,
RegExp.prototype.exec excluded, fixed-string commands + comments ignored) —
passes clean against ruflo's own multi-line `gcloud secrets … execSync`.
- AUD-10 judge-injection (WARN): scans the agent's PRODUCED answer/outputs for
judge-directed prompt-injection markers. Distinct from AUD-3 oracle-leakage
(input-side gold-answer scan), so both run.
Unlike AUD-1/3/4 these need no trajectory instrumentation — they enforce today
and skip honestly (never false-pass) when the source population is absent
(standalone/ejected run or --skip-source-scan). The #2547 provenance stamp and
--allow-integrity-override are NOT ported: #2543's signed attestation and the
`/gaia submit --allow-dirty` gate already own provenance + override, so the new
checks flow through the same attestation.clean/strict_clean path.
CLI adds --repo-root, --dataset-dir (repeatable), --skip-source-scan.
Fixtures: planted evil runner (eval + interpolated exec), planted answer-key
read, judge-injection results. Tests: 44 green (17 new). Docs: ADR-167 §3/§7
check table, gaia-validate.md, gaia-submission SKILL.md.
Co-Authored-By: RuFlo <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_01S7GYqnVUVxBfZ5W8znqry3
* fix(ci): advance ruflo-workflows smoke version assertion to 0.5.0
The AUD-8/9/10 port bumped plugin.json to 0.5.0 but the plugin's own
smoke step 1 still asserted 0.4.0 — the single failing step behind both
smoke-all and the meta-smoke leg of smoke-without-metaharness on #2543.
Co-Authored-By: RuFlo <ruv@ruv.net>
rUv