mirror of
https://github.com/ruvnet/ruflo.git
synced 2026-09-14 14:01:28 +08:00
rUv
e0d0af1cef
* feat(gaia): pre-submission exploit audit + signed attestation (ADR-167) Motivated by UC Berkeley RDI's April-2026 finding that all 8 major agent benchmarks were gamed to ~98-100% without solving a single task (GAIA leaked ~98% of answers via public answer DBs + normalization collisions; o3/Claude-3.7 monkey-patch graders in 30%+ of runs per METR). Signing proves transport-integrity; this audit proves earning-integrity. The audit report is deterministic ($0, offline, pure-core) and is registered as an ADR-103 witness fix marker so its sha256 + "clean": true marker are signed INTO manifest.md.json. - ADR-167 (v3/docs/adr): threat model, check catalogue mapped to RDI/METR vectors, manifest-attestation design, honest harness-gap limitations. - gaia-audit.mjs: 7 pure checks (answer-leakage, no-work, oracle-leakage, grader-isolation, normalization-collision, voting-disclosure, split-integrity); CLI exit 0 clean / 1 CRITICAL / 2 usage. Schema-blocked checks return skip with a harness_gap note, never a false pass. - Tests: 27 passing; clean + dirty fixtures triggering every vector. - Wired --audit into /gaia validate; pre-sign gate into /gaia submit (refuses CRITICAL fail unless --allow-dirty) + gaia-submission skill. - Bump ruflo-workflows 0.4.0 → 0.5.0. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_01V84YJJK7RBF9TuUo4eVSW7 * docs(adr): ADR-168 — GAIA harness evidence recording (implements ADR-167 §7 forward contract) Co-Authored-By: RuFlo <ruv@ruv.net> * docs(adr): ADR-169 — benchmark reporting integrity standard (strict-EM headlines, view-labeled scaling, disclosed gaps) Codifies the discipline the FRAMES ablation self-audit measured against the Berkeley RDI vectors. Completes the integrity family: ADR-167 audits the submission, ADR-168 records the evidence, ADR-169 governs how numbers are reported. Co-Authored-By: RuFlo <ruv@ruv.net> * feat(gaia): port static source-scan checks (AUD-8/9/10) into gaia-audit Ports the unique, non-overlapping checks from the reverted #2547 duplicate gaia-integrity.mjs into gaia-audit.mjs as a new static source-scan family, matching ADR-167's check-registry / status / harness_gap design: - AUD-8 answer-key-reads (CRITICAL, fail-closed): scans runner sources + produced artifacts for answer/gold/solution/ground-truth-shaped paths referenced outside the sanctioned dataset dir(s). - AUD-9 dynamic-eval (CRITICAL, fail-closed): static scan of the gaia-bench runner sources for eval()/new Function()/exec-family calls with a non-literal first arg. Preserves the #2547 false-positive refinements (multi-line aware, RegExp.prototype.exec excluded, fixed-string commands + comments ignored) — passes clean against ruflo's own multi-line `gcloud secrets … execSync`. - AUD-10 judge-injection (WARN): scans the agent's PRODUCED answer/outputs for judge-directed prompt-injection markers. Distinct from AUD-3 oracle-leakage (input-side gold-answer scan), so both run. Unlike AUD-1/3/4 these need no trajectory instrumentation — they enforce today and skip honestly (never false-pass) when the source population is absent (standalone/ejected run or --skip-source-scan). The #2547 provenance stamp and --allow-integrity-override are NOT ported: #2543's signed attestation and the `/gaia submit --allow-dirty` gate already own provenance + override, so the new checks flow through the same attestation.clean/strict_clean path. CLI adds --repo-root, --dataset-dir (repeatable), --skip-source-scan. Fixtures: planted evil runner (eval + interpolated exec), planted answer-key read, judge-injection results. Tests: 44 green (17 new). Docs: ADR-167 §3/§7 check table, gaia-validate.md, gaia-submission SKILL.md. Co-Authored-By: RuFlo <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_01S7GYqnVUVxBfZ5W8znqry3 * fix(ci): advance ruflo-workflows smoke version assertion to 0.5.0 The AUD-8/9/10 port bumped plugin.json to 0.5.0 but the plugin's own smoke step 1 still asserted 0.4.0 — the single failing step behind both smoke-all and the meta-smoke leg of smoke-without-metaharness on #2543. Co-Authored-By: RuFlo <ruv@ruv.net>