* feat(gaia): pre-submission exploit audit + signed attestation (ADR-167)
Motivated by UC Berkeley RDI's April-2026 finding that all 8 major agent
benchmarks were gamed to ~98-100% without solving a single task (GAIA leaked
~98% of answers via public answer DBs + normalization collisions; o3/Claude-3.7
monkey-patch graders in 30%+ of runs per METR).
Signing proves transport-integrity; this audit proves earning-integrity. The
audit report is deterministic ($0, offline, pure-core) and is registered as an
ADR-103 witness fix marker so its sha256 + "clean": true marker are signed INTO
manifest.md.json.
- ADR-167 (v3/docs/adr): threat model, check catalogue mapped to RDI/METR
vectors, manifest-attestation design, honest harness-gap limitations.
- gaia-audit.mjs: 7 pure checks (answer-leakage, no-work, oracle-leakage,
grader-isolation, normalization-collision, voting-disclosure, split-integrity);
CLI exit 0 clean / 1 CRITICAL / 2 usage. Schema-blocked checks return skip
with a harness_gap note, never a false pass.
- Tests: 27 passing; clean + dirty fixtures triggering every vector.
- Wired --audit into /gaia validate; pre-sign gate into /gaia submit
(refuses CRITICAL fail unless --allow-dirty) + gaia-submission skill.
- Bump ruflo-workflows 0.4.0 → 0.5.0.
Co-Authored-By: claude-flow <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_01V84YJJK7RBF9TuUo4eVSW7
* docs(adr): ADR-168 — GAIA harness evidence recording (implements ADR-167 §7 forward contract)
Co-Authored-By: RuFlo <ruv@ruv.net>
* docs(adr): ADR-169 — benchmark reporting integrity standard (strict-EM headlines, view-labeled scaling, disclosed gaps)
Codifies the discipline the FRAMES ablation self-audit measured against the
Berkeley RDI vectors. Completes the integrity family: ADR-167 audits the
submission, ADR-168 records the evidence, ADR-169 governs how numbers are
reported.
Co-Authored-By: RuFlo <ruv@ruv.net>
* feat(gaia): port static source-scan checks (AUD-8/9/10) into gaia-audit
Ports the unique, non-overlapping checks from the reverted #2547 duplicate
gaia-integrity.mjs into gaia-audit.mjs as a new static source-scan family,
matching ADR-167's check-registry / status / harness_gap design:
- AUD-8 answer-key-reads (CRITICAL, fail-closed): scans runner sources +
produced artifacts for answer/gold/solution/ground-truth-shaped paths
referenced outside the sanctioned dataset dir(s).
- AUD-9 dynamic-eval (CRITICAL, fail-closed): static scan of the gaia-bench
runner sources for eval()/new Function()/exec-family calls with a non-literal
first arg. Preserves the #2547 false-positive refinements (multi-line aware,
RegExp.prototype.exec excluded, fixed-string commands + comments ignored) —
passes clean against ruflo's own multi-line `gcloud secrets … execSync`.
- AUD-10 judge-injection (WARN): scans the agent's PRODUCED answer/outputs for
judge-directed prompt-injection markers. Distinct from AUD-3 oracle-leakage
(input-side gold-answer scan), so both run.
Unlike AUD-1/3/4 these need no trajectory instrumentation — they enforce today
and skip honestly (never false-pass) when the source population is absent
(standalone/ejected run or --skip-source-scan). The #2547 provenance stamp and
--allow-integrity-override are NOT ported: #2543's signed attestation and the
`/gaia submit --allow-dirty` gate already own provenance + override, so the new
checks flow through the same attestation.clean/strict_clean path.
CLI adds --repo-root, --dataset-dir (repeatable), --skip-source-scan.
Fixtures: planted evil runner (eval + interpolated exec), planted answer-key
read, judge-injection results. Tests: 44 green (17 new). Docs: ADR-167 §3/§7
check table, gaia-validate.md, gaia-submission SKILL.md.
Co-Authored-By: RuFlo <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_01S7GYqnVUVxBfZ5W8znqry3
* fix(ci): advance ruflo-workflows smoke version assertion to 0.5.0
The AUD-8/9/10 port bumped plugin.json to 0.5.0 but the plugin's own
smoke step 1 still asserted 0.4.0 — the single failing step behind both
smoke-all and the meta-smoke leg of smoke-without-metaharness on #2543.
Co-Authored-By: RuFlo <ruv@ruv.net>
* feat(workflows): add native Workflow JS orchestration surface (ADR-0002)
ruflo-workflows now documents both workflow surfaces: the existing 10
workflow_* MCP tools and the native Claude Code Workflow JS capability
(.claude/workflows/*.js — agent/parallel/pipeline/phase fan-out).
- ADR-0002 (Accepted): adopts native orchestration alongside the MCP surface
- Reference workflow .claude/workflows/plugin-contract-audit.js (fans smoke
contracts across all plugins, diagnoses failures in parallel)
- README: native orchestration section + four-hook API + surface decision table
- workflow-create/workflow-run skills, workflow-specialist agent, /workflow
command made surface-aware
- plugin.json 0.3.0 -> 0.4.0; native-workflow keywords + component block
- smoke.sh reconciled (stale version + ADR-0001 status) and extended 11 -> 15
checks; smoke-gaia.sh version assertion bumped to 0.4.0
Co-Authored-By: RuFlo <ruv@ruv.net>
* fix(routing): #bugB stale route cache + #bugC --explore false (3.10.8)
Two routing-learning correctness bugs from the intelligence audit's remaining
punch-list (docs/reviews/intelligence-system-audit-2026-05-29.md §Remediation).
- Bug B (stale route cache): QLearningRouter.update() only invalidated the
whole route cache every 50 updates, so a freshly-learned Q-update stayed
hidden behind a stale cached decision — feedback appeared to have no effect
on routing in-process until 50 updates accumulated. Now update() invalidates
the updated state's cache entry immediately (new invalidateCacheEntry).
Verified: learned route flips coder→researcher within 10 updates (was 50+).
- Bug C (--explore false ignored): boolean flags dropped an explicit space-form
value, forcing a default-true boolean (explore) to true even with
, so exploitation could never be forced. parser.ts now
consumes a true/false literal for boolean flags (--explore false / -e false),
while --explore=false and --no-explore keep working. Verified deterministic.
+4 regression tests (15/15 bug-cluster pass); 52/52 parser tests pass; cli
build clean. Audit doc updated with full remediation status (3.10.7 + 3.10.8
shipped; SONA-default/MicroLoRA/EWC-Fisher/per-task-bandit deferred with
honest rationale — the latter two need an ADR/upstream fix, not a patch).
Co-Authored-By: RuFlo <ruv@ruv.net>
ruvnet