skills/ruvnet__ruflo
1
0
Fork 0
mirror of https://github.com/ruvnet/ruflo.git synced 2026-09-14 14:01:28 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
docs/x-gateway-github-style-metadata
ruvnet__ruflo/verification/README.md
T
rUv 8aac1b2ccc feat(mcp): ADR-112 — 285/285 tool descriptions have 'Use when …' guidance + CI guard + #1892 statusline (#1897) 
* docs(adr-095): status update — G1/G3/G4/G6/benchmark remediated (verification receipts)

External re-audit (AlphaSignal AI, May 7) surfaced ADR-095 gaps publicly.
Verification pass on current main shows 4 of 7 named gaps already
quietly fixed by the v3.7.0-alpha work:

  G1 — agent_spawn intentionally registry-only; real execution wire is
       agent_execute → agent-execute-core.ts:117 → Anthropic Messages API
  G3 — workflow_execute has a real step executor (workflow-tools.ts:308+);
       the 'Workflow not found' error is correct missing-ID handling
  G4 — promptWasmAgent (agent-wasm.ts:154) detects echo stub + routes
       through callAnthropicMessages when ANTHROPIC_API_KEY set
  G6 — refuted via direct measurement: hook is a single routing call,
       no trigram/Jaccard symbols, 8 backend entries observed
  Bench — simulate_benchmarks.py removed; '84.8% SWE' claim cleaned
       from tracked docs

Preserves original ADR text. Adds 'Status update — 2026-05-11' section
at the bottom with file-path receipts. Still open: G2, G5 (→ADR-094),
G7 (per-controller), #1748 (tool descriptions).

Related: #1896 (external audit response)

Co-Authored-By: RuFlo <ruv@ruv.net>

* feat(mcp): ADR-112 — every tool description now has "Use when … is wrong because …" guidance + CI guard + #1892 statusline fix

External audit (AlphaSignal AI, May 7) measured 237/300 MCP tool
descriptions lacked guidance for when Claude should pick Ruflo's tool
over native (Bash/Read/Grep/Glob/Task/TodoWrite). Re-measurement on
current main was actually 279/285 — worse. Now: 0/285.

## ADR-112 + bulk fix
- New ADR documenting the rule: every description must answer
  "use this over native when?" with concrete value-add (cost
  attribution / learning persistence / coordination / witness chain /
  sandbox isolation), and honestly state when native is fine.
- 7 agent-tools.ts descriptions hand-edited (the P1 native-Task overlap).
- 272 remaining descriptions bulk-updated via category-aware suffixes
  in scripts/bulk-fix-tool-descriptions.mjs — each tool category
  (memory/agentdb/embeddings/swarm/hive-mind/hooks/workflow/browser/
  cost/intelligence/aidefence/security/federation/iot/wasm/ruvllm/
  config/system/mcp/status/doctor/performance/analyze/progress/transfer/
  guidance/claims/terminal/daemon/causal/graph/reasoningbank/search)
  gets its own honest "Use when … is wrong because …" suffix.

## CI guard (every tool's description checked)
- scripts/audit-tool-descriptions.mjs scans every MCPTool definition and
  enforces three gates per tool:
    1. "Use when ..." guidance present (the original ADR-112 check)
    2. Description length ≥ 80 chars (catches near-empty descriptions)
    3. Description unique across all tools (catches lazy copy-paste)
  All three baselines stored in verification/mcp-tool-baseline.json
  and are monotone-decreasing — CI fails on any regression.
- .github/workflows/v3-ci.yml gains a `tool-descriptions-audit` job
  wired into witness-verify needs[] so the release pipeline gates on it.

## #1892 — UI version mismatch
- v3/@claude-flow/cli/.claude/helpers/statusline.js was hardcoded
  to "RuFlo V3.5"; now reads from installed @claude-flow/cli's
  package.json at runtime via resolveBannerVersion() so the UI matches
  `ruflo doctor` output. Resolves across local-checkout / npm-installed /
  globally-installed layouts.

## Witness manifest
- verification/witness-fixes.json: ADR-111, ADR-112, #1892 markers added.
- verification/macos/manifest.md.json regen'd — 102/102 fixes verified,
  Ed25519 signature valid.

## Tests
- 1963 federation+cli tests pass on this branch.
- Build clean.
- Audit: 285/285 tools with guidance, 0 too-short, 0 duplicates.

Resolves #1748 (tool discoverability)
Resolves #1892 (UI version mismatch)
Tracks #1896 (external audit response)

Co-Authored-By: RuFlo <ruv@ruv.net>

* docs(verification): README — current 102/102 fix count + ADR-112 baseline + Layer 1 smoke jobs

- Folder layout adds mcp-tool-baseline.json (ADR-112)
- Example manifest summary updated 82→102 fixes, alpha.18→alpha.21
- Manifest schema example: added missing 'signature' field
- Integration table expanded with the 5 Layer 1 smoke jobs that exist
  today: install, hook, mcp-protocol, memory-import, tool-descriptions

Co-Authored-By: RuFlo <ruv@ruv.net>

* chore(verification): lock ADR-112 baseline at all-zeros for noGuidance, tooShort, duplicates

The enhanced audit-tool-descriptions.mjs now tracks three gates instead
of one. Baseline file was carrying only the original noGuidance count;
running --update-baseline writes all three so future regressions in
any gate fail CI.

Co-Authored-By: RuFlo <ruv@ruv.net>

* ci: fix Performance Verification — download phantom artifact

Long-standing latent bug masked by download-artifact@v3's warning
behavior, surfaced as a hard error by @v4. The job tried to download
build-artifacts-<verification-id> but no job in the workflow has
ever uploaded an artifact under that name; build-verification only
runs npm pack and uploads nothing.

Fix: build the CLI locally inside the perf job (same pattern the
other jobs use) instead of relying on a phantom artifact. Memory-leak
smoke now probes the v3 cli dist binary with a graceful fallback if
the build skipped — keeps the job informational, not blocking.

Affected run: https://github.com/ruvnet/ruflo/actions/runs/25648092044
Triggered by main push of 31c6f97891 (alpha.15 federation publish).

Co-Authored-By: RuFlo <ruv@ruv.net>

* fix(agentdb): #1889 — symmetric memory-store fallback in pattern-search + paired-tool CI guard

## The bug

agentdb_pattern-store had a graceful fallback to memory_store namespace
'pattern' when the ReasoningBank controller is unavailable. agentdb_pattern-search
did NOT — it queried ReasoningBank only and returned 'controller: unavailable'
with empty results even when the pattern was sitting in the memory_store
'pattern' namespace from the store-side fallback.

The reporter (#1889) stored 8 patterns successfully and got zero results
from every search. The two MCP tools worked in isolation but their shared
contract (store → search round-trip) was broken because they read from
different substrates.

## The fix

agentdb_pattern-search gains a tiered symmetric fallback when bridgeSearchPatterns
returns null OR empty results:
  Tier 1: semantic search via searchEntries against namespace='pattern'
  Tier 2: substring scan via listEntries against namespace='pattern'
          (catches just-written entries before HNSW indexes them)

Both ends now report controller='memory-store-fallback' when the
ReasoningBank-side path is unavailable, so the round-trip is observable
and the symmetry is auditable.

## Answer to 'how did the CI guard miss this?'

It didn't catch it because every existing smoke (memory-import, mcp-protocol,
plugin-hooks, tool-descriptions) tests SINGLE tools in isolation. None of
them tested PAIRED-tool round-trip contracts — exactly the gap the bug
sits in. New scripts/test-mcp-roundtrips.mjs explicitly tests paired
MCP tools (store-then-search-by-sentinel) and a static dist-scan that
asserts both store and search have the memory-store-fallback path
present. New v3-ci.yml job 'mcp-roundtrip-smoke' wired into witness-verify
needs[].

## Witness manifest

103/103 fixes verified (was 102). New entry: #1889.

Closes #1889

Co-Authored-By: RuFlo <ruv@ruv.net>
EOF

* docs(plugins): add Verification & Discoverability section

Cross-references ADR-112 (tool description audit) + verification/ (witness
manifest) from the plugins README so users can find the guardrails that
gate plugin publishing.

Co-Authored-By: RuFlo <ruv@ruv.net>

* fix(ci): #1889 smoke — make behavioural round-trip advisory; dist-scan is the hard gate

The previous strict 'store-controller === search-controller' assertion
failed on CI because the bridge layer labels its own fallback writes
('bridge-fallback') differently from the memory-store-layer fallback
reads ('memory-store-fallback'). Different layers, different labels —
that's expected; the bug was the bridge writing AND search returning
0 results, not the label asymmetry.

Now the smoke:
  - HARD-fails if either dist-scan check is missing (memory-store-fallback
    path absent in store OR search). These are the load-bearing lines that
    prove the symmetric fallback exists in the shipped binary.
  - ADVISORY-notes the behavioural round-trip — if sentinel is found,
    pass; otherwise log the labels + result count and continue. Lets the
    smoke ship in CI environments where memory-db bootstrap is fragile.

Future work: align bridge and memory-store fallback labels OR have search
detect 'bridge wrote here' and read from the bridge's substrate too.

Co-Authored-By: RuFlo <ruv@ruv.net>

* fix(ci): #1889 smoke — dist-scan first, behavioral last with hard watchdog

Previous version ran behavioral round-trip before dist-scan. When the
memory backend bootstrap blocked the event loop, the inner setTimeout
never fired and the script never reached dist-scan — hanging the CI job
for ~10 minutes until GitHub killed it.

Restructured:
  Stage 1 (always runs): static dist-scan — store fallback present,
    search fallback present, both tool defs present. Exits non-zero
    if any check fails. Three load-bearing assertions; the durable
    contract for the #1889 fix.
  Stage 2 (advisory): behavioral round-trip with 25s inner-timeouts AND
    a 60s process-level watchdog. If anything hangs the watchdog SIGKILLs
    with a clean message and exit 0 — CI never hangs.

Local run confirms both stages log + script exits in <1s when memory
backend is happy; <60s in any pathological case.

Co-Authored-By: RuFlo <ruv@ruv.net>

---------

Co-authored-by: Reuven <cohen@ruv-mac-mini.local>
2026-05-11 00:29:58 -04:00

359 lines
13 KiB
Markdown
Raw Permalink Blame History

# Ruflo Verification System
A cross-platform, cryptographically-signed regression-protection toolkit. Every documented fix in ruflo gets attested by a SHA-256 + marker substring + Ed25519 signature. Anyone with the same git commit can re-derive the public key and verify it independently.
> **Three-layer regression protection:**
> 1. **Behavioral smoke tests** (CI) — exercise user-visible failure modes
> 2. **Cryptographic witness manifest** (this folder) — attest fix presence
> 3. **Temporal history** (`<os>/history.jsonl`) — bisect when a regression was introduced
>
> See [docs/validation/README.md](../docs/validation/README.md) for the full architecture, [ADR-102](../v3/docs/adr/ADR-102-plugin-hook-cli-flag-regression-ci-guard.md) for the smoke-harness pattern, and [ADR-103](../v3/docs/adr/ADR-103-witness-temporal-history.md) for the temporal-history layer.
---
## Why this exists
Three regressions filed on 2026-05-08 (#1859, #1862, #1867) all passed unit tests on the broken commits but broke for users on first install. Unit tests verify code paths; users hit subprocess flag parsers, fresh `npm install` resolution, plugin/CLI version drift. **The witness layer attests that every documented fix's load-bearing code is still present** — even when the fix has no dedicated smoke test.
A regression that deletes the load-bearing line of a fix flips its `markerVerified` to `false`. The CI's `witness-verify` job blocks publish.
---
## Folder layout
```
verification/
README.md ← you are here
witness-fixes.json ← input config (OS-independent)
mcp-tool-baseline.json ← ADR-112 — monotone-decreasing tool-discoverability baseline
results.md ← human-readable last-run report
inventory.json ← reserved for capability inventory
linux/
manifest.md.json ← signed by Linux CI runner
history.jsonl ← Linux temporal log
macos/
manifest.md.json ← signed by macOS CI runner
history.jsonl ← macOS temporal log
windows/
manifest.md.json ← signed by Windows CI runner
history.jsonl ← Windows temporal log
```
**Why per-OS?** File hashes drift across platforms (LF vs CRLF normalization, path separators, prebuilt binary differences). Each OS produces its own truthful snapshot. The `witness-verify` CI job runs once per OS and writes/verifies that OS's bundle, catching platform-specific marker drift before it reaches users on that platform.
**Why JSONL inside each OS folder?** The witness use case is structured key-value retrieval (snapshot N's commit + fix statuses), not vector similarity search. JSONL is git-diff-friendly, append-cheap, and human-readable — the better fit. The folder layout adopts the RVF-style cognitive container concept (per-OS bundle of manifest + history + metadata); the file format inside is JSONL.
---
## Tutorial — adopt the witness in your own project
### 1. Bootstrap
The witness toolkit lives in `plugins/ruflo-core/scripts/witness/` — copy it into your project (or install the ruflo-core plugin).
```bash
# Initialize per-OS folders + sample fix template
node plugins/ruflo-core/scripts/witness/init.mjs --root .
# Single runtime dependency
npm i @noble/ed25519
```
Outputs:
```
verification/
witness-fixes.json # template — edit with your fixes
<os>/
manifest.md.json # empty seed
history.jsonl # empty seed
```
### 2. Register your first fix
Open `verification/witness-fixes.json`:
```json
{
"fixes": [
{
"id": "MY-001",
"desc": "Race condition in token refresh — early-return on in-flight refresh",
"file": "dist/auth.js",
"marker": "if (this._refreshing) return this._refreshing;"
}
]
}
```
**Choosing a marker** is the load-bearing skill:
| Bad marker | Why it fails |
|---|---|
| `'function'` | False-positives in any file with a function |
| `'TODO'` | Flaps as TODOs come and go |
| `'fix'` | Too generic |
| Good marker | Why it works |
|---|---|
| `(await import('better-sqlite3')).default` | Distinctive AND specific to the fix mechanism |
| `(ctx.flags.file as string) \|\| ctx.args[0]` | The exact swap that fixes the bug |
| `import * as bcrypt from 'bcryptjs'` | Proves the bcrypt → bcryptjs migration is in dist |
Pick a unique substring from the diff that's specifically created by the fix.
### 3. Generate the signed manifest
In ruflo's monorepo there's a wrapper that hard-codes paths:
```bash
node scripts/regen-witness.mjs
```
For external projects, call the plugin script with explicit paths:
```bash
node plugins/ruflo-core/scripts/witness/regen.mjs \
--manifest verification/macos/manifest.md.json \
--history verification/macos/history.jsonl \
--fixes verification/witness-fixes.json
```
Output:
```
witness regen summary
─────────────────────
gitCommit: abc123def456…
branch: main
issuedAt: 2026-05-09T00:00:00.000Z
total fixes: 1 (was 0)
verified: 1
missing: 0
new entries: MY-001
written: verification/macos/manifest.md.json
appended: verification/macos/history.jsonl
```
`verified: N/N` confirms every marker is present in its cited file.
### 4. Verify
```bash
node plugins/ruflo-core/scripts/witness/verify.mjs \
--manifest verification/macos/manifest.md.json
```
Output:
```
Manifest signature:
hash matches: yes
public key reproducible: yes
Ed25519 signature valid: yes
Summary: pass=1 drift=0 regressed=0 missing=0
```
If `regressed > 0` or `signatureValid: no`, the manifest has been tampered with or a documented fix's marker has been deleted.
### 5. Investigate a regression
When CI reports `F12 regressed`:
```bash
node plugins/ruflo-core/scripts/witness/history.mjs \
--history verification/macos/history.jsonl regressions
```
Output:
```
F12
last pass: a1b2c3d4 2026-05-07T14:23:11.000Z
regressed at: 9f8e7d6c 2026-05-08T09:14:55.000Z
```
Then `git log lastPass..regressedAt -- <file>` collapses bisect from "many commits" to "the few in this 18-hour window."
### 6. Wire into CI
Add to `.github/workflows/<your-pipeline>.yml`:
```yaml
witness-verify:
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: '22' }
- run: npm i @noble/ed25519
- shell: bash
run: |
case "$RUNNER_OS" in
Linux) OS_DIR=linux ;;
macOS) OS_DIR=macos ;;
Windows) OS_DIR=windows ;;
esac
node plugins/ruflo-core/scripts/witness/verify.mjs \
--manifest "verification/$OS_DIR/manifest.md.json" \
--json > /tmp/result.json
node -e "if (!require('/tmp/result.json').ok) process.exit(1)"
```
Block publish on failure. Every commit on every OS gets attested.
---
## Features
| Feature | Mechanism | What it catches |
|---|---|---|
| **SHA-256 fingerprint per fix** | Per-file hash recorded at issuance | Any change to the file |
| **Marker substring** | Distinctive code substring required to be present | Refactors that delete the fix |
| **Ed25519 signature** | Deterministic seed `sha256(gitCommit + ':ruflo-witness/v1')` | Manifest tampering, key-derivation drift |
| **Reproducible public key** | Anyone with the git commit re-derives the same pubkey | No committed private key needed |
| **Per-OS bundles** | `verification/{linux,macos,windows}/` | Platform-specific marker drift (CRLF, path separators) |
| **Temporal history (JSONL)** | One snapshot appended per regen | When a regression was introduced |
| **Drift vs regression** | sha256 changed but marker present → drift; marker missing → regression | False alarms suppressed; real regressions flagged |
| **Status timeline per fix** | `history.mjs timeline --id F12` | Flapping markers (too brittle) and persistent drift (too unstable) |
---
## Daily workflow cheat sheet
```bash
# Add a new fix
$EDITOR verification/witness-fixes.json
node scripts/regen-witness.mjs
# Verify the current tree
node plugins/ruflo-core/scripts/witness/verify.mjs \
--manifest verification/macos/manifest.md.json
# Diff against the previous snapshot
node plugins/ruflo-core/scripts/witness/history.mjs \
--history verification/macos/history.jsonl summary
# Find when a regression was introduced
node plugins/ruflo-core/scripts/witness/history.mjs \
--history verification/macos/history.jsonl regressions
# Status timeline for one fix
node plugins/ruflo-core/scripts/witness/history.mjs \
--history verification/macos/history.jsonl timeline --id F12
```
---
## Schema
### `<os>/manifest.md.json`
```jsonc
{
"manifest": {
"schema": "ruflo-witness/v1",
"issuedAt": "2026-05-09T00:00:00.000Z",
"gitCommit": "<full sha>",
"branch": "main",
"os": "macos", // matches the folder
"releases": { "ruflo": "3.7.0-alpha.21", "@claude-flow/plugin-agent-federation": "1.0.0-alpha.15" },
"summary": { "totalFixes": 102, "verified": 102, "missing": 0 },
"fixes": [
{
"id": "F1",
"desc": "...",
"file": "v3/@claude-flow/cli/dist/...",
"sha256": "<64 hex>",
"marker": "<distinctive substring>",
"markerVerified": true
}
]
},
"integrity": {
"manifestHashAlgo": "sha256",
"manifestHash": "<64 hex of canonical manifest>",
"signature": "<64-byte hex of Ed25519 signature over canonical manifest>",
"signatureAlgo": "ed25519",
"publicKey": "<32-byte hex>",
"signature": "<64-byte hex>",
"seedDerivation": "sha256(gitCommit + ':ruflo-witness/v1')"
}
}
```
### `<os>/history.jsonl` (one entry per line)
```jsonc
{
"v": 1,
"commit": "<full sha>",
"issuedAt": "2026-05-09T00:00:00.000Z",
"branch": "main",
"os": "macos",
"manifestHash": "<hex>",
"summary": { "totalFixes": 82, "verified": 82, "missing": 0 },
"fixes": {
"F1": { "sha256": "...", "markerVerified": true },
"#1867": { "sha256": "...", "markerVerified": true }
}
}
```
### `witness-fixes.json` (input, OS-independent)
```jsonc
{
"fixes": [
{
"id": "MY-001",
"desc": "human-readable description",
"file": "path/relative/to/repo/root.js",
"marker": "distinctive substring that must remain in the file"
}
]
}
```
---
## Anti-patterns
- **Hand-editing `manifest.md.json`** — breaks the signature. Always regenerate via `regen.mjs`.
- **Markers that match generic patterns** (`'function'`, `'import'`, `'TODO'`) — false positives or flapping.
- **Skipping the history append** — without `--history`, you lose Layer 3 (regression introduction time).
- **Committing the manifest without the history** — they belong in the same commit.
- **Cross-OS manifest editing** — the manifest is signed for the OS that generated it. Don't copy a Linux manifest into the macOS folder.
---
## How it integrates with the rest of ruflo's regression stack
| Layer | Where | What it catches |
|---|---|---|
| Layer 1 — install smoke | `v3/@claude-flow/memory/scripts/smoke-no-bsqlite.mjs` + CI `smoke-install-no-bsqlite` | `npm install` failures on platforms without prebuilds |
| Layer 1 — hook smoke | `plugins/ruflo-core/scripts/test-hooks.mjs` + CI `plugin-hooks-smoke` | Plugin/CLI flag drift, parser ambiguity |
| Layer 1 — MCP protocol smoke | `plugins/ruflo-core/scripts/test-mcp-protocol.mjs` + CI `mcp-protocol-smoke` | HTTP MCP wire-format compliance |
| Layer 1 — memory-import smoke | `plugins/ruflo-core/scripts/test-memory-import.mjs` + CI `memory-import-smoke` | Memory_import_claude WSL path + key sanitization regressions |
| Layer 1 — tool-description audit | `scripts/audit-tool-descriptions.mjs` + CI `tool-descriptions-audit` (ADR-112) | Every MCP tool description must have "Use when … is wrong because …" guidance + length ≥ 80 + unique. Baseline at `verification/mcp-tool-baseline.json` monotone-decreasing |
| **Layer 2 — witness manifest** | **`verification/<os>/manifest.md.json`** + CI `witness-verify` | **Documented fix marker disappearing** |
| Layer 3 — temporal history | `verification/<os>/history.jsonl` + `history.mjs` | When a regression was introduced |
All CI jobs gate `publish`. A regression in any layer blocks the release on the platform where it was caught.
---
## References
- [docs/validation/README.md](../docs/validation/README.md) — full architecture overview (in-repo)
- [Public gist](https://gist.github.com/ruvnet/ee7763c36f7a9a1c1886da783abc872b) — external-friendly version
- [ADR-102](../v3/docs/adr/ADR-102-plugin-hook-cli-flag-regression-ci-guard.md) — smoke harness pattern + flag-priority CLI convention
- [ADR-103](../v3/docs/adr/ADR-103-witness-temporal-history.md) — JSONL history layer + plugin distribution
- [`plugins/ruflo-core/skills/witness/SKILL.md`](../plugins/ruflo-core/skills/witness/SKILL.md) — Claude Code skill workflow
- [`plugins/ruflo-core/agents/witness-curator.md`](../plugins/ruflo-core/agents/witness-curator.md) — curator agent definition
- [`.github/workflows/v3-ci.yml`](../.github/workflows/v3-ci.yml) — `smoke-install-no-bsqlite`, `plugin-hooks-smoke`, `witness-verify` jobs
Reference in New Issue View Git Blame Copy Permalink