mirror of
https://github.com/ruvnet/ruflo.git
synced 2026-09-14 14:01:28 +08:00
docs/chatgpt-plugin-reference
411 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ruv
|
bc38e38108 |
fix(eject + mcp-tools-test): context.flags + oia_audit chain budget (iter 128)
Two related router-compat/eject-dryrun fixes. (1) eject — `--name is required` despite passing --name The eject action read from context.args via a local parseArgs, but the CLI parser already consumed declared flags into context.flags (NOT context.options or context.args). context.args was always empty in normal CLI invocation. Switched to read from context.flags first, fall back to parseArgs(context.args) for programmatic / test invocations where the parser wasn't used. Local: `node bin/cli.js eject --name my-test-harness --format json` now emits the expected dry-run plan with dryRun:true. (2) test-mcp-tools — `metaharness_oia_audit handler did not throw` oia_audit composes 5 sub-audits (oia-manifest + threat-model + mcp-scan + score + genome) in parallel — each shells out via npx. In CI cold-cache, the 60s budget was insufficient. Bumped its timeout to 90s (same bracket as drift_from_history) and added --dry-run input so the test doesn't pollute the audit namespace. Local: 117 passed, 0 failed (was 117/2 in CI cold-cache). Co-Authored-By: RuFlo <ruv@ruv.net> |
||
|
ruv
|
c4fc017f81 |
fix(smoke): 17z15 accepts test-mcp-tools SKIPPED-on-missing-dist (iter 126)
iter-125 added grep tolerance for ERR_MODULE_NOT_FOUND but test-mcp-tools.mjs self-detects missing dist BEFORE the dynamic import and emits this instead: # test-mcp-tools — SKIPPED Compiled dist not present: …/dist/src/mcp-tools/metaharness-tools.js Build the CLI first: ... Exit 0 — this script is meaningfully runnable only post-build. So the smoke's RTOUT in CI's score job contained "SKIPPED" / "Compiled dist not present", not ERR_MODULE_NOT_FOUND. Both grep targets added. Verified locally by hiding the dist file: greps match the actual SKIPPED-path stdout. Co-Authored-By: RuFlo <ruv@ruv.net> |
||
|
ruv
|
14c36a4083 |
fix(smoke): tolerate CI degraded paths in 17z28 + 17z15 (iter 125)
Two metaharness smoke contracts still tripped in PR #2405 CI even after iter-124's contract-grep fixes. Root cause for both: CI's score job runs the smoke against a fresh checkout WITHOUT building the CLI dist or seeding the metaharness-audit memory namespace, so two runtime checks landed in legitimate degraded paths the smoke didn't cover. (1) 17z28 — drift-from-history runtime fields `node drift-from-history.mjs --dry-run --format json` was expected to emit `timing.parallelWallMs` + `timing.parallelSpeedup` unconditionally. But when the metaharness-audit namespace has zero records (always true in fresh CI), drift-from-history correctly exits 2 with `{error, stderrTail}` — the documented no-history degraded path (iter 57 + iter 58). Skip the runtime field check on that path. (2) 17z15 — test-mcp-tools.mjs runtime envelope The runtime arm of this gate runs `node test-mcp-tools.mjs` which imports MCP tool handlers from dist/src/mcp-tools/. In CI's score job the dist isn't built (only the metaharness-real-data job builds it via `pnpm -F "@claude-flow/cli..." run build`), so the import fails with ERR_MODULE_NOT_FOUND. The structural greps higher in the step already lock the Phase 4 assertions on the source file; runtime confirmation continues to run in metaharness-real-data which DOES build the dist. Local: 120 passed, 0 failed (was 119/0, lost no ground). Co-Authored-By: RuFlo <ruv@ruv.net> |
||
|
ruv
|
0215e7b633 |
fix(metaharness): 5 smoke contract failures (iter 124)
5 metaharness smoke contracts had grown out of sync with the source.
Local + CI smoke now goes 119/0 (was 5/120 failing).
(1) 17i — audit-trend severity rank grep was stale
Old check: literal `SEVERITY_RANK = { clean: 0, low: 1, medium: 2, high: 3 }`
But iter 63 refactored to shared `import { SEVERITY_RANK } from './_harness.mjs'`.
Updated grep to accept either form.
(2) 17z — ADR-152 §3.1 oia-audit fingerprint grep was stale
Old check: `score = runOne(['score'` / `genome = runOne(['genome'`
But iter 56 parallelized to `runMetaharnessAsync(['score', path])`.
Updated grep to accept either form.
(3) 17z14 — roundtrip Stage 7 hardcoded the assertion count
Old check: `node test-pipeline-roundtrip.mjs | grep "31 passed, 0 failed"`
But stages have been added since iter-51; count is now 66.
Updated to gate on "≥ 31 passed AND 0 failed" — no failures + floor.
(4) 17z6 — test-mcp-tools.mjs metaharness_audit_list intermittent 30s timeout
The first MCP tool that shells out via npx pays the cold-cache warmup
(~25s observed); 30s was tight. Bumped default per-tool timeout 30s
→ 60s (chain-tools like drift_from_history stay at 90s).
(5) test-mcp-tools.mjs mcp_scan first-finding message string
Upstream metaharness@0.1.x JSON path emits {id, severity, title, detail};
our parseMcpScanText emits {severity, message}. The test asserted
`message` only. Normalized both paths in mcp-scan.mjs to a unified
shape: {severity, message, title?, detail?} where `message` falls
back to `title || detail || ''` when the upstream path doesn't
supply one. Downstream consumers (audit-trend fingerprint diff,
test-mcp-tools assertions) now don't care which parser ran.
Local smoke runs: was 115/5, now 119/0 (and counts will grow further
as stages get added — the assertion-count floor protects against
regressions, the literal exact-match did not).
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
36ef4922e7 |
fix(ruflo-core): commit .mcp.json + exempt plugin-shipped templates from gitignore
The ruflo-core smoke contract step 2 ("registers a 'ruflo' MCP server")
fails on CI because plugins/ruflo-core/.mcp.json is gitignored by the
two repo-level `.mcp.json` rules (lines 67 + 128). The file exists
locally for everyone who has run smoke, but CI does a fresh clone and
never sees it.
Plugin-shipped .mcp.json templates are part of the plugin's contract
(they declare which MCP server the plugin registers). They MUST be
tracked. User-specific .mcp.json at the repo root or other paths
remains ignored for security (no secret leakage).
Fix is two-part:
1. .gitignore — add `!plugins/*/.mcp.json` negation after both
`.mcp.json` rules so the plugin templates escape the ignore.
2. Force-add plugins/ruflo-core/.mcp.json (the only file currently
matching the new exemption). Contents are the standard registration:
{
"mcpServers": {
"ruflo": {
"command": "npx",
"args": ["-y", "@claude-flow/cli@latest"],
"env": { "CLAUDE_FLOW_MCP_TRANSPORT": "stdio" }
}
}
}
Smoke locally: 10/10 PASS (unchanged — file always existed locally).
Smoke under CI conditions (fresh clone): was 9/10, now 10/10 once
this commit lands.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
6ad8e4f364 |
fix(cost-tracker): add Windows shim ruflo-hook.cjs (#1902/#1903/#1904, #2132)
The Plugin package install-safety smoke (#1902/#1903/#1904) checks that every plugin marked _platform=posix in its hooks.json has a corresponding plugins/<name>/scripts/ruflo-hook.cjs sibling for Windows users. plugins/ruflo-cost-tracker/hooks/hooks.json declares "_platform": "posix" (its Stop hook is a /bin/bash one-liner) but had no Node shim. The audit-plugin-hooks-cross-platform.mjs gate fails with: plugins/ruflo-cost-tracker/hooks/hooks.json:0 POSIX-exempt but Windows shim missing cmd: Expected plugins/ruflo-cost-tracker/scripts/ruflo-hook.cjs This shim is the Windows-portable equivalent of the existing POSIX hook: spawns scripts/track.mjs with TRACK_QUIET=1, swallows output, times out at 30s, always exits 0 (telemetry is best-effort). When ruflo init runs on Windows, it writes a .claude/settings.json that points Stop at this file instead of bash — same behaviour, no /bin/bash dependency. Local audit: $ node scripts/audit-plugin-hooks-cross-platform.mjs ok: all POSIX-exempt files have a corresponding Windows Node shim Co-Authored-By: RuFlo <ruv@ruv.net> |
||
|
ruv
|
1de69daa95 |
test(metaharness): anchor oia-audit-weekly iter-108 hard-fail + iter-109 dispatch inputs (iter 116)
Two load-bearing invariants accumulated in the weekly cron workflow,
neither of which had a smoke anchor:
iter 108 — hard-fail when timing.path != "file"
The cron's --baseline-file flag is the load-bearing invariant for
cron-budget correctness. If it disappears, drift detection silently
falls back to slow path (~26s instead of ~1.4s), and the iter-7
weekly-cron budget exhausts on slower runners. Iter 108 added an
explicit `if [ "$PATH_LABEL" != "file" ]; then exit 1` block so
a regression produces a ::error:: annotation, not silent slowdown.
iter 109 — workflow_dispatch policy inputs
Parameterized `threshold` + `alert_on_new_severity` as
workflow_dispatch inputs so ad-hoc runs can explore stricter/looser
gates without editing YAML. The drift step references both via
`${{ inputs.threshold || '0.85' }}` and
`${{ inputs.alert_on_new_severity || 'high' }}`.
Both invariants are *removable* by a careless refactor; the YAML
still validates; the cron still runs; only the gate disappears.
That's exactly the failure mode smoke anchors are designed to catch.
WHAT SHIPPED — smoke step 17z79
Seven grep checks against .github/workflows/oia-audit-weekly.yml:
1. iter-108 fail-fast annotation string present
2. iter-108 PATH_LABEL != "file" check present
3. iter-109 workflow_dispatch: trigger present
4. iter-109 threshold: input declaration present
5. iter-109 alert_on_new_severity: input declaration present
6. iter-109 inputs.threshold wiring into drift step present
7. iter-109 inputs.alert_on_new_severity wiring present
Specific failure markers: "iter-108-hard-fail-removed",
"iter-109-threshold-input-removed", etc. — operator can see exactly
which invariant regressed.
CROSS-REFERENCE INTEGRITY MATRIX (post-iter-116)
| Category | Surfaces | Iters |
|------------------------|--------------------------------|-------------|
| Source → Target | 10 | 73-94, 106 |
| Target → Source | 1 | 102 |
| Internal consistency | 4 | 103-105, 107|
| ADR-150 invariants | 2 | 110, 111 |
| Workflow-job set | 1 | 115 |
| Workflow-step set | 1 (iter-108 + iter-109 anchors)| 116 | ◀ this iter
Nineteen smoke-gated surfaces.
GATES THIS ITER
smoke: 117 → 118 (step 17z79 green)
ci yml: 7 invariant strings × file-presence verified
PATTERN
Iter-115 anchored *which jobs* exist in a workflow. Iter-116 anchors
*which steps within a job* exist. Both are anti-deletion guards on
files that YAML-validate even after the gate is gone.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
b851f849a5 |
test(metaharness): metaharness-ci.yml contains all 6 expected jobs (iter 115)
metaharness-ci.yml accumulated 6 jobs across iters 1-48:
score (iter 1) — score harness against ruflo
mcp-scan (iter 1) — security finding scan
router-compat (iter 12) — @metaharness/router API tripwire
eject-dryrun (iter 4) — Phase-2 differentiator
similarity-tests (iter 40) — ADR-152 §3.1 contract
metaharness-real-data (iter 48) — load-bearing integration gate
A refactor that accidentally deletes a job header passes YAML
validation (the file is still valid YAML) but silently disables the
gate. The remaining jobs would still run, the CI status would be
green, and the deleted job's protection would just disappear.
WHAT SHIPPED — smoke step 17z78
For each of the 6 expected jobs:
- grep `^ jobname:$` (exact YAML job-header form)
- Specific failure: "missing-job-jobname"
Plus a count floor: ≥6 jobs expected (the count subtracts 1 for the
`push:` trigger block which also matches `^ ...:$`).
If a future iter:
- Drops a job header → "missing-job-eject-dryrun"
- Renames a job → "missing-job-old-name" + count stays same
- Adds a 7th job → all 6 pass, smoke still green (floor is
inclusive, not exact — additions OK)
The smoke gate is anti-deletion + anti-rename, not anti-addition.
That's the right asymmetry for a job-set assertion.
CROSS-REFERENCE INTEGRITY MATRIX (post-iter-115)
| Category | Surfaces | Iters |
|------------------------|--------------------------------|----------|
| Source → Target | 10 (description, SUBCOMMANDS, | 73-94, |
| | runScript, SKILL.md, MCP enum, | 106, 107 |
| | tool names + subcmd to CLAUDE, | |
| | CI tripwire refs, ADR refs) | |
| Target → Source | 1 (orphan-script gate) | 102 |
| Internal consistency | 4 (skill dir↔frontmatter, | 103-105, |
| | allowed-tools, desc+hint, | 107 |
| | ADR-151 scope) | |
| ADR-150 invariants | 2 (tilde pin, pin alignment) | 110, 111 |
| Workflow-job set | 1 (metaharness-ci jobs) | 115 | ◀ this iter
Eighteen smoke-gated surfaces.
GATES THIS ITER
smoke: 117 → 118 (step 17z78 green)
contract: 6 CI workflow jobs × file-presence verified
PATTERN
iter-115 is the workflow-layer equivalent of iter-89's SUBCOMMANDS-map
gate. Both verify that an enumerated set of identifiers (CLI subcommands
/ CI job names) all exist where they should.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
c712f4973a |
test(metaharness): bench JSON has results[] with numeric meanUs (iter 114)
iter-88 verified bench output JSON.parses. iter-114 goes deeper into
the schema: the bench JSON must have a non-empty `results[]` array
with `meanUs: number` and `label: string` per entry.
The bug-class iter-114 catches:
- Empty results array (e.g., wrong loop bounds): JSON.parse passes,
iter-82 GITHUB_STEP_SUMMARY table renders empty, iter-99 wall-budget
reads undefined meanUs
- Non-numeric meanUs (e.g., a bug stringifies it): downstream
consumers compute Number(meanUs) → NaN → broken comparisons
- Missing label: artifact analysis can't distinguish per-category
measurements
These are real failure modes that iter-88's parse-only gate misses.
WHAT SHIPPED — smoke step 17z77
For each of bench-similarity + bench-parse-mcp-scan:
- Run with --iters 500 --format json
- Parse + assert:
results is an array
results.length > 0
every entry has meanUs as number (int or float)
every entry has label as string
Specific failure form: ${bench}-empty / ${bench}-non-numeric-meanUs
/ ${bench}-not-array / ${bench}-non-string-label
GATES THIS ITER
smoke: 116 → 117 (step 17z77 green)
contract: 2 bench scripts × results[].meanUs + label shape verified
BENCH SCRIPT CONTRACT (post-iter-114)
| Gate | What it catches | Iter |
|-------------------------|------------------------------|------|
| `--format json` parses | output contamination | 88 |
| has generatedAt | timestamp emission | 112 |
| has results[] | array shape + numeric meanUs | 114 |
| --max-mean-us threshold | perf regression | 41+86|
| CI artifact upload | historical baseline tracking | 82+87|
Five gates per bench script. Future bench-scripts should adopt the
same shape contract.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
05b8e2f548 |
test(metaharness): oia-audit startedAt + finishedAt timestamp pair (iter 113)
Companion to iter-112's generatedAt gate. iter-112 documented oia-audit
as the multi-step exception: it emits startedAt + finishedAt instead
of a single generatedAt because it composes 5 subprocess calls and
the duration spans those calls.
iter-113 verifies the iter-7 / iter-56 contract — both timestamps
present and finishedAt >= startedAt.
WHAT SHIPPED — smoke step 17z76
Runs oia-audit --dry-run --format json and checks:
- startedAt is non-empty (≥20 chars ISO timestamp)
- finishedAt is non-empty
- finishedAt >= startedAt (sanity — composite never finishes before
it started; a clock-skew bug or a startedAt set AFTER the loop
would fail this check)
Failure form: "oia-audit-timestamp-${reason}" where reason is one of:
- missing-startedAt
- missing-finishedAt
- short-timestamps
- finished-before-started
iter-56's parallelization preserved the startedAt-set-before-runAllParallel
invariant; iter-113 anchors it at smoke-time.
CROSS-REFERENCE INTEGRITY MATRIX (post-iter-113)
iter-112 + iter-113 together codify the timestamp emission contract:
- Single-step scripts: generatedAt (iter 112)
- Multi-step composite (oia-audit): both (iter 113)
A future refactor that drops either field — or breaks the temporal
ordering — surfaces as a specific named failure.
GATES THIS ITER
smoke: 115 → 116 (step 17z76 green)
contract: oia-audit timestamp pair + ordering verified
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
38767da86f |
feat(metaharness): generatedAt timestamp on all --format json outputs (iter 112)
Pre-iter-112 audit of every --format json script revealed inconsistent
timestamp emission:
✓ score, threat-model, audit-list, audit-trend, similarity,
drift-from-history, bench-similarity, bench-parse-mcp-scan
✗ genome, mcp-scan (no generatedAt field)
⊘ oia-audit (uses startedAt/finishedAt — multi-step composite,
by design)
Downstream consumers (audit-trend's introduced/cleared diff, the
weekly cron's drift summary, ops scripts archiving artifacts) all
benefit from a canonical "when was this run" field. iter-112 fills
the two gaps.
WHAT SHIPPED
1. plugins/ruflo-metaharness/scripts/genome.mjs — payload now
includes `generatedAt: new Date().toISOString()`
2. plugins/ruflo-metaharness/scripts/mcp-scan.mjs — same field
appended to the --format json payload
3. plugins/ruflo-metaharness/scripts/smoke.sh step 17z75:
- Iterates 9 scripts (omits oia-audit which by design uses
startedAt/finishedAt) with minimal valid args
- For each: parses --format json output, asserts generatedAt
field exists with ≥20 chars (ISO timestamp format check)
- Failure form: "${name}-no-generatedAt"
DISCOVERY DURING IMPLEMENTATION
First-pass smoke step used `declare -A` (bash 4+ associative arrays).
macOS bash 3.2 doesn't support them — smoke threw "unbound variable"
immediately. Rewrote using the pipe-delimited list pattern from iter
88's same-style gate. Documented in the script-tests array shape.
The bash-portability bug-class is now showing up across multiple iters
(iter 94 BSD-sed `\s`, iter 112 declare -A). Future smoke contributors
should default to bash-3.2-compatible patterns.
GATES THIS ITER
smoke: 114 → 115 (step 17z75 green)
contract: 9 scripts × generatedAt ISO timestamp emitted
regression: none — purely additive field
CONSISTENT JSON CONTRACT (post-iter-112)
Every --format json output now emits:
{ ..., generatedAt: ISO-8601 timestamp }
oia-audit's startedAt/finishedAt remains the multi-step exception
(documented in iter-7 + the smoke-step exclusion comment).
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
82c8bacbc6 |
test(metaharness): pin versions consistent across 3 package.json files (iter 111)
iter-110 gated each pin's TILDE PREFIX. iter-111 gates the VERSION
ALIGNMENT across the 3 package.json files:
- package.json (root, dev install)
- ruflo/package.json (npm i ruflo)
- v3/@claude-flow/cli/package.json (npm i @claude-flow/cli)
Drift scenario:
- Someone updates ruflo/package.json to `metaharness: ~0.2.0`
- Forgets to update the other 2
- User installing ruflo gets metaharness 0.2.x
- CI tests against root's `~0.1.11` (still 0.1.x)
- Production behavior diverges from CI behavior — invisible until
production hits a metaharness 0.2.x-only bug
iter-111 enforces the pins match across all 3.
WHAT SHIPPED — smoke step 17z74
For each of metaharness, @metaharness/router, @metaharness/kernel:
- Read the optionalDependencies entry from all 3 package.json files
- Assert exactly 1 unique pin string (or 0, if package absent from all 3)
- Failure form: `metaharness-pin-drift:root=~0.1.11,ruflo=~0.2.0,cli=~0.1.11`
The failure message NAMES all 3 versions so the fixer immediately
sees which file is the outlier.
CROSS-REFERENCE INTEGRITY MATRIX (post-iter-111)
| Direction | Surface | Iter |
|--------------------------|-------------------------------|------|
| Source → Target (10 forward) |
| Target → Source (1 reverse) |
| Internal consistency (4) |
| ADR-150 invariants (2, iters 110 + 111) |
| package.json pins | tilde prefix | 110 |
| pin versions | consistent across 3 files | 111 | ◀ this iter
17 smoke-gated surfaces.
GATES THIS ITER
smoke: 113 → 114 (step 17z74 green)
contract: 3 metaharness packages × cross-file pin alignment verified
PIN-POLICY ARC (iters 110 + 111)
110 — pin SHAPE: must start with ~
111 — pin VALUE: must match across 3 package.json files
Together they encode the policy: "every metaharness pin uses tilde,
and the same tilde-pin appears wherever the package is referenced."
A drift in either shape OR value surfaces as a specific named smoke
failure.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
c1bc0c7408 |
test(metaharness): tilde-pin gate prevents caret regression (iter 110)
ADR-150's architectural-constraint review-round-1 mandated tilde pinning (`~0.1.11`) for @metaharness/* packages, not caret (`^0.1.11`). Rationale: upstream is unstable — iter-3 observed 5 releases in 2.7 hours. Tilde auto-absorbs PATCHES but stays on the same MINOR. Caret would auto-absorb minor bumps that might break ruflo's integration. Currently all 3 metaharness packages in package.json files use tilde: metaharness: ~0.1.11 @metaharness/router: ~0.3.2 @metaharness/kernel: ~0.1.0 A future contributor doing routine dep updates might run `npm install --save metaharness@latest` which writes caret pinning by default. That regression would be invisible at the package.json layer until upstream ships a 0.2.x release. WHAT SHIPPED — smoke step 17z73 For each of: - package.json (root) - ruflo/package.json - v3/@claude-flow/cli/package.json Iterates optionalDependencies and asserts every metaharness/@metaharness entry has a tilde-prefix version string. Caret, exact, or range-pinning all trip the gate. Failure form: "non-tilde-pin-in-ruflo:@metaharness/router=^0.3.2" Names BOTH the file AND the offending pin. CROSS-REFERENCE INTEGRITY MATRIX (post-iter-110) | Direction | Surface | Iter | |------------------------|-------------------------------|------| | Source → Target (10 forward, iters 73-94, 106, 107) | | Target → Source (1 reverse, iter 102) | | Internal consistency (4, iters 103-105, 107) | | ADR-150 invariants (1, iter 110) | | package.json pins | tilde-only for metaharness | 110 | ◀ this iter The "thing X must look like Y" pattern from iters 73-107 extends to the dependency-pinning policy. GATES THIS ITER smoke: 112 → 113 (step 17z73 green) contract: 3 package.json × 3 metaharness packages × tilde-prefix verified ADR-150 ARCHITECTURAL-CONSTRAINT ENFORCEMENT (post-iter-110) Constraint #1 (removable): iter 74 negative grep Constraint #2 (optional-only): iter 12 static-deps check Constraint #3 (graceful degrade): iter 19 + 55 drill Constraint #4 (CI gate): iter 7 no-metaharness-smoke.yml + Tilde pin (review-round-1): iter 110 ◀ this iter All four ADR-150 architectural constraints plus the review-mandated tilde pin are now smoke-gated. Co-Authored-By: RuFlo <ruv@ruv.net> |
||
|
ruv
|
51db9b381e |
ci(metaharness): weekly cron exposes policy thresholds via workflow_dispatch inputs (iter 109)
Pre-iter-109 the cron's policy thresholds (--threshold 0.85,
--alert-on-new-severity high) were hardcoded. To test stricter or
looser gates ad-hoc, maintainers had to:
1. Edit oia-audit-weekly.yml
2. Commit
3. Push
4. Wait for the workflow trigger
5. Revert
Iter 109 adds workflow_dispatch inputs so maintainers can test
different thresholds via the Actions UI's "Run workflow" form without
touching the YAML.
WHAT SHIPPED — oia-audit-weekly.yml
workflow_dispatch:
inputs:
threshold:
description: 'Structural similarity threshold (default 0.85 —
scheduled). Lower = stricter drift alert.'
type: string
default: '0.85'
alert_on_new_severity:
description: 'Alert on any introduced finding ≥ this severity'
type: choice
options: [info, low, medium, warn, high, error, critical]
default: high
Drift step uses the inputs via fallback expressions:
THRESHOLD="${{ inputs.threshold || '0.85' }}"
ALERT_SEV="${{ inputs.alert_on_new_severity || 'high' }}"
Scheduled runs (no manual input) get the hardcoded defaults
(0.85 / high) — same behavior as pre-iter-109.
USAGE EXAMPLES
- Standard weekly run (Sundays 04:17 UTC):
Uses 0.85 + high (defaults)
- Ad-hoc stricter check via Actions UI "Run workflow":
threshold: 0.95
alert_on_new_severity: medium
→ Alerts on smaller drift + any new MEDIUM-or-higher finding
- Permissive ad-hoc check (e.g., investigating a known-noisy week):
threshold: 0.50
alert_on_new_severity: critical
→ Only structural collapse (overall < 0.5) OR a critical finding fires
The enum for alert_on_new_severity matches iter-78's MCP enum
(iter-92's smoke step keeps these aligned).
SMOKE STEP 17z72 anchors:
- threshold + alert_on_new_severity inputs present
- Both have safe defaults (scheduled runs preserved)
- Drift step uses the inputs via $THRESHOLD / $ALERT_SEV vars
- Choice list contains all 7 severity values from iter-78
PLUS iter-79's smoke step 17z42 relaxed: now accepts either the
pre-iter-109 literal `--alert-on-new-severity high` OR the post-iter-109
parameterized form `--alert-on-new-severity "$ALERT_SEV"`. Same forward-
compat pattern as iter-60's relaxation in iter 84.
GATES THIS ITER
smoke: 111 → 112 (step 17z72 green; 17z42 grep relaxed)
yaml: python yaml.safe_load passes
cron: scheduled run behavior unchanged; ad-hoc tuning unlocked
OBSERVATION
Iters 78+79+108+109 are the policy-control arc:
78 — added --alert-on-new-severity flag at script + MCP
79 — wired it into cron (hardcoded 'high')
108 — hard-fail when fastpath label drifts away from 'file'
109 — parameterize the policy thresholds via dispatch inputs
The cron now has hardcoded SAFETY invariants (path === 'file') and
TUNABLE POLICY (threshold + alert severity). The split reflects what
each value's purpose is: safety = static, policy = dynamic.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
08757d2db1 |
ci(metaharness): weekly cron hard-fails on slow-path regression (iter 108)
iter-69 + iter-79 wired the weekly cron to use --baseline-file fastpath.
iter-97 surfaced the path in $GITHUB_STEP_SUMMARY (informational). iter-99
added a wall-budget gate to the metaharness-ci dispatcher check. But the
WEEKLY CRON itself had no hard fail on slow-path regression.
The drift scenario:
- Someone refactors oia-audit-weekly.yml and accidentally drops
`--baseline-file /tmp/prior/oia-audit.json` from the drift step
- drift-from-history falls through to slow path
- timing.path === 'slow', wall ~26s instead of expected ~1.4s
- Workflow STILL PASSES (just slower)
- Path: slow line appears in summary but no one notices for weeks
- Eventually the 10-min job timeout-minutes catches it, but only
after MANY weeks of slowness
iter-108 makes the regression fail immediately and specifically.
WHAT SHIPPED — oia-audit-weekly.yml drift step extension
After the existing summary lines:
PATH_LABEL=$(node -e "
const j = JSON.parse(require('fs').readFileSync('/tmp/drift-trend.json'));
console.log(j.timing?.path || 'unknown');
")
if [ "$PATH_LABEL" != "file" ]; then
echo "::error::Cron drift step expected timing.path='file' but got
'$PATH_LABEL' — the iter-67 fastpath flag may have regressed"
exit 1
fi
The GHA `::error::` annotation surfaces in the Actions UI as a red-X
annotation pointing at the failing line, not just a generic step
failure. Maintainers see "iter-67 fastpath flag may have regressed"
and know exactly where to look.
WHY HARD-FAIL HERE BUT NOT IN ITER-99
iter-99 gates `wall < 30000ms` because CI runners are slow and 30s
is the conservative-safe upper bound. iter-108 gates the SPECIFIC
fastpath label because the cron's --baseline-file flag is explicit
in the YAML — there's no reason for the path label to be anything
other than 'file' on a healthy run.
If the path is 'key' (someone changed to --baseline-key) or 'slow'
(no fast-path flag), iter-108 fails. iter-99's wall gate is the
generic backstop; iter-108's path gate is the specific contract.
SMOKE STEP 17z71 anchors:
- PATH_LABEL variable
- timing?.path extraction with 'unknown' fallback
- The != "file" conditional
- The error message referencing iter-67
- GHA ::error:: annotation form
GATES THIS ITER
smoke: 110 → 111 (step 17z71 green)
yaml: python yaml.safe_load passes
cron: drift step hard-fails on path-label regression
ITER 65 + 95 + 96 + 97 + 99 + 108 — FAST-PATH OBSERVABILITY + ENFORCEMENT
65 — timing.parallelSpeedup field (observe)
95 — timing.path derived field (observe)
96 — CLI table mode surfaces Path: (observe — human)
97 — Weekly cron summary surfaces Path: (observe — UI)
99 — CI dispatcher wall budget < 30s (enforce — generic)
108 — Weekly cron path === 'file' exact (enforce — specific) ◀
Observability is complete; enforcement is now layered: generic
wall-budget catches gross regressions, specific path-label catches
the exact iter-67 fastpath drop.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
6638530ba8 |
test(metaharness): ADR-152 refs resolve + ADR-151 scope-refs documented (iter 107)
iter-106 gated ADR-150's cross-references. iter-107 extends to ADR-152
(which only references existing ADRs) AND adds a structural assertion
to ADR-151 (which references the not-yet-created ADR-153-156 by design).
THE TWO ASYMMETRIC ADRs
ADR-152 (Genome Similarity Search):
- References ADR-150 (parent: Phase 1+2)
- References ADR-151 (parent: Phase-3 scope)
- All refs MUST exist → standard cross-reference gate
ADR-151 (Phase-3 scope shell):
- References ADR-150 (parent)
- References ADR-152 (existing child)
- References ADR-153, 154, 155, 156 (future children, scope-only)
- The future ADRs are INTENTIONALLY referenced before they exist;
ADR-151's design is to scope-lock them
- File-existence gate would fail here for the WRONG reason
WHAT SHIPPED — smoke step 17z70
- ADR-152: iter-106 style file-existence check on every ADR ref
- ADR-151: structural assertion that:
(a) explicitly contains "scope-only" or "scope only" marker
(b) explicitly references ADR-153, 154, 155, 156 (the 4 future ADRs)
If a future iter:
- Drops the scope-only marker → "adr151-no-scope-only-marker"
- Forgets to mention any of 153-156 → "adr151-missing-ADR-NNN"
The dual-check pattern lets each ADR be gated by the rule that
applies to it:
- "all refs exist" for ADRs with hard cross-refs
- "scope-refs named" for the scope-shell ADR
CROSS-REFERENCE INTEGRITY MATRIX (post-iter-107)
| Direction | Surface | Iter |
|------------------------|------------------------|------|
| Source → Target (10 forward) |
| ADR-150 refs | ADR files | 106 |
| ADR-152 refs | ADR files | 107 | ◀ this iter
| Target → Source (1 reverse, iter 102) |
| Internal consistency (4, iters 103-105 + 107) |
| ADR-151 scope marker | 153-156 references | 107 | ◀ this iter
Fourteen smoke-gated surfaces.
GATES THIS ITER
smoke: 109 → 110 (step 17z70 green)
contract: ADR-152's 3 refs verified + ADR-151's 4 scope markers anchored
OBSERVATION
Different ADRs have different reference semantics. Implementation ADRs
(150, 152) make hard cross-references. Scope-shell ADRs (151) make
forward references to future ADRs that don't exist yet. The smoke
gate distinguishes these via STRUCTURAL assertions specific to each
ADR's pattern, not a one-size-fits-all rule.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
f80977b566 |
test(metaharness): ADR-150 cross-references resolve (iter 106)
iter-91 gated SKILL.md → script-path references. iter-106 applies the
same pattern to the ADR layer: ADR-150 references ADR-151 (Phase-3
scope shell) and ADR-152 (Genome Similarity Search). If either gets
renamed/moved, ADR-150's links rot silently.
Drift scenario:
- Someone renames `ADR-151-harness-intelligence-layer.md` to
`ADR-151-phase-3-scope.md` (cleaner slug)
- ADR-150 still links to the old name
- Markdown renders but the link 404s
- Future maintainers reading ADR-150 hit a dead end finding the
Phase-3 scope
WHAT SHIPPED — smoke step 17z69
- Extract every `ADR-15[0-9]` reference from ADR-150's body
- For each ref, assert the matching file exists in v3/docs/adr/
using glob ADR-NNN-*.md (tolerates slug renames? No — the file
must literally be there with the ADR-NNN-...md naming).
- Floor count ≥ 3 (ADR-150 must reference itself + at least 151 + 152)
Current refs: ADR-150, ADR-151, ADR-152 (all resolve)
If a future iter:
- Renames ADR-151 file but not the ADR-150 references
→ "ADR-151-not-found"
- Adds an ADR-153 reference without creating the file
→ "ADR-153-not-found"
CROSS-REFERENCE INTEGRITY MATRIX (post-iter-106)
| Direction | Surface | Iter |
|------------------------|------------------------|------|
| Source → Target (9 forward) |
| description string | SUBCOMMANDS keys | 73 |
| SUBCOMMANDS values | scripts/*.mjs | 89 |
| MCP runScript() | scripts/*.mjs | 90 |
| SKILL.md refs | scripts/*.mjs | 91 |
| MCP enum | SEVERITY_RANK | 92 |
| MCP tool names | CLAUDE.md catalog | 93 |
| SUBCOMMANDS keys | CLAUDE.md catalog | 94 |
| CI workflow refs | tripwire scripts | 84 |
| ADR-150 refs | ADR-NNN files | 106 | ◀ this iter
| Target → Source (1 reverse, iter 102) |
| Internal consistency (3, iters 103-105) |
Thirteen smoke-gated cross-reference surfaces.
GATES THIS ITER
smoke: 108 → 109 (step 17z69 green)
contract: 3 ADR references × 3 file-existence checks pass
DOCUMENTATION-LAYER INTEGRITY
Source-code references already gated at multiple layers. iter-106
extends to the ADR layer — the documentation form of cross-reference.
Now both code references AND ADR references have smoke-time existence
gates.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
d36c902133 |
test(metaharness): SKILL.md description + argument-hint populated (iter 105)
Companion to iter-104's allowed-tools gate. Three frontmatter fields
matter for an agent invoking a skill:
name: skill ID (gated iter 103)
description: agent uses this to PICK the right skill
argument-hint: agent uses this to KNOW what args to pass
allowed-tools: Claude Code uses this to authorize tools (iter 104)
Without `description:` the skill is invisible in tool registries.
Without `argument-hint:` the agent guesses at args and fails.
WHAT SHIPPED — smoke step 17z68
For each skill:
- description: ≥ 20 chars (catches stub descriptions like "TODO" or "X")
- argument-hint: field MUST exist (empty quoted-string is OK for no-arg
skills, but the FIELD must be declared so the agent knows there's
no surprise required positional arg)
If a future iter:
- Drops description → "harness-foo-description-too-short:0"
- Writes a stub → "harness-foo-description-too-short:3" (e.g. 'TODO')
- Drops argument-hint → "harness-foo-no-argument-hint"
The 20-char minimum is arbitrary but conservative: real descriptions
in current skills are 100-400 chars. 20 catches genuine stubs without
forcing verbosity.
CROSS-REFERENCE INTEGRITY MATRIX (post-iter-105)
| Direction | Surface | Iter |
|------------------------|------------------------|------|
| Source → Target (8 forward) |
| Target → Source (1 reverse) |
| Internal consistency (3, iters 103-105) |
| skill dir | SKILL.md frontmatter name | 103 |
| SKILL.md | allowed-tools content | 104 |
| SKILL.md | description + argument-hint | 105 | ◀ this iter
Twelve smoke-gated surfaces.
GATES THIS ITER
smoke: 107 → 108 (step 17z68 green)
contract: 8 skills × description + argument-hint each verified
SKILL-FRONTMATTER 3-FIELD GATE COMPLETE
name ← iter 103
description ← iter 105
argument-hint ← iter 105
allowed-tools ← iter 104
All four Claude-Code-recognized frontmatter fields for skill
discovery+invocation are now smoke-gated.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
5ca3c857c5 |
test(metaharness): SKILL.md allowed-tools must be non-empty + known (iter 104)
Each SKILL.md frontmatter has an `allowed-tools:` field that Claude
Code reads to decide which tools an agent invoking the skill may
call. If the field is missing or empty:
- Agents can't run the skill (no tools authorized)
- The skill becomes dead documentation that LOOKS callable but
silently fails when invoked
Iter 87's broader skill-audit ran during init/upgrade flows. iter 104
codifies the assertion at smoke-time for the metaharness skills
specifically.
WHAT SHIPPED — smoke step 17z67
For each subdirectory of skills/:
- Assert `allowed-tools:` field exists in SKILL.md
- Assert the value after the colon is non-empty
- Assert each comma-separated tool name matches the known set:
Bash | Read | Write | Edit | MultiEdit | Glob | Grep
| Task | TaskCreate
Current state: all 8 metaharness skills use `allowed-tools: Bash`
(they each shell out to a script). The gate fires if a future iter:
- Drops the field → "harness-foo-no-allowed-tools-field"
- Empties it → "harness-foo-allowed-tools-empty"
- Adds an unknown tool → "harness-foo-unknown-tool:GodMode"
Floor count ≥6 protects against mass-delete of skills.
CROSS-REFERENCE INTEGRITY MATRIX (post-iter-104)
| Direction | Surface | Iter |
|------------------------|------------------------|------|
| Source → Target (8 forward) |
| Target → Source (1 reverse, iter 102) |
| Internal consistency (2, iters 103 + 104) |
| skill dir | SKILL.md frontmatter | 103 |
| SKILL.md | allowed-tools content | 104 | ◀ this iter
Eleven smoke-gated surfaces.
GATES THIS ITER
smoke: 106 → 107 (step 17z67 green)
contract: 8 skills × 1 known tool each verified
KNOWN-TOOL ENUM
The `KNOWN_TOOLS` regex is conservative: only the tools currently
used by metaharness skills + commonly-needed alternates. If a
future iter wants to add Task tool authorization (e.g., for a skill
that spawns sub-agents), this gate fires "unknown-tool:Task" and
forces the maintainer to deliberately extend the regex. That's the
intended friction.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
3c9eed9e6b |
test(metaharness): SKILL.md frontmatter name ↔ directory name (iter 103)
Skills are auto-discovered by Claude Code via the .claude/skills
pattern. Each directory under skills/ is a skill; tooling that maps
dir → skill-id assumes the SKILL.md's frontmatter `name:` field
matches the directory name.
Drift scenario:
- Directory: plugins/ruflo-metaharness/skills/harness-similarity/
- SKILL.md: `name: harness-foo` (renamed without changing dir)
- Result: agents listing skills by directory see "harness-similarity"
but tooling looking up by frontmatter sees "harness-foo".
Two truths, conflicting answers, no test catches it.
WHAT SHIPPED — smoke step 17z66
For each subdirectory of skills/:
- Extract `name:` frontmatter from SKILL.md
- Assert it equals the directory basename
Floor count ≥6 (current set is 8: harness-drift-from-history,
harness-genome, harness-mcp-scan, harness-mint, harness-oia-audit,
harness-score, harness-similarity, harness-threat-model). If a
mass-delete drops below 6, the floor catches it.
If a future iter renames the directory without updating the
frontmatter (or vice versa):
→ "harness-foo-name-mismatch:harness-bar"
The output names BOTH ends of the mismatch so the fixer knows which
to align.
CROSS-REFERENCE INTEGRITY MATRIX (post-iter-103)
| Direction | Surface | Iter |
|------------------------|------------------------|------|
| Source → Target (8 forward, iters 73-94) |
| Target → Source (1 reverse, iter 102: orphaned-script gate) |
| Internal consistency (NEW, iter 103) |
| skill dir | SKILL.md frontmatter | 103 | ◀ this iter
Ten cross-reference surfaces now smoke-gated.
GATES THIS ITER
smoke: 105 → 106 (step 17z66 green)
contract: 8 skill directories × 8 frontmatter-name matches verified
OBSERVATION
Iters 102 + 103 explore two new directions beyond forward references:
- 102: reverse (target lacks any source)
- 103: internal (one source contradicts itself across keys)
Both produce specific failure messages when drift occurs. The
"specific named failure" pattern is the canonical defense against
the drift-class of bug.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
9bf72af589 |
test(metaharness): orphaned-script detector — every .mjs referenced (iter 102)
Iters 89/90/91 gated cross-references from source (SUBCOMMANDS, MCP
handlers, SKILL.md) → target (scripts/*.mjs). Iter 102 gates the
REVERSE direction: every .mjs file in scripts/ must be referenced by
at least one consumer.
The bug-class this catches:
- Someone adds scripts/foo.mjs as an experimental utility
- Never wires it into SUBCOMMANDS or runScript()
- Eventually forgets about it; the file lingers as dead code
- 6 months later a maintainer doing audit asks "what's this for?"
Negative result: the file rots in place, growing stale assumptions
about ruflo's surface.
WHAT SHIPPED — smoke step 17z65
For each .mjs in scripts/:
- Underscore-prefix files (_harness, _similarity, _spike-similarity):
accept either (a) imported by another script OR (b) referenced
by smoke.sh / CI workflows as standalone invocation
- Other files: must appear in SUBCOMMANDS map (CLI), runScript()
call (MCP), SKILL.md inline ref, smoke.sh test target, OR CI
workflow step
Counts ≥20 scripts (sanity floor — script set has been stable around
23 for many iters; below 20 means something was deleted en masse).
DISCOVERY DURING IMPLEMENTATION
First pass had a stricter underscore rule: "must be imported by
another script". _spike-similarity.mjs failed — it's a STANDALONE
regression anchor (iter 35) invoked directly via `node scripts/
_spike-similarity.mjs` from smoke + CI, never imported.
Relaxed the underscore rule to accept either import OR direct-
invocation reference. Documents the convention that underscore
prefix means "internal/utility" but doesn't mandate import-only
usage.
CROSS-REFERENCE INTEGRITY MATRIX (post-iter-102)
| Direction | Surface | Iter |
|------------------------|------------------------|------|
| Source → Target (8 surfaces, iters 73-94) |
| description string | SUBCOMMANDS keys | 73 |
| SUBCOMMANDS values | scripts/*.mjs | 89 |
| runScript() | scripts/*.mjs | 90 |
| SKILL.md | scripts/*.mjs | 91 |
| MCP enum | SEVERITY_RANK | 92 |
| MCP tool names | CLAUDE.md catalog | 93 |
| SUBCOMMANDS keys | CLAUDE.md catalog | 94 |
| CI workflow refs | tripwire scripts | 84 |
| Target → Source (REVERSE, iter 102) |
| scripts/*.mjs | at least one consumer | 102 | ◀ this iter
Nine cross-reference surfaces now smoke-gated. A script that gets
added without being wired up surfaces as `orphaned-foo.mjs` in smoke.
GATES THIS ITER
smoke: 104 → 105 (step 17z65 green)
contract: 23 scripts × at least 1 reference each verified
the floor (≥20) tolerates intentional pruning while catching
accidental mass-delete
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
196b06496e |
test(metaharness): JSON-output gate covers all 10 scripts (iter 101)
Iter 88's family-wide JSON-output contract gate covered 8 scripts.
Two were deliberately omitted because they need multi-file args:
audit-trend (--baseline + --current) and similarity (--a + --b).
Iter 88's commit comment said "covered by roundtrip Stages 1-12
instead" — true for happy path, but the JSON-PARSE contract is a
distinct invariant that the roundtrip doesn't directly verify (it
parses the output but via a regex-extract, not strict parse-of-file).
Iter 101 fills the gap.
WHAT SHIPPED — smoke step 17z51 extension + new step 17z64
- Two new synthetic fixtures in the tmpdir:
/tmp/.../fixA.json — similarity input shape (score + genome)
/tmp/.../fixB.json — audit-trend input shape (composite +
components + fingerprint)
- SCRIPT_TESTS array extended:
similarity --a fixA.json --b fixA.json --format json
audit-trend --baseline fixB.json --current fixB.json --format json
- Both produce JSON.parse-able output
Smoke step 17z64 anchors:
- similarity + audit-trend fixture invocations in SCRIPT_TESTS
- The fixture-emit heredocs (TMP/fixA.json + TMP/fixB.json)
- Fixture shape requirements (agent_topology, composite.worst)
COVERAGE MATRIX (post-iter-101)
| Script | Iter 88 | Iter 101 |
|---------------------|:-------:|:--------:|
| oia-audit | ✓ | ✓ |
| score | ✓ | ✓ |
| genome | ✓ | ✓ |
| mcp-scan | ✓ | ✓ |
| threat-model | ✓ | ✓ |
| audit-list | ✓ | ✓ |
| bench-similarity | ✓ | ✓ |
| bench-parse-mcp-scan| ✓ | ✓ |
| similarity | | ✓ | ◀ NEW
| audit-trend | | ✓ | ◀ NEW
All 10 JSON-emitting scripts now contract-verified by parse-roundtrip.
The two remaining scripts NOT covered (drift-from-history, mint) are
intentionally excluded:
- drift-from-history: slow (~25s on real audit-list) — covered by
Stage 8 (iter 68) instead
- mint: filesystem-write protection — covered by graceful-
degradation drill (iter 19)
GATES THIS ITER
smoke: 103 → 104 (step 17z64 green)
contract: 10 scripts × JSON parse-roundtrip pass
The family-wide JSON-output contract is now exhaustively gated for
all scripts that downstream code (CI, ops, agents) consumes via JSON
parsing.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
3b3018b48a |
docs(adr): 100-iter milestone — ADR-150 reflects iters 83-99 (iter 100)
The symbolic 100th iteration of /loop on ADR-150's feat/metaharness-integration-research branch. This iter brings the canonical engineering record current through iter 99 and adds a retrospective section. WHAT'S NEW IN THE ADR 1. Status header: "Phase 3 §3.1 ✅ iters 33-82" → "iters 33-99" 2. Date marker: "82 iterations" → "**100 iterations of /loop**" 3. New section "Iters 83-99 — cross-reference integrity + observability" with three sub-arcs: - Cross-reference integrity matrix (iters 73, 84, 89-94) Eight surfaces where source A references source B by name. Each smoke-gated; specific failure message when references drift. - Fast-path observability arc (iters 95-99) The iter-66/67 fast-paths (~14x and ~19x speedups) are now observable at every consumption layer — JSON, CLI table, CI summary, CI dispatcher round-trip (shape + wall budget). - Anti-regression pattern reinforcement (iters 83-92) Positive-presence guards, perf characterization, JSON-output contract gate, MCP enum alignment. 4. Fleet status (post-iter-99): - Smoke step count: 85 → 102 - MCP runtime: 117 → 120 assertions - 8 cross-reference integrity surfaces - 3 negative-guard invariants - 3 compat tripwires - 2 bench scripts in CI with artifact tracking - 4 CI workflows - ~290 total ruflo-metaharness smoke assertions 5. 100-iter retrospective — names two reusable patterns: (a) BUG-DISCOVERY VIA END-TO-END TESTS Iter-47 schema bug hid for 9 iters. Iter-50 dead code hid for 28. Iter-87 JSON contamination hid for 5. All surfaced only when a test exercised the REAL chain — not when smoke greps verified source markers. Lesson: smoke-grep on source is a COMPLEMENT to, not a substitute for, real-output verification. (b) CROSS-REFERENCE INTEGRITY When source A references source B by name, drift is the default without an explicit gate. Eight surfaces are now gated; each produces a specific named failure when the reference rots. SMOKE STEP 17z63 anchors: - "iters 33-99" status marker - "100 iterations of /loop" date marker - "Iters 83-99 — cross-reference integrity" section header - Cross-reference integrity matrix table - Fast-path observability arc header - 100-iter retrospective section header - Fleet status post-iter-99 marker PLUS iter-60 + iter-83 smoke steps relaxed for forward-compat: - "Phase 3 §3.1 ✅ iters 33-N" with N as any number - "<N> iterations" with N as any digit-string or word-form Future ADR refreshes won't break these gates. GATES THIS ITER smoke: 102 → 103 (step 17z63 green; 17z23 + 17z46 grep relaxed) fleet: 33/33 (doc-only iter) no code changes — pure documentation milestone THE 100-ITER ARC Phase 1 (iters 1-3) — MVP shipped Phase 2 (iters 4-32) — expansion, eject, oia-audit composite Phase 3 §3.1 (iters 33-99) — Genome Similarity Search: - 35-41: spike + production module + CLI skill + MCP tool + CI - 42-44: bug-hardening arc (dispatcher flags, wrapper semantic) - 45-52: doctor + MCP runtime + roundtrip + tripwire scaffolding - 53-67: drift-from-history primitive + 3-tier fastpath ladder - 68-79: roundtrip stages 8-12 + alert-on-new-severity + cron auto - 80-99: compat tripwire family + cross-reference integrity matrix + observability arc + dispatcher CI cross-checks ADR-151 Phase 3 §3.2-§3.5 remain scope-locked. KRR retrain still blocked on production data. Two upstream issues filed at ruvnet/agent-harness-generator (#15, #16); both still OPEN as of iter 99. Co-Authored-By: RuFlo <ruv@ruv.net> |
||
|
ruv
|
16150fd116 |
ci(metaharness): dispatcher round-trip enforces wall budget (iter 99)
iter-98 added a CLI-dispatcher round-trip for drift-from-history but
only asserted JSON shape. iter-67 promised ~19x speedup via
--baseline-file; iter-98 didn't verify that promise actually holds
in CI. A regression that reverts the fast-path to slow would still
produce the correct JSON shape (path field, skippedAuditList) but
take ~26s instead of ~1.4s.
WHAT SHIPPED — metaharness-ci.yml iter-98 step extended
After the iter-95 path + iter-66 skippedAuditList assertions:
WALL=$(node -e "
const j = JSON.parse(\`\$OUT\`);
console.log(j.timing?.parallelWallMs ?? 0);
")
if [ "$WALL" -gt 30000 ]; then
echo "FAIL: dispatcher fast-path wall ${WALL}ms > 30000ms (regression?)"
exit 1
fi
Budget rationale: iter-67 measured 1.4s on Apple Silicon. CI runners
are ~3-5x slower. 30s budget gives ~5x headroom from the local
baseline while still catching a 10x regression (slow path's 26s
fits inside the budget — so this catches the regression to slow path
ONLY when CI is already slow, but most CI runs are <30s for the
fast path).
A stricter budget (5s, 10s) would risk false positives on slow CI
runners. 30s is the lowest budget that's reliably safe across
GitHub-Actions runner classes.
The pass-message now includes the actual measured wall:
✓ drift-from-history dispatcher round-trip green (fast-path via CLI; wall 1342ms)
So even when the budget passes, the Actions UI shows what was measured —
ops users can spot creeping regressions (e.g., wall trending from 1.4s
to 5s over weeks) before they breach the 30s ceiling.
SMOKE STEP 17z62 anchors:
- Wall-budget failure message literal
- parallelWallMs extraction in node -e
- WALL variable assignment
GATES THIS ITER
smoke: 101 → 102 (step 17z62 green)
yaml: python yaml.safe_load passes
ci: iter-98 step now also gates wall < 30s
DISPATCHER-LAYER ARC + PERF (iters 42-66-67-95-97-98-99)
42 — dispatcher flag-drop fix
66 — --baseline-key fast-path (~14x)
67 — --baseline-file fast-path (~19x)
95 — timing.path field
97 — weekly cron summary surfaces path
98 — CI dispatcher JSON shape contract
99 — CI dispatcher wall budget ◀ this iter
The dispatcher path is now contract-anchored at BOTH the shape level
(iter 98) AND the performance level (iter 99). A regression in either
fails CI with a specific named error.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
fccee28486 |
ci(metaharness): drift-from-history dispatcher CI cross-check (iter 98)
The metaharness-real-data job tests the score subcommand through the
CLI dispatcher (iter-47 pattern). But the drift-from-history dispatcher
path was never CI-exercised. The iter-66/67 fast-paths + iter-95
timing.path field rely on:
- iter-42's dispatcher flag-roundtrip
- drift-from-history's synthetic listResult logic
- audit-trend's --baseline file dispatch
A regression in ANY of those would survive smoke (which tests at the
script layer) but fail real ops use (which goes through the CLI).
Iter 98 adds the missing dispatcher round-trip.
WHAT SHIPPED — metaharness-ci.yml, metaharness-real-data job
New step at job end:
1. Generate fresh audit as baseline-file input
2. Invoke through CLI: `node bin/cli.js metaharness drift-from-history
--baseline-file /tmp/drift-baseline.json --dry-run --format json`
3. Assert output contains `"path": "file"` (iter-95 + iter-67 contract)
4. Assert output contains `"skippedAuditList": true` (iter-66 contract)
VERIFIED LOCALLY
$ node bin/cli.js metaharness drift-from-history --baseline-file /tmp/baseline.json --dry-run --format json
"skippedAuditList": true,
"path": "file"
"verdict": "unchanged"
WHAT THIS CATCHES THAT EXISTING TESTS DON'T
smoke 17z23+ tests test-pipeline-roundtrip.mjs at the SCRIPT layer
smoke 17z61 tests the SAME path through the CLI DISPATCHER
A regression in:
- iter-42 dispatcher flag-drop (silent for 6 iters before iter 42 found it)
- iter-67 baseline-file branch (synthetic listResult drift)
- iter-95 timing.path derivation
would now surface in the metaharness-real-data job as
"FAIL: dispatcher did not propagate baseline-file or timing.path missing".
PARALLEL TO ITER 47'S CROSS-CHECK PATTERN
iter 47 — score dispatcher round-trip:
$ node bin/cli.js metaharness score --path . | grep harnessFit
iter 98 — drift-from-history dispatcher round-trip:
$ node bin/cli.js metaharness drift-from-history --baseline-file X
| grep '"path": "file"'
Both verify a specific CLI surface produces specific JSON keys.
GATES THIS ITER
smoke: 100 → 101 (step 17z61 green)
ci: new dispatcher round-trip step in metaharness-real-data job
yaml: python yaml.safe_load passes
runtime: dry-run produces all 3 expected JSON markers locally
ITER 42-66-67-95-97-98 DISPATCHER-LAYER ARC
42 — dispatcher flag-drop fix
66 — --baseline-key fast-path
67 — --baseline-file fast-path
95 — timing.path derived field
97 — weekly cron summary surfaces path
98 — CI dispatcher round-trip locked at job-level ◀ this iter
The dispatcher → drift-from-history path is now contract-anchored at
the CI level, not just the smoke level.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
de8b3e44e8 |
ci(metaharness): weekly cron summary surfaces timing.path (iter 97)
iter 96 surfaced timing.path in the drift-from-history CLI's
human-readable table mode. The same field is also useful in the
weekly cron's GITHUB_STEP_SUMMARY — it confirms which fastpath the
cron actually used.
Why this matters operationally:
- iter 79 wired the cron to use --baseline-file fastpath
- Expected wall: ~1.4s (iter 67's measured baseline)
- If a future bug accidentally falls through to the slow path,
wall would balloon to ~26s but the WORKFLOW WOULD STILL PASS
(just slower). The summary's Path: line makes the regression
visible at the Actions-UI level.
WHAT SHIPPED — .github/workflows/oia-audit-weekly.yml
Drift step's summary block extended:
## Drift vs prior week: `near-identical` (similarity=1)
Alert status: `OK`
Path: `file (wall 1342ms)` ◀ NEW (iter 97)
The PATH_TAKEN extraction reads:
- timing.path ('file' / 'key' / 'slow' / 'unknown' fallback)
- timing.parallelWallMs ('?' fallback if missing)
If a future drift-from-history version drops these fields, the
summary still renders cleanly with 'unknown' / '?'.
SMOKE STEP 17z60 anchors:
- PATH_TAKEN variable definition
- t.path || 'unknown' fallback
- t.parallelWallMs extraction
- echo "Path:" line written to $GITHUB_STEP_SUMMARY
GATES THIS ITER
smoke: 99 → 100 (step 17z60 green; 100-step milestone)
yaml: python yaml.safe_load passes
cron: summary now includes path observability
ITER 65 + 66 + 67 + 79 + 95 + 96 + 97 — FAST-PATH OBSERVABILITY ARC
65 — timing.parallelSpeedup field
66 — --baseline-key flag (~14x speedup)
67 — --baseline-file flag (~19x speedup)
79 — weekly cron uses --baseline-file
95 — derived timing.path field
96 — CLI table mode surfaces Path:
97 — weekly cron summary surfaces Path: ◀ this iter
The fast-path is now observable at every consumption layer: JSON,
table mode, and the CI Actions summary.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
1e856ab967 |
feat(metaharness): drift-from-history table surfaces path + wall (iter 96)
iter 95 added timing.path to the JSON payload. The human-readable table mode (--format != json) didn't show it — users running drift detection interactively had no signal whether the fast-path activated. WHAT SHIPPED — drift-from-history.mjs table mode New line between "Current:" and the Structural-similarity block: # drift-from-history Baseline: file:/tmp/baseline.json Current: 2026-06-17T04:06:50.180Z Path: file (wall 659ms) ◀ NEW (iter 96) Structural similarity: 1 (near-identical) Distance: 0 ✓ similarity ≥ 0.95 — OK The wall-clock annotation makes the speedup visible at a glance: - Path: file (wall 659ms) — iter-67 fastest path - Path: key (wall ~2s) — iter-66 audit-list-skip - Path: slow (wall ~26s) — default, audit-list ran A user running `drift-from-history` regularly sees their actual performance instead of having to JSON-parse the timing field. GATES THIS ITER smoke: 98 → 99 (step 17z59 green) regression: roundtrip 66/66 (no behavior change on default JSON path) ITER 95 + 96 OBSERVABILITY ARC 95 — timing.path derived field in JSON payload 96 — timing.path + wall surfaced in human table mode The fast-path observability story is now complete at both the machine-readable (iter 65 + 95) and human-readable (iter 96) layers. Co-Authored-By: RuFlo <ruv@ruv.net> |
||
|
ruv
|
b87d2b58f4 |
feat(metaharness): drift-from-history exposes derived timing.path label (iter 95)
Pre-iter-95 consumers had to combine two booleans to know which path
ran:
- skippedAuditList: bool (iter 66)
- usedBaselineFile: bool (iter 67)
- Implied path:
usedBaselineFile=true → 'file'
skippedAuditList=true, file=false → 'key'
both false → 'slow'
The 3-way derived label was inferable but every consumer (CI workflow,
agent code, ops scripts) had to compute it locally. Now they read
`timing.path` directly.
WHAT SHIPPED
drift-from-history.mjs — timing payload extended:
timing.path: 'slow' | 'key' | 'file'
derived: usedBaselineFile ? 'file' : (skippedAuditList ? 'key' : 'slow')
The two underlying booleans STAY in the payload (don't break iter-65/66/67
consumers). The new field is purely additive — a consumer-ergonomics
improvement.
VERIFIED VIA SMOKE STEP 17z58
- --baseline-file → timing.path === 'file' ✓
- --baseline-key → timing.path === 'key' (not gated at smoke level
because key resolution requires memory access; covered by Stage 9
+ manual verification at iter-95 commit time)
- default (slow) → timing.path === 'slow' (implicit by the truth
table; if either fast-path were active, file/key would be set)
GATES THIS ITER
smoke: 97 → 98 (step 17z58 green)
regression: roundtrip 66/66 (no behavior change on default path);
Stage 8 + 9 see timing.path field but don't assert it
(consumer-ergonomic, not contract-required)
CONSUMER VALUE
Before:
if (data.timing.usedBaselineFile) { ... }
else if (data.timing.skippedAuditList) { ... }
else { ... }
After:
switch (data.timing.path) {
case 'file': ...
case 'key': ...
case 'slow': ...
}
The booleans stay for backward compat. New consumers SHOULD use
timing.path. The truth-table comment in the source documents the
mapping.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
3bcebcfb39 |
test(metaharness): every CLI subcommand documented in CLAUDE.md (iter 94)
Companion to iter-93's MCP-tool documentation gate. CLAUDE.md also
serves as a CLI catalog — each subcommand from metaharness.ts's
SUBCOMMANDS map should appear as `npx ruflo metaharness <X>` so users
can discover it via the docs.
Pre-iter-94 coverage: only iter-40 verified `similarity` appears in
CLAUDE.md. The other 9 subcommands had no documentation gate. If a
future iter added a `metaharness compare` subcommand but forgot to
update CLAUDE.md, the CLI tool would exist but be undiscoverable.
WHAT SHIPPED — smoke step 17z57
- Extract SUBCOMMANDS keys from metaharness.ts (handles quoted +
unquoted forms: 'mcp-scan' vs score)
- For each key, grep `npx ruflo metaharness <key>` in CLAUDE.md
- Lock count at 10
BSD-sed portability fix:
- Pre-iter-94 used `\s` in sed expressions (GNU-only)
- Macos BSD sed treats `\s` literally → substitution silently fails
- Replaced with `[ \t]` character class throughout
DISCOVERY DURING IMPLEMENTATION
First-pass smoke regex used `\s` in BSD sed and silently produced no
substitution. The smoke step then "passed" by accumulating ALL 20
lines (10 keys + 10 values) as "subcommand keys" — each then failed
the CLAUDE.md grep with names like `'audit-list':-not-in-claude-md`.
The output exposed the bug. Without the iter-94 mismatch, the iter-89
smoke step's similar regex would have silently passed (its sed didn't
use `\s` in the substitution, only in the grep).
Lesson: BSD/GNU portability is a real concern even when "the test
passes" — silent substitution failures produce confused-looking diff
output that takes a beat to recognize.
CROSS-REFERENCE INTEGRITY MATRIX (post-iter-94)
| Source | Target | Iter |
|------------------------------|----------------------|------|
| description string | SUBCOMMANDS keys | 73 |
| SUBCOMMANDS values | scripts/*.mjs | 89 |
| MCP handler runScript() | scripts/*.mjs | 90 |
| SKILL.md inline refs | scripts/*.mjs | 91 |
| MCP enum | SEVERITY_RANK keys | 92 |
| MCP tool names | CLAUDE.md catalog | 93 |
| SUBCOMMANDS keys | CLAUDE.md catalog | 94 | ◀ this iter
| CI workflow run commands | tripwire scripts | 84 |
Eight cross-reference surfaces now smoke-gated.
GATES THIS ITER
smoke: 96 → 97 (step 17z57 green; portability fix surfaced)
contract: 10 CLI subcommands × 10 CLAUDE.md catalog entries verified
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
f79ba66f4c |
test(metaharness): every MCP tool documented in CLAUDE.md (iter 93)
CLAUDE.md is the agent-facing tool catalog. When agents are deciding
which MCP tool to call for a task, they read CLAUDE.md's listing. If
a future iter adds a tool to metaharness-tools.ts but forgets to
update CLAUDE.md, the tool exists in the registry but is INVISIBLE
to agents browsing the documentation.
The bug-class:
- metaharness-tools.ts has `name: 'metaharness_NEW_TOOL'`
- test-mcp-tools.mjs registers it and asserts the contract
- smoke 17z (MCP runtime) passes
- But CLAUDE.md still lists 9 tools, not 10
- Agents reading CLAUDE.md don't know to call metaharness_NEW_TOOL
- The tool is dead documentation
Iter 93 gates against this.
WHAT SHIPPED — smoke step 17z56
- Extract every `name: 'metaharness_X'` from metaharness-tools.ts
- For each, grep `mcp__claude-flow__metaharness_X` in CLAUDE.md
(the agent-facing tool-name format)
- Lock count at 9 (mint deliberately excluded per iter-73 §Sandboxing)
If a future iter:
- Adds a 10th MCP tool without CLAUDE.md update → "tool-X-not-in-claude-md"
- Removes a tool without CLAUDE.md update → mismatch
- Lists mint in CLAUDE.md as mcp__... → enforced via iter-73's anti-mint guard
CROSS-REFERENCE INTEGRITY MATRIX (post-iter-93)
| Source | Target | Iter |
|------------------------------|-----------------------|------|
| description string | SUBCOMMANDS keys | 73 |
| SUBCOMMANDS values | scripts/*.mjs | 89 |
| MCP handler runScript() | scripts/*.mjs | 90 |
| SKILL.md inline refs | scripts/*.mjs | 91 |
| MCP enum | SEVERITY_RANK keys | 92 |
| MCP tool names | CLAUDE.md catalog | 93 | ◀ this iter
| CI workflow run commands | tripwire scripts | 84 |
Seven cross-reference surfaces now smoke-gated. The documentation
layer is no longer a dangling reference — every MCP tool registered
in source has a corresponding entry in the human/agent-facing catalog.
GATES THIS ITER
smoke: 95 → 96 (step 17z56 green)
contract: 9 MCP tools × 9 CLAUDE.md catalog entries verified
OBSERVATION
Documentation-as-contract is now a smoke-anchored idea, not just a
convention. ADR-150 and CLAUDE.md both have specific entries that
smoke verifies. Future contributors who update one without the other
get an immediate, named failure pointing at the missing surface.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
45c9650513 |
test(metaharness): MCP enum + SEVERITY_RANK vocabulary aligned (iter 92)
Two declarations of the same severity vocabulary exist:
1. _harness.mjs::SEVERITY_RANK keys (clean, info, low, medium, warn,
high, error, critical)
2. metaharness-tools.ts::metaharness_drift_from_history's
alertOnNewSeverity.enum
If a future iter adds 'urgent' to SEVERITY_RANK but forgets the MCP
enum (or vice versa), users see "alertOnNewSeverity must be one of..."
rejection of a supposedly-valid severity. The two sources drift
silently — no test catches it because each smoke step only looks at
ONE source.
WHAT SHIPPED — smoke step 17z55
- Extract SEVERITY_RANK keys via awk-range + grep
(8 keys: clean, info, low, medium, warn, high, error, critical)
- Extract enum values via grep on the alertOnNewSeverity bracket
(7 values: info, low, medium, warn, high, error, critical)
- Cross-check:
✓ Every enum value is in SEVERITY_RANK (subset)
✓ The only SEVERITY_RANK key NOT in the enum is 'clean'
(alerting on 'clean' is meaningless)
If a future iter:
- Adds 'urgent: 5' to SEVERITY_RANK only → "enum-missing-keys-mismatch"
- Adds 'urgent' to enum only → "enum-urgent-not-in-rank"
- Adds 'clean' to enum → mismatch (clean intentionally excluded)
DISCOVERY DURING IMPLEMENTATION
First-pass regex extracted unintended matches:
- `[a-z]+: [0-9]` matched 'Mode' (from recommendedMode) → 'ode'
- `'[a-z]+'` on the enum line captured 'string' from `type: 'string'`
Final regex: scoped to the SEVERITY_RANK literal range (awk pattern)
+ scoped to the bracket contents within `enum: [...]`. Documents the
right scoping pattern for future contributors who add similar gates.
CROSS-REFERENCE INTEGRITY MATRIX (post-iter-92)
| Source | Target | Iter |
|--------------------------|-------------------------|------|
| description string | SUBCOMMANDS keys | 73 |
| SUBCOMMANDS values | scripts/*.mjs | 89 |
| MCP handler runScript() | scripts/*.mjs | 90 |
| SKILL.md inline refs | scripts/*.mjs | 91 |
| MCP enum | SEVERITY_RANK keys | 92 | ◀ this iter
| CI workflow run commands | tripwire scripts | 84 |
Six cross-references now smoke-gated.
GATES THIS ITER
smoke: 94 → 95 (step 17z55 green)
contract: 7 enum values × 8 SEVERITY_RANK keys subset+complement check
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
8b3d11215a |
test(metaharness): SKILL.md script refs point at existing files (iter 91)
Third application of iters 89/90's cross-reference integrity pattern.
SKILL.md files embed paths like `scripts/foo.mjs` in their text. If a
future iter renames a script:
- SKILL.md still renders (markdown is permissive)
- The relative link 404s (a hidden break in the documentation)
- Ops users hunting for the implementation hit a dead end weeks
later, after the original committer has forgotten the context
WHAT SHIPPED — smoke step 17z54
Greps every SKILL.md for `scripts/[a-z_-]+\.mjs` references, asserts
each target exists in plugins/ruflo-metaharness/scripts/. Locks count
at ≥5 (at least one SKILL.md per skill references its implementation;
several reference multiple scripts).
If a future iter:
- Renames a script → "skill-ref-foo.mjs-missing"
- Deletes the SKILL.md set → "skill-ref-count-too-low:N"
The count floor (≥5 instead of exactly N) tolerates SKILL.md authors
adding cross-references between skills, which is legitimate and
shouldn't force a smoke update.
CROSS-REFERENCE INTEGRITY MATRIX (post-iter-91)
| Source | Target | Iter |
|--------------------------|-------------------------|------|
| description string | SUBCOMMANDS keys | 73 |
| SUBCOMMANDS values | scripts/*.mjs | 89 |
| MCP handler runScript() | scripts/*.mjs | 90 |
| SKILL.md inline refs | scripts/*.mjs | 91 | ◀ this iter
| CI workflow run commands | tripwire scripts | 84 |
Five distinct cross-reference surfaces are now smoke-gated. A rename
in scripts/ that misses ANY of the four reference sources produces a
specific failure message naming the missing file at smoke-time, not
a confused user weeks later.
GATES THIS ITER
smoke: 93 → 94 (step 17z54 green)
contract: 5+ SKILL.md script refs all resolve to existing files
THE FAMILY-WIDE INTEGRITY GATE
Smoke now anchors:
- Positive references: 4 surfaces × cross-check
- Negative references: 3 invariants (mint not-in-MCP, no from-repo,
no static metaharness imports)
Together they form a shape-of-codebase contract: smoke fails iff any
required reference goes stale OR any forbidden surface appears.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
0212ec9690 |
test(metaharness): MCP-tool runScript() refs point at existing scripts (iter 90)
Companion to iter-89's SUBCOMMANDS check. metaharness-tools.ts has 9
MCP-tool handlers, each calling runScript('<name>.mjs', args). If a
future iter renames a script but forgets to update its handler, the
agent-callable MCP surface fails at runtime with "Script not found".
THE THREE-WAY CONTRACT (post-iter-90, fully anchored)
CLI dispatcher: SUBCOMMANDS['X'] = 'X.mjs' (iter 73 description ↔ key)
│ (iter 89 value ↔ file)
v
scripts/X.mjs ← file exists
^
│
MCP handler: runScript('X.mjs', args) (iter 90 ref ↔ file)
Every reference from BOTH the CLI dispatcher AND the MCP wrapper
now has its target verified at smoke-time.
WHAT SHIPPED — smoke step 17z53
Parses every runScript('foo.mjs', ...) call site in metaharness-tools.ts
via regex:
/runScript\('([a-z-]+\.mjs)'/
For each unique reference: asserts the .mjs file exists in
plugins/ruflo-metaharness/scripts/. Locks count at 9 (one per MCP
tool; mint deliberately excluded per ADR-150 §Sandboxing).
If a future iter:
- Renames a script → "mcp-script-foo.mjs-missing"
- Adds a 10th MCP tool → "mcp-script-count-stale:10-expected-9"
- Removes a tool → "mcp-script-count-stale:8-expected-9"
GATES THIS ITER
smoke: 92 → 93 (step 17z53 green)
contract: 9 MCP runScript() refs × 9 file-existence checks all pass
ITER 73 + 84 + 89 + 90 CROSS-REFERENCE INTEGRITY MATRIX
| Reference source | Reference target | Iter |
|-----------------------------|------------------|------|
| description string | SUBCOMMANDS keys | 73 |
| SUBCOMMANDS values | scripts/*.mjs | 89 |
| MCP handler runScript() | scripts/*.mjs | 90 |
| CLI workflow run commands | tripwire scripts | 84 |
Four cross-references now produce specific failure messages when
the references drift.
PATTERN: SAME REGEX, DIFFERENT SOURCE
Iters 89 + 90 both:
1. Parse a regex match from a source file (the reference list)
2. For each match, assert a target file exists
3. Assert the count matches the documented contract
Future iters that add new cross-reference surfaces (e.g., a third
MCP-tool registry, a SKILL.md → script-path mapping) can adopt this
3-step shape. The pattern is now self-documenting.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
c6ecf14236 |
test(metaharness): SUBCOMMANDS map points at existing script files (iter 89)
The CLI dispatcher in v3/@claude-flow/cli/src/commands/metaharness.ts
maintains a SUBCOMMANDS map that routes `metaharness X` to
`plugins/ruflo-metaharness/scripts/X.mjs`. If a future iter:
a) Renames a script but forgets to update SUBCOMMANDS
b) Adds a SUBCOMMANDS entry but forgets to create the script
c) Deletes a script but leaves the entry
…the user gets a runtime "Script not found at <path>" error when
they run `npx ruflo metaharness X`. Pre-iter-89 no test caught these.
WHAT SHIPPED — smoke step 17z52
- Parses SUBCOMMANDS entries from metaharness.ts via regex:
/^\s+('?[a-z-]+'?):\s*'([a-z-]+\.mjs)'/
Handles both quoted ('mcp-scan': '...') and unquoted (mint: '...')
key forms that coexist in the source.
- For each mapping, asserts the .mjs file exists in
plugins/ruflo-metaharness/scripts/
- Locks COUNT === 10 (matches iter-73's documented dispatchable
subcommand list). If a future iter adds/removes an entry without
updating the count assertion, smoke fails with
`mapping-count-stale:N-expected-10`.
THE THREE-WAY CONTRACT NOW ANCHORED
v3/@claude-flow/cli/src/commands/metaharness.ts → SUBCOMMANDS map
|
v
plugins/ruflo-metaharness/scripts/<name>.mjs ← file exists
Plus iter-73's description string lists the 10 subcommands.
Total contract:
iter-73 — description ↔ map keys (literal subcommand names match)
iter-89 — map values ↔ file existence (script files actually exist)
iter-73 + iter-74 — mint deliberately NOT in MCP (negative guard)
iter-84 — all 3 compat tripwires present + executable
Together: every CLI subcommand has a documented description, a
dispatched script that exists, and a deliberate MCP-or-not policy.
GATES THIS ITER
smoke: 91 → 92 (step 17z52 green)
contract: 10 SUBCOMMANDS entries × 10 file-existence checks pass
PATTERN: CROSS-FILE REFERENCE INTEGRITY
Iters 73 + 89 share a pattern: source A references source B (by name).
A future iter modifying one without the other creates dangling
references. The smoke step that anchors the cross-reference produces a
SPECIFIC failure message ("script-foo.mjs-missing") instead of the
generic "ruflo metaharness X errors" the user would otherwise hit.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
95bacf8e8a |
test(metaharness): family-wide JSON-output contract gate (iter 88)
Iter 87 caught a bug where bench-similarity + bench-parse-mcp-scan
emitted a markdown header BEFORE the JSON payload when --format json
was passed. Iter-82's CI step JSON.parse'd the captured file and
silently failed.
The bug-class — "console.log fires regardless of --format json branch"
— could exist in any script. Iter 88 codifies a family-wide gate:
every metaharness script's --format json output must JSON.parse
without contamination.
WHAT SHIPPED — smoke step 17z51
Iterates 8 scripts with minimal valid args + --format json:
- oia-audit (--dry-run)
- score / genome / mcp-scan / threat-model (--path .)
- audit-list
- bench-similarity (--iters 1000)
- bench-parse-mcp-scan (--iters 1000)
For each: redirect stdout to a tempfile, then
node -e "JSON.parse(require('fs').readFileSync('$file'))"
If JSON.parse throws, the script's --format json mode is contaminated
and the smoke step fails with `${name}-not-parseable`.
WHAT THIS CATCHES THAT THE INDIVIDUAL TESTS DON'T
Each existing smoke step grep'd the SOURCE for "--format json"
support. None of them VERIFIED the output is actually parseable.
Iter-87 discovered that grep-level checks miss the contamination
bug-class.
Now: any future iter that adds an unconditional console.log to ANY
metaharness script — bench or otherwise — surfaces immediately as
${script}-not-parseable in smoke.
SCRIPTS NOT IN THE LIST (deliberate omissions)
- audit-trend, similarity: require multi-file args (would need
fixture setup in smoke). Already covered indirectly via roundtrip
Stages 1-12. Family-wide gate focuses on scripts with simple
invocations.
- drift-from-history: slow (~25s on real run because of audit-list
ONNX warmup). Smoke runs in seconds; this gate would 26x the
step's wall-clock. Covered by Stage 8 (iter 68) instead.
- mint: requires --name + filesystem-write protection. Tested via
graceful-degradation drill separately.
The 8 scripts covered here are the "tight loop" scripts that any
inline `node -e JSON.parse(...)` step in CI workflows would consume.
GATES THIS ITER
smoke: 90 → 91 (step 17z51 green)
runtime: 8 scripts × JSON.parse roundtrip in <2s total
PATTERN ANCHORED
The "verify the actual output, not just the source code" pattern from
iter 87 is now generalized. The same approach (mktemp + redirect +
parse-roundtrip) can be applied to any other contract that downstream
code depends on but smoke-grep doesn't directly verify.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
4ef9f36f18 |
fix(metaharness): bench scripts emit valid JSON + iter-86 wired to CI (iter 87)
DISCOVERED BUG: both bench scripts (iter 41 bench-similarity, iter 86
bench-parse-mcp-scan) emit a markdown header line BEFORE the JSON
payload when --format json. The header contaminates the captured file:
$ node bench-similarity.mjs --format json > /tmp/bench.json
$ cat /tmp/bench.json
# bench-similarity — ADR-152 §3.1 per-call cost ← contamination
iters: 5,000 ← contamination
{
"iters": 5000, "results": [...], ...
}
Iter 82 added a CI step that does:
$ node -e "JSON.parse(require('fs').readFileSync('/tmp/bench-similarity.json'))"
That JSON.parse SILENTLY FAILED in CI because of the leading `#` text.
The artifact upload still worked (the file existed) but the
GITHUB_STEP_SUMMARY table population threw a JSON.parse error and never
produced the table.
The bug was invisible because:
- Smoke step 17z45 grep'd workflow file structure (passed)
- The bench script's own --max-mean-us gate worked (passed)
- The artifact upload step has `if: always()` (succeeded)
Only the inline `node -e` parse-and-summarize step would have failed
in real CI runs.
WHAT SHIPPED
1. plugins/ruflo-metaharness/scripts/bench-similarity.mjs — guarded
the markdown-header `console.log` behind `ARGS.format !== 'json'`.
2. plugins/ruflo-metaharness/scripts/bench-parse-mcp-scan.mjs — same
guard. (Same bug shipped by iter 86 because I copied the
iter-41 pattern verbatim.)
3. .github/workflows/metaharness-ci.yml — iter-86's bench now wired
into the similarity-tests job, mirroring iter-82's pattern:
- Performance gate (--max-mean-us 5)
- JSON output captured to /tmp/bench-parse-mcp-scan.json
- GITHUB_STEP_SUMMARY table per category
- 90-day artifact upload
4. plugins/ruflo-metaharness/scripts/smoke.sh — step 17z50:
- Both bench scripts contain the iter-87 format-guard literal
- RUNTIME: each script's --format json output is JSON.parseable
(mktemp + redirect + node -e JSON.parse round trip)
- CI workflow wires bench-parse-mcp-scan
- Artifact upload step present
VERIFIED EACH BENCH NOW EMITS PARSEABLE JSON
$ node bench-similarity.mjs --iters 5000 --format json | head -1
{
$ node -e "JSON.parse(require('fs').readFileSync('/tmp/...'))"
parses OK with 3 categories
GATES THIS ITER
smoke: 89 → 90 (step 17z50 green; RUNTIME parse-roundtrip catches the bug-class)
yaml: python yaml.safe_load passes
ci: new bench step + artifact upload for parser perf
regression: bench-parse-mcp-scan runtime baseline ~0.07μs / 0.34μs / 0.95μs (unchanged)
RETROSPECTIVE — why iter-82's bug went undetected
Smoke step 17z45 (iter 82) anchored:
✓ workflow path /tmp/bench-similarity.json
✓ --format json flag
✓ Upload step name
✓ Artifact name template
✓ Summary header literal
None of those exercised JSON.parse on the actual output. The fix was
to add a RUNTIME assertion that mktemps the bench output and confirms
JSON.parse succeeds — exactly the kind of round-trip test that catches
the contamination class of bug.
Lesson: smoke-grep on the workflow YAML proves the YAML is correctly
structured, but doesn't prove the STEPS work. iter 87's runtime
assertion in step 17z50 closes that gap for both bench files.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
16894cfe71 |
perf(metaharness): bench-parse-mcp-scan — iter-50 parser characterization (iter 86)
iter 41 shipped bench-similarity. iter 50's parseMcpScanText was
promoted to a hot-path consumer (used by every oia-audit invocation
via the iter-56 parallel batch). But the parser had no perf
characterization — if a future refactor accidentally added an O(n²)
pass, the regression would only surface as drift-from-history wall
clock growing.
Iter 86 adds the matching perf characterization for the iter-50
parser.
MEASURED BASELINE (50k iters, Apple Silicon / Node 22)
| Category | mean | p50 | p99 |
|----------------------|-----------:|-----------:|-----------:|
| empty | 0.074μs | 0.042μs | 0.208μs |
| typical (1 INFO) | 0.352μs | 0.333μs | 0.375μs |
| rich (5 findings) | 1.085μs | 1.042μs | 1.167μs |
INTERPRETATION
- empty: essentially free (~zero-pass loop)
- typical (real ruflo output): 0.35μs — faster than similarity()'s
~0.4μs, confirms parser is NOT on the critical perf path
- rich (5 findings, 14 lines including continuations): 1.09μs —
scales linearly with line count
Conclusion: even at 1000 parse/sec the parser costs <2ms total.
Below the noise floor for any conceivable production usage.
WHAT SHIPPED
plugins/ruflo-metaharness/scripts/bench-parse-mcp-scan.mjs — 3
fixture categories + standard bench harness (matches iter-41 pattern):
- --iters N (default 100k)
- --format json
- --max-mean-us N (CI regression gate)
SMOKE STEP 17z49 anchors:
- Bench file executable + parses
- All 3 fixture constants present (EMPTY/TYPICAL/RICH)
- Imports from production _harness.mjs (not a forked copy)
- --max-mean-us gate flag exposed
- Runtime: --max-mean-us 5 passes (proves baseline < 5μs)
- Runtime: --max-mean-us 0.0001 trips (proves gate logic correct)
GATES THIS ITER
smoke: 88 → 89 (step 17z49 green)
runtime: bench produces sub-2μs results across all 3 categories
BENCH FAMILY (post-iter-86)
bench-similarity.mjs (iter 41) — similarity() function
bench-recordpair-overhead (iter 24) — iter-12 dispatch overhead
bench-parse-mcp-scan.mjs (THIS ITER) — parseMcpScanText parser
Three hot-path primitives now have measured baselines + CI gates.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
3a0033df1b |
test(metaharness): MCP-layer alertOnNewSeverity Phase 4 assertions (iter 85)
Iter 78 added --alert-on-new-severity at the script layer and the
MCP-tool input schema. Iter 79 wired it into the weekly cron. Iter 79
also added Stage 12 to the roundtrip — but that tests the SCRIPT
layer, not the MCP tool's wrapper. If a future iter accidentally
drops the args.push('--alert-on-new-severity', ...) line in
metaharness-tools.ts, no test catches it.
Iter 85 fixes that with positive-case Phase 4 assertions exercising
the input through the actual MCP handler.
WHAT SHIPPED — test-mcp-tools.mjs Phase 4 extension (5 new assertions)
✓ alertOnNewSeverity echoed in payload (alert.newSeverityThreshold === 'info')
✓ exitCode === 1 when alert triggered (CLI exit propagated through MCP wrapper)
✓ success === false when alert fires (iter 44 wrapper-semantic intact)
Plus the existing iter-71 + iter-54 Phase 4 assertions for the same
tool keep firing — total positive-case coverage for metaharness_
drift_from_history is now: baselineFile fastpath, skippedAuditList,
alertOnNewSeverity echo, exit code propagation, success semantic.
THE SYNTHETIC FIXTURE
baselineNoFindings.json = {
fingerprint: { score: {...}, genome: {...} },
components: { mcpScan: { json: { findings: [] } } }
}
baseline has no findings. Current ruflo audit has 1 INFO finding.
With alertOnNewSeverity='info' the gate fires (introduced finding ≥
info threshold) → triggered=true → exit 1 → success=false.
The asymmetric setup (baseline clean, current with findings) is the
realistic "regression introduced new finding" scenario.
GUARDED CONTRACTS
If a future iter:
- drops the alertOnNewSeverity MCP input → echo assertion fails
- removes args.push('--alert-on-new-severity', ...) → echo fails
- reverts iter-44's success semantic → success assertion fails
- breaks the underlying script's exit code → exitCode assertion fails
Each failure mode produces a specific, named assertion failure.
SMOKE STEP 17z48 anchors:
- All 3 critical assertion text markers
- Synthetic fixture filename
- alertOnNewSeverity: 'info' literal input
GATES THIS ITER
smoke: 87 → 88 (step 17z48 green)
runtime: MCP test 117 → 120 assertions (+5 alertOnNewSeverity asserts)
ITER 78-79-85 ALERT-GATE FULL-STACK COVERAGE
78 — flag at script + MCP schema + docs + smoke 17z41
79 — wired into weekly cron + Stage 12 (script-layer verify)
85 — MCP wrapper layer verified end-to-end ◀ this iter
The alert gate is now contract-anchored at three layers: script,
wrapper, and runtime.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
a6113fe8e2 |
test(metaharness): positive-presence guard for all 3 compat tripwires (iter 84)
The 3 compat tripwires (router API / mcp-scan text / fingerprint schema)
each protect against silent upstream drift. But until iter 84, only the
CI workflow that RUNS them caught accidental deletion — and only AFTER
the metaharness-real-data job ran (~5 minutes per PR).
A smoke-level positive-presence guard catches deletion in ~50ms.
WHAT SHIPPED
plugins/ruflo-metaharness/scripts/smoke.sh — step 17z47:
For each of the 3 tripwire scripts:
✓ file exists
✓ file is executable
✓ syntax-parses (node --check)
✓ supports --format json (CI-consumable contract)
✓ has process.exit(2) graceful-error path
Plus 3 grep guards that the CI workflow actually invokes each:
✓ check-metaharness-compat.mjs referenced in metaharness-ci.yml
✓ check-mcp-scan-format.mjs referenced
✓ check-fingerprint-schema.mjs referenced
WHAT THIS CATCHES THAT THE EXISTING CI DOESN'T
Today (pre-iter-84):
1. Someone deletes scripts/check-mcp-scan-format.mjs
2. PR opens
3. PR runs smoke locally → passes (smoke doesn't reference the file)
4. PR pushed → CI runs
5. metaharness-real-data job hits the missing-file step → fails
~5min into the run
6. Author scratches head, finds out which file disappeared
Post-iter-84:
1. Someone deletes scripts/check-mcp-scan-format.mjs
2. PR opens
3. Smoke runs → 17z47 fails IMMEDIATELY with "missing-check-mcp-scan-format.mjs"
4. Author fixes before push
The smoke gate is ~5min faster + more specific (file name) than the
CI gate.
ITER 60 SMOKE STEP RELAXED
iter-83's ADR update bumped "sixty iterations" → "eighty-two iterations"
and "iters 33-59" → "iters 33-82". iter-60's smoke step 17z23 grep was
hard-coded to those old literals. Relaxed to regex form that accepts
ANY iter range and ANY iteration-count word, so future ADR refreshes
(iter 100, 200, etc.) don't break the iter-60 coverage assertion.
GATES THIS ITER
smoke: 86 → 87 (step 17z47 green; 17z23 grep relaxed for forward-compat)
no code changes — pure anti-regression infrastructure
POSITIVE + NEGATIVE GUARD COMPLETION
Negative guards (iter 73-74): "thing X must NOT exist"
Positive guards (iter 84): "thing Y must exist"
Together they form a complete shape-of-codebase contract: the smoke
gate fails iff someone adds something forbidden OR removes something
required.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
a4d1878df5 |
docs(adr): ADR-150 notes catch up through iter 82 (iter 83)
Iter 60 brought the ADR current through iter 59. Iters 60-82 added a second wave of work that wasn't documented: parallelization sweep, 3-tripwire upstream defense, anti-regression infrastructure, drift autonomy arc completion, doctor expansion, MCP-CLI parity, artifact tracking. This iter records the canonical version. WHAT'S NEW IN THE ADR 1. Header status: "Phase 3 §3.1 ✅ iters 33-59" → "33-82" 2. Date marker: "60 iterations" → "82 iterations" 3. New section "Iters 60-82 — performance / observability / contract hardening" with 7 sub-arcs: - Parallelization sweep (iters 56-59 + 65 + 67) 4.59x oia-audit, 19x drift-from-history fastpath ladder - Three-tripwire upstream-contract defense (iters 12 + 80 + 81) check-metaharness-compat / check-mcp-scan-format / check-fingerprint-schema - Anti-regression infrastructure (iters 64 + 72 + 73 + 74) Primitive unit tests + negative-grep guards - Bug-discovery + fix arc (iters 47-77) 4 latent bugs surfaced + 5-layer severity defense - Drift-detection autonomous arc (iters 53-79) From CLI primitive → MCP tool → weekly cron → failure-path coverage → orthogonal severity alert gate - Doctor expansion (iters 45 + 52 + 61) Ruflo-side integration check distinct from upstream-dep check - MCP-CLI parity matrix (iters 53 → 71) Full parity for all 9 MCP tools; deliberate mint-CLI-only asymmetry enforced - Artifact-tracking family (iters 7 + 69 + 82) 90-day cross-artifact comparability 4. Fleet status updated: - Smoke step count: 62 → 85 - MCP runtime: 115 → 117 - Roundtrip: 31 → 66 across 12 stages - Unit tests: 53 → 94 - +3 compat tripwires - +4 CI workflows SMOKE STEP 17z46 anchors: - Phase 3 ✅ iters 33-82 status - 82-iter marker - Section header for iters 60-82 - 3 sub-section headers (tripwire / drift arc / artifact family) - Fleet status post-iter-82 WHY THIS MATTERS ADR-150 is the canonical engineering record. Letting it drift behind 22 iterations of substantive work creates a future-maintainer cliff: they'd have to walk the git log + read every smoke step to understand what happened. Now anyone reading the ADR sees the full 82-iter arc in one place. GATES THIS ITER smoke: 85 → 86 (step 17z46 green; no code changes — doc only) fleet: 33/33 (no fleet impact — pure documentation) ITER 60 + 83 ADR-UPDATE CADENCE 60 — caught up to iter 59 83 — caught up to iter 82 ◀ this iter Pattern: every ~20 iters of substantive work warrants an ADR refresh. Future iters can adopt this cadence — if the work has shipped, the ADR should reflect it. Co-Authored-By: RuFlo <ruv@ruv.net> |
||
|
ruv
|
e49124e688 |
ci(metaharness): bench-similarity artifact + per-run summary (iter 82)
Iter 41 added the performance gate (--max-mean-us 10) but the actual
bench results weren't persisted. Every CI run measured perf, gated on
the ceiling, then threw the numbers away. There was no way to look at
"how has perf changed over the last 90 days" without re-running benches
locally and comparing against unknown baselines.
Iter 82 keeps the gate AND captures the numbers as a 90-day artifact.
WHAT SHIPPED
.github/workflows/metaharness-ci.yml — similarity-tests job:
- Performance gate step extended:
bench-similarity.mjs --iters 100000 --max-mean-us 10 \
--format json > /tmp/bench-similarity.json
JSON output captures all 3 categories (cheap, typical, rich)
with mean / p50 / p99 wall-clock per category.
- GITHUB_STEP_SUMMARY enriched with a per-category table:
| cheap | mean 0.343μs | p99 0.500μs |
| typical | mean 0.412μs | p99 0.583μs |
| rich (32 agents) | mean 1.875μs | p99 2.167μs |
Visible in the Actions summary tab without clicking through.
- New Upload step:
name: bench-similarity-${{ github.run_id }}
path: /tmp/bench-similarity.json
retention-days: 90
VALUE ACCUMULATION
Today: 1 baseline (this PR's run)
After 7 days: ~7 baselines across 7 PRs
After 90 days: ~90 baselines
A future regression analyzer can:
- Download N most-recent artifacts
- Plot mean over time per category
- Flag any category that drifted upward by >50% vs trailing-7d average
- Catch performance regressions the smoke gate's loose 10μs ceiling
misses (a regression from 0.4μs to 5μs passes smoke but is 12x
slower than baseline — exactly the kind of silent erosion a
dedicated analyzer would surface).
The artifact-history infrastructure costs nothing per run (~1KB JSON,
90-day retention is essentially free) but unlocks the regression
detection that the smoke gate can't do alone.
SMOKE STEP 17z45 anchors:
- /tmp/bench-similarity.json path
- --format json flag in the bench command
- "Upload bench-similarity artifact" step name
- Artifact name template with github.run_id
- 90-day retention
- GITHUB_STEP_SUMMARY header literal
GATES THIS ITER
smoke: 84 → 85 (step 17z45 green)
ci: new artifact + summary step
yaml: python yaml.safe_load passes
ARTIFACT-TRACKING FAMILY (post-iter-82)
oia-audit-weekly.yml uploads oia-audit JSON (iter 7)
oia-audit-weekly.yml uploads drift-trend JSON (iter 69)
metaharness-ci.yml uploads bench-similarity JSON (THIS ITER)
Three independent perf/posture artifacts now accumulate per run. All
three converge on a 90-day retention window so cross-artifact analysis
stays comparable.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
b0ac5998ea |
test(metaharness): fingerprint-schema compat tripwire (iter 81)
Third compat tripwire in the family. Protects _similarity.mjs from
silent upstream schema drift in `metaharness score` / `metaharness genome`.
THE BUG-CLASS THIS PROTECTS AGAINST
iter 47 discovered the harness/metaharness binary schema mismatch.
The fix routed score+genome through runMetaharness. But:
Pre-iter-81: nothing asserted that the metaharness schema STAYS
correct. If upstream renames `harnessFit` to `score` or
`harness_fit`:
- projectToVec reads `s.harnessFit` → undefined
- All 5 score-derived numerics collapse to 0
- cosine signal degrades to flat-zero
- similarity returns only categorical+jaccard agreement
- Drift detection silently breaks (similar verdict on ANY input)
And no test would fail — the iter-72 unit tests use hand-built
fixtures that hard-code the correct field names. The iter-47/68
roundtrip uses real CLI output, but if the CLI changed AND the
unit fixtures kept using the old names, the roundtrip would
produce similarity=0.0 → fail Stage 3 (overall===1) BUT only on
live CI, not on the unit-test layer.
iter-81 makes the schema invariant explicit at CI-time, BEFORE
the roundtrip — pinpoints the breakage instead of cascading.
WHAT SHIPPED
1. scripts/check-fingerprint-schema.mjs — runs `metaharness score`
and `metaharness genome` against ruflo, asserts each of the 14
fields _similarity.mjs reads is present in the output:
score: harnessFit, compileConfidence, taskCoverage, toolSafety,
memoryUsefulness, estCostPerRunUsd, recommendedMode,
archetype, template
genome: repo_type, agent_topology, risk_score, test_confidence,
publish_readiness
Tests `value !== undefined` (key exists, regardless of value type)
so the tripwire fires on RENAMES but tolerates value-shape changes
(numbers becoming strings, arrays becoming nested objects, etc.).
Exit 0 — all 14 fields present OR metaharness absent (graceful skip)
Exit 1 — at least one field missing
Exit 2 — unexpected error
2. .github/workflows/metaharness-ci.yml — new step in metaharness-
real-data job runs the tripwire alongside the iter-80 mcp-scan-
format check. Both run BEFORE the roundtrip so format/schema
regressions surface FIRST.
3. Smoke step 17z44 anchors all 14 field literals (anti-shrink: a
future iter that adds a 15th projectToVec input must also add
the field to this script AND its smoke marker).
COMPAT TRIPWIRE FAMILY (post-iter-81)
check-metaharness-compat.mjs (iter 12)
→ @metaharness/router public API
check-mcp-scan-format.mjs (iter 80)
→ harness mcp-scan text-output format
check-fingerprint-schema.mjs (THIS ITER)
→ metaharness score / genome JSON field names
Each tripwire protects a distinct upstream-dependency surface that
ruflo's code reads. Combined, they form an "upstream contract"
inspection layer — if any upstream surface drifts, ruflo CI fails
with a SPECIFIC tripwire error before cascading to downstream
symptoms.
GATES THIS ITER
smoke: 83 → 84 (step 17z44 green)
ci: new pre-roundtrip step in metaharness-real-data job
runtime: 14/14 fields verified against metaharness@0.1.14
tsc: no TS source changes
DEFENSE-IN-DEPTH OBSERVATION
Each compat tripwire adds:
- A specific upstream contract under explicit version control
- A clear failure message that points at WHICH upstream surface drifted
- A pre-roundtrip CI gate that catches the regression before
downstream symptoms cascade
Cost: ~5s per tripwire run. Payoff: catching upstream drift at the
specific PR that introduces it instead of weeks later when production
behavior degrades silently.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
ca56902b27 |
test(metaharness): upstream mcp-scan text-format compat tripwire (iter 80)
Iter 50's parseMcpScanText reverse-engineered `harness mcp-scan` text
output to extract structured findings. If upstream changes the format
(drops `[]` markers, switches to JSON, changes the Result: regex), the
parser silently returns empty findings and iter-49's introduced/cleared
diff regresses to the original dead-code state.
This iter adds a compat tripwire mirroring scripts/check-metaharness-
compat.mjs (which protects the @metaharness/router API surface).
WHAT SHIPPED
1. scripts/check-mcp-scan-format.mjs — runs `harness mcp-scan` against
ruflo and asserts 4 invariants:
✓ At least one ` [SEV] message` line in expected regex shape
✓ Result line matches `Result: SEV (N finding, M high)`
✓ parseMcpScanText loadable (sanity)
✓ parseMcpScanText extracts ≥ 1 finding from real output
Exit 0 — invariants intact OR metaharness not installed (graceful skip)
Exit 1 — invariant violated, parser will silently break
Exit 2 — unexpected error
The "extract ≥1 finding from real output" assertion is the
load-bearing check. It catches drift the regex above might miss
(e.g., if upstream switched to identical-looking but slightly
different whitespace).
2. .github/workflows/metaharness-ci.yml — new step in the
`metaharness-real-data` job runs the tripwire BEFORE the roundtrip.
If upstream format drifted, CI fails with a clear "format invariant
violated" message before chasing downstream symptoms.
3. plugins/ruflo-metaharness/scripts/smoke.sh — step 17z43 anchors:
- tripwire script exists + parses
- All 3 invariant text markers (finding/result/parser-roundtrip)
- CI workflow references the script
- Runtime: tripwire passes against the locally-installed metaharness
WHY TWO TRIPWIRES NOW EXIST
scripts/check-metaharness-compat.mjs (iter 12)
→ Protects @metaharness/router public API (constructor, route(),
fromExamples(), TrainedRouter.fromJSON())
scripts/check-mcp-scan-format.mjs (THIS ITER)
→ Protects `harness mcp-scan` text format (regex + parser-roundtrip)
These cover different upstream surfaces. The router compat-check ran
at the API level; the new check is at the CLI-output level. Both are
necessary because ruflo depends on both surfaces in different ways.
VERIFIED LOCALLY
$ node scripts/check-mcp-scan-format.mjs
✓ All 4 invariants intact.
GATES THIS ITER
smoke: 82 → 83 (step 17z43 green)
ci: new pre-roundtrip step in metaharness-real-data job
runtime: 4 format invariants pass against metaharness@0.1.14
PROTECTION RATIONALE
Iter 50 was a downstream workaround for issue ruvnet/agent-harness-
generator#16 (`harness mcp-scan` emits text only with --json). When
upstream eventually fixes that issue and emits structured JSON, the
parser's regex-based extraction will STILL work (the parser falls back
gracefully). But the tripwire will fail on the FIRST run after upstream
adds the JSON output AND drops the text format — which is a real risk
because upstream is in v0.1.x.
When the tripwire eventually fires, the fix is upstream-aware:
- If text format dropped → switch parseMcpScanText to read r.json directly
- If JSON added alongside text → prefer r.json, keep parser as fallback
The tripwire makes that decision visible at the PR that introduces it,
not 4 weeks later when someone notices drift detection is silent.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
f7383595e7 |
feat(metaharness): weekly cron uses iter-78 alert-on-new-severity (iter 79)
Iter 78 added --alert-on-new-severity but no caller used it. Iter 79
wires it into the weekly cron AND adds the corresponding roundtrip
verification.
WHAT SHIPPED
1. .github/workflows/oia-audit-weekly.yml — drift step extended:
drift-from-history \
--baseline-file /tmp/prior/oia-audit.json \
--threshold 0.85 \
--alert-on-new-severity high ◀ NEW
--format json
The weekly cron now alerts on EITHER:
- structural similarity < 0.85
- OR any new finding ≥ high severity
This closes the iter-78-flagged gap: previously a regression that
added a new mcp-scan HIGH finding without moving similarity below
threshold would have been invisible in the Actions summary.
2. plugins/ruflo-metaharness/scripts/test-pipeline-roundtrip.mjs — new
Stage 12 (6 assertions):
Setup:
- baseline: clean baseline-file (no findings)
- current: fresh ruflo audit (has 1 INFO finding)
- --threshold 0.5 (similarity won't fire — structures match)
- --alert-on-new-severity info (gate fires on the 1 introduced INFO)
Assertions:
✓ drift-from-history produced JSON
✓ alert.triggered === true (gate fires)
✓ reasons mention "finding(s) at or above" (not similarity)
✓ elevatedFindings non-empty (which finding tripped)
✓ exit === 1 (CLI exit reflects alert)
✓ alert.newSeverityThreshold echoed (input plumbed through)
This verifies the iter-78 gate works through the drift-from-history
wrapper, not just at the script flag level.
WHY THIS COMPLETES THE iter-78 LOOP
Iter 78 timeline:
- Added flag to drift-from-history.mjs
- Surfaced in MCP tool input schema
- Documented in CLAUDE.md
- Unit-tested via smoke step 17z41
But the CRON, the actual production consumer of drift detection,
wasn't updated. iter 79 connects the wires.
GATES THIS ITER
smoke: 81 → 82 (step 17z42 green)
roundtrip: 60 → 66 assertions (+6 for Stage 12)
yaml: python yaml.safe_load passes
cron: drift step now has 2 orthogonal alert sources
OPERATIONAL IMPACT
Pre-iter-79 weekly cron:
- Structural drift > 15% → workflow fails, drift-trend artifact uploaded
- New CRITICAL mcp-scan finding with structure intact → workflow stays
"passing" (similarity above threshold). User sees green. Misses regression.
Post-iter-79 weekly cron:
- Structural drift > 15% → workflow fails
- New HIGH-or-higher mcp-scan finding → workflow ALSO fails
- Either signal lights up the alert.
The "deep integration" story now has a fully cron-enforced security
posture watcher, not just a structural-similarity watcher.
ITER 78-79 ALERT-GATE COMPLETION ARC
78 — added --alert-on-new-severity at script + MCP + docs
79 — wired it into the actual cron + roundtrip-verified ◀ this iter
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
0c5782a920 |
feat(metaharness): drift-from-history --alert-on-new-severity gate (iter 78)
Pre-iter-78, drift-from-history's alert triggered ONLY on structural-
distance threshold. A weekly cron running with --threshold 0.95 would
MISS a sudden CRITICAL mcp-scan finding if the broader structure
stayed similar.
Example failure: someone commits a new MCP server with default-allow
permissions. score/genome don't change meaningfully (still 'node_mcp_ci',
still 'typescript-sdk-harness') → overall similarity stays 0.99. But
mcp-scan would now report a CRITICAL finding. Pre-iter-78 the weekly
cron's drift detection would report 'near-identical' — and the human
reading the Actions summary would see "no drift" + miss the new finding.
Iter 78 adds an orthogonal alert gate.
WHAT SHIPPED
1. plugins/ruflo-metaharness/scripts/drift-from-history.mjs
- New `--alert-on-new-severity <level>` flag (default off → no behavior change)
- When set: any introduced finding with rankSeverity(severity) ≥
rankSeverity(level) flips alertTriggered=true and appends a
reason like "N new finding(s) at or above critical severity"
- alert.elevatedFindings[] surfaced in payload so consumers see WHICH
findings tripped the gate (not just a count)
- alert.reasons[] array supplements .reason string for structured access
- Imports rankSeverity from _harness.mjs (iter 63 shared util)
2. v3/@claude-flow/cli/src/mcp-tools/metaharness-tools.ts
- alertOnNewSeverity input added to metaharness_drift_from_history
- enum constrained: info|low|medium|warn|high|error|critical
(matches iter-63 SEVERITY_RANK vocabulary)
- args.push('--alert-on-new-severity', input.alertOnNewSeverity)
- Description calls out orthogonality: "A CRITICAL finding triggers
even if structural similarity > threshold."
3. CLAUDE.md surfaces the new flag
4. Smoke step 17z41 anchors:
- CLI flag + args key + elevatedFindings + rankSeverity import
- MCP input + dispatch
- CLAUDE.md mention
- RUNTIME: synthetic clean baseline + ruflo's real INFO finding →
alert-on-new-severity info exits 1 (gate triggers correctly)
VERIFIED END-TO-END
$ # clean baseline (no findings)
$ node oia-audit.mjs --dry-run --format json > /tmp/baseline.json
$ jq '.components.mcpScan.json = {findings:[]}' /tmp/baseline.json > /tmp/clean.json
$ # default (no flag) — does not fire on similarity > threshold
$ drift-from-history --baseline-file /tmp/clean.json --threshold 0.5
alert.triggered: false
$ # with --alert-on-new-severity info — fires
$ drift-from-history --baseline-file /tmp/clean.json --threshold 0.5 \
--alert-on-new-severity info
alert.triggered: true
alert.reasons: ['1 new finding(s) at or above info severity']
alert.elevatedFindings: [{ severity: 'info', message: 'No MCP security issues found...' }]
exit: 1
DESIGN: ORTHOGONAL, NOT COMPOUND
The two alert gates work independently:
triggered = (similarity < threshold) OR (any new finding ≥ severity)
Either gate can fire. This matches the operational reality: structural
drift and security-posture drift are different failure modes that
should each have their own threshold + own visibility.
WHY OPT-IN (NOT DEFAULT)
Default behavior unchanged (no flag = no new alert). Existing cron
scripts that depend on the iter-53 behavior keep working. Ops users
who want the new gate explicitly opt in by passing the flag.
GATES THIS ITER
smoke: 80 → 81 (step 17z41 green)
runtime: smoke RUNS the actual workflow + asserts exit=1 on alert
tsc: CLI rebuilds clean
roundtrip: 60/60 (no behavior change on default path)
CLI-MCP PARITY (post-iter-78)
--path ✓ both
--baseline-since ✓ both
--baseline-key ✓ both (iter 66/71)
--baseline-file ✓ both (iter 67/71)
--threshold ✓ both
--alert-on-new-severity ✓ both (iter 78)
--dry-run ✓ both
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
a9c518249e |
test(metaharness): roundtrip Stage 11 — diff symmetry + dedup (iter 77)
Stage 10 (iter 76) proved the cleared side of audit-trend's
introduced/cleared diff works. Three cases still uncovered: the
introduced side, the both-different case, and the identical-dedup
case. Stage 11 closes all three.
WHAT SHIPPED — Stage 11 (9 assertions across 3 sub-cases)
11a — INTRODUCED symmetry
Setup: baseline clean, current has finding A
Expected: introducedCount === 1, clearedCount === 0
Verifies the cleared-side test (Stage 10) wasn't a fluke that
happened to work only because of dedup-key collisions.
11b — BOTH-DIFFERENT (cleared + introduced simultaneously)
Setup: baseline has finding B, current has finding A
Expected: introducedCount === 1 (A), clearedCount === 1 (B)
Verifies dedup correctly DISCRIMINATES the two findings.
Finding-A and Finding-B have different severity/id/server/tool
so the fingerprint key MUST differ — if dedup collapses them,
this test fails.
11c — IDENTICAL DEDUP
Setup: baseline and current have THE SAME finding A
Expected: introducedCount === 0, clearedCount === 0
Verifies the fingerprint dedup actually works — same data on
both sides produces no diff. If the dedup key were buggy (e.g.,
missing a field), this test would report 1+1 on each side.
DEDUP KEY EXERCISED
audit-trend.mjs's fingerprint() function builds a string from
severity + id + server + tool + message-prefix. Stage 11b's
finding-A vs finding-B differ in ALL five → the fingerprints
discriminate. If the function were truncated to use only severity
(e.g.) the test would fail because both have different severities
but the WRONG one would be classified.
ITER 49→50→72→76→77 INTRODUCED/CLEARED ARC
49: documented dead code
50: fixed the data layer (parser)
72: unit-tested the parser
76: cleared side proven on real diffs (Stage 10)
77: introduced + both-different + dedup proven ◀ this iter
5 iters to fully verify what was originally a 28-iter-old latent bug.
SMOKE STEP 17z40 anchors:
- Stage 11 section header
- Both synthetic finding ids (anti-shrink)
- 11a introducedCount assertion
- 11b dedup-discriminates assertion
- 11c identical-dedup assertion
- Runtime: roundtrip ≥60 passing
GATES THIS ITER
smoke: 79 → 80 (step 17z40 green)
roundtrip: 51 → 60 assertions (+9 for Stage 11)
The audit-trend introduced/cleared diff is now contract-anchored at
all four corners: clear, introduce, swap, dedup.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
1a4fe374a3 |
test(metaharness): roundtrip Stage 10 — introduced/cleared findings diff (iter 76)
Iter 49 documented that audit-trend's findings.introducedCount /
clearedCount diff was dead code (mcp-scan emitted text only, no
structured findings to walk).
Iter 50 added parseMcpScanText so findings ARE now structured.
But no roundtrip test verified the diff actually FIRES on a real
finding-set change. Stage 7 mutates fingerprint (score/genome) but
not mcpScan.json.findings — so introducedCount/clearedCount stay at
0 even through all the prior mutation tests.
Stage 10 fixes that: takes the real audit, ADDS a synthetic finding
to a baseline copy, and asserts audit-trend reports
clearedCount === 1 with the synthetic's severity + id intact.
WHAT SHIPPED — Stage 10 (5 assertions)
✓ audit-trend produced JSON
✓ clearedCount === 1 (the synthetic appeared in baseline, missing in current)
✓ introducedCount === 0
✓ cleared finding severity preserved through the diff (medium)
✓ cleared finding id preserved (iter-76-synthetic-finding)
THE SYNTHETIC FINDING
{
severity: 'medium',
id: 'iter-76-synthetic-finding',
server: 'synthetic-server',
tool: 'synthetic-tool',
message: 'Stage 10 synthetic finding for clearedCount verification'
}
Added to baseline.components.mcpScan.json.findings[]. The current
audit (Stage 1's untouched output) has no such finding → audit-trend
walks the fingerprint() dedup hash and correctly classifies the
synthetic as "cleared" (in baseline, not in current).
WHY THIS IS THE FINAL PROOF FOR THE ITER-49→50 CHAIN
Iter 49 — noticed findings.findings array was always empty in
production audits → flagged "dead code"
Iter 50 — added parseMcpScanText → findings now populated
Iter 72 — unit-tested the parser in isolation
Iter 76 — proved audit-trend's introduced/cleared LOGIC works
with the now-populated findings ◀ this iter
Without Stage 10, the iter-50 fix was verified at the parser level
(iter 72 unit tests) and at the data-shape level (iter 49 schema
contracts) but never at the DIFF level. A regression that broke
audit-trend's fingerprint() dedup key — even if findings parsed
correctly — would have been invisible until production.
SMOKE STEP 17z39 anchors:
- Stage 10 section header
- Synthetic finding id literal (anti-shrink guard)
- clearedCount === 1 assertion
- introducedCount === 0 assertion
- Severity + id preservation
- Runtime: roundtrip ≥51 passing
GATES THIS ITER
smoke: 78 → 79 (step 17z39 green)
roundtrip: 46 → 51 assertions (+5 for Stage 10)
ITER 49-50-72-76 INTRODUCED/CLEARED ARC COMPLETE
49: documented dead code
50: fixed the data layer (parser)
72: unit-tested the parser
76: proved the consumer logic on real diffs ◀ this iter
4 iters to fully verify what was originally a 28-iter-old latent bug.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
6ace68c0ef |
test(metaharness): roundtrip Stage 9 — drift-from-history fastpath catches mutation (iter 75)
Iter 68's Stage 8 proved the iter-67 --baseline-file fastpath correctly
identifies self-match. It did NOT prove the fastpath ALSO catches drift.
A regression that broke the fastpath's similarity computation (e.g.,
silently returning {overall: 1} for all inputs) would pass Stage 8 but
be useless in production.
Stage 9 closes the gap by re-using Stage 7's mutated audit as a
baseline file and asserting the fastpath wrapper detects drift.
WHAT SHIPPED — test-pipeline-roundtrip.mjs Stage 9 (8 assertions)
✓ drift-from-history with mutated baseline produced JSON
✓ Stage 9 fastpath: usedBaselineFile === true (fastpath did fire)
✓ Stage 9: structuralDistance present
✓ Stage 9: verdict !== near-identical (drift detected)
Measured: 'moderate-drift'
✓ Stage 9: overall < 1 (~0.7086)
✓ Stage 9: distance > 0 (~0.2914)
✓ Stage 9: --threshold 0.95 fires on mutated baseline via fastpath
✓ Stage 9: drift-from-history exit=1 when alert triggered
The mutation re-uses Stage 7's payload (harnessFit -40, taskCoverage
-35, toolSafety -30, compileConfidence -25, archetype + template
swapped, agent_topology rebuilt). This calibrates Stage 9 against the
same threshold-crossing magnitude — overall drops from 1.0 to ~0.71.
WHY THIS CLOSES A SUBSTANTIVE GAP
Pre-iter-75 coverage matrix:
self-match mutation
audit-trend (Stage 3) ✓ ✗
audit-trend (Stage 7) ✓ ✓
drift-from-history (Stage 8) ✓ ✗
drift-from-history (Stage 9) ✓ ✓ ◀ THIS ITER
A future change that broke the wrapper's similarity-result extraction
(e.g., reading the wrong nested field, or assuming a different output
shape) would pass Stage 8 (which only verifies self === 1) but fail
Stage 9 (which verifies non-self ≠ 1).
Bug-shape this catches: anything that converts the fastpath into a
"silent always-match" — those are the worst kind of regression
because they look healthy at the highest aggregate level.
SMOKE STEP 17z38 anchors:
- Stage 9 section header
- usedBaselineFile === true assertion (fastpath fires)
- Verdict-flip assertion (drift detected)
- Alert-fires assertion (end-to-end alert chain)
- exit=1 assertion (CLI exit semantics)
- Runtime: roundtrip ≥46 passing
GATES THIS ITER
smoke: 77 → 78 (step 17z38 green)
roundtrip: 38 → 46 assertions (+8 for Stage 9)
ITER 47-75 ROUNDTRIP COVERAGE PROGRESSION
47 — basic roundtrip 15 assertions
49 — non-similarity schemas 24
51 — drift detection on mutation (audit-trend) 31
68 — drift-from-history end-to-end (self-match) 38
75 — drift-from-history fastpath catches drift 46
The roundtrip is now exhaustively bidirectional at BOTH the
audit-trend layer AND the drift-from-history wrapper layer.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
45c0f9bd3d |
test(metaharness): three more architectural-constraint negative guards (iter 74)
Iter 73 introduced the negative-grep pattern for the mint-anti-MCP
rule. This iter extends the pattern to two more ADR-150 invariants
that previously had no test:
GUARD 1 — ADR-150 §Sandboxing: no `from-repo` skill or MCP tool
`harness from-repo <url>` clones arbitrary git URLs. The upstream
supports it; ruflo deliberately doesn't wrap it because exposing
arbitrary-URL-fetch to LLM agents is too powerful. Currently no
test catches a future iter that adds it.
Three negative greps cover the three surfaces where it could leak:
- plugins/ruflo-metaharness/skills/*/SKILL.md (skill-layer leak)
- plugins/ruflo-metaharness/scripts/*.mjs (script-layer leak)
- v3/@claude-flow/cli/src/mcp-tools/ (MCP-tool-layer leak)
GUARD 2 — ADR-150 architectural constraint #1: no static @metaharness/*
imports outside neural-router.ts
Ruflo's architectural-constraint #1 says: removing all @metaharness/*
packages must still produce a working CLI. The ONE allowed
exception is `neural-router.ts` using a dynamic import (triple-gated
per ADR-148). Any OTHER static import would tether ruflo to the
optional dep.
The static-grep counts non-allowed static imports across the CLI
src tree. Today: 0. Any future iter that accidentally writes
`import { Router } from '@metaharness/router'` in a non-router file
trips the guard.
The `// allowed:` opt-out marker lets a deliberate static import
(with explicit justification) bypass the guard while keeping the
ratchet tight against accidents.
GUARD 3 — confirmation iter-73's mint-anti-MCP check remains in place
Inlined again in step 17z37 so the three guards live in one place,
easier for future iters to audit collectively.
WHY NEGATIVE GUARDS MATTER (collective rationale)
The 3 invariants:
- No from-repo skill/script/MCP tool
- No new static @metaharness/* imports outside neural-router.ts
- No metaharness_mint MCP tool
These are all "thing X must not exist". Normal positive testing can't
catch the violation because no existing test would FAIL — the new
code would simply ADD a passing test alongside the existing suite.
Only a negative grep catches the addition.
Three negative guards is now a documented pattern: a single smoke
step that codifies multiple ADR architectural invariants in one
place, so future maintainers see them grouped.
SMOKE STEP 17z37 anchors:
- All 3 from-repo surface negatives
- Static-import count == 0
- Mint MCP-tool negative
GATES THIS ITER
smoke: 76 → 77 (step 17z37 green; all guards pass)
no surface changes — pure anti-regression infrastructure
ITER 73-74 NEGATIVE-GUARD ARC
73 — mint anti-MCP (one invariant)
74 — from-repo (3 surface checks) + static-import-leak + mint ◀ this iter
Pattern is now self-documenting: any future "thing X must not exist"
constraint should land here as another negative grep in 17z37 (or a
sibling step). The grep cost is microseconds; the architectural
guarantee compounds.
Co-Authored-By: RuFlo <ruv@ruv.net>
|
||
|
ruv
|
96b436706f |
fix(metaharness): current subcommand list + mint anti-MCP guard (iter 73)
Two related correctness fixes.
PROBLEM A — STALE SUBCOMMAND DESCRIPTION
`v3/@claude-flow/cli/src/commands/metaharness.ts` exposed `options[]`
with a description string that listed only the iter-3 set:
"One of: score | genome | mcp-scan | threat-model | mint"
5 subcommands were missing from the description: oia-audit (iter 7),
audit-list (iter 16), audit-trend (iter 15), similarity (iter 36),
drift-from-history (iter 53). The help text would surface those via
`metaharness help`, but `metaharness --help` (the auto-generated
option-list output) still claimed only 5.
PROBLEM B — NO ANTI-MINT-MCP GUARD
ADR-150 §Sandboxing explicitly says mint must NEVER be exposed via
MCP because it writes to the filesystem with explicit --confirm. The
9 metaharness MCP tools deliberately omit metaharness_mint.
But there's no test enforcing the omission. A future iter that mass-
generates MCP tools from the SUBCOMMANDS map (a tempting refactor)
would silently include metaharness_mint and violate the sandboxing
rule. The agent-callable mint would let LLMs scaffold harnesses
without the human confirmation gate.
WHAT SHIPPED
1. v3/@claude-flow/cli/src/commands/metaharness.ts — description
string now lists all 10 dispatchable subcommands, with a comment
reminding maintainers to keep it in sync with the SUBCOMMANDS map.
2. plugins/ruflo-metaharness/scripts/smoke.sh — step 17z36:
- Positive: all 10 subcommand names appear in metaharness.ts
- NEGATIVE (the load-bearing guard): grep returns no match for
`name: 'metaharness_mint'` in metaharness-tools.ts. If it ever
does, miss="mint-leaked-to-mcp" and smoke fails.
WHY THE NEGATIVE GREP MATTERS
This is the kind of gap that doesn't surface in normal testing — no
existing test exercises mint via MCP because it isn't supposed to
exist. A regression that adds it would simply ADD a passing test
case alongside the 9 existing ones. The negative grep is the only
shape of check that catches this class of bug.
GATES THIS ITER
smoke: 75 → 76 (step 17z36 green)
tsc: clean rebuild of metaharness.ts
--help: now lists all 10 subcommands accurately
ITER 53-73 MCP-CLI SAFETY MATRIX
CLI subcommands: 10 (score, genome, mcp-scan, threat-model,
oia-audit, audit-list, audit-trend,
similarity, drift-from-history, mint)
MCP tools: 9 (= 10 - mint per ADR-150 §Sandboxing)
Help text: CURRENT (was iter-3 stale)
Anti-MCP guard: PRESENT (was: implicit, now: smoke-asserted)
The 10:9 asymmetry between CLI and MCP is now contract-anchored.
Co-Authored-By: RuFlo <ruv@ruv.net>
|