Files
ruvnet__ruflo/v3/@claude-flow/cli/__tests__/agenticow-tools.test.ts
rUv d759af6d6b feat(agenticow): COW memory substrate + governed distillation loop (ADR-170..173) (#2562)
* feat(agenticow): complete verb surface (ingest/query/diff/lineage/status) + nativeAnn + 0.2.4

Step 1 of the agenticow integration: the lifecycle verbs (branch/
checkpoint/rollback/promote) existed but the read/write verbs that make
a branch usable did not — you could create a COW branch via MCP but
couldn't populate or read it. Adds:
- agenticow_ingest — write {id?,vector,text?} records into a branch/base
- agenticow_query — k-NN across the full COW lineage (parent ∪ edits)
- agenticow_diff / _lineage / _status — introspection
- nativeAnn option on agenticow_branch (Rust dual-graph ANN fast path)
- targeted rollback via optional checkpointId

Floor bumped agenticow ~0.2.3 → ~0.2.4 (the upstream fix — text payloads
now survive the save/load round-trip our wrapper does on every call;
ruvnet/agenticow#3, published 0.2.4). 9 tools total; tests 10/10.

Co-Authored-By: RuFlo <ruv@ruv.net>

* feat(swarm): per-agent COW memory branches via agenticow (agenticow step 2)

Wire the agenticow COW-memory primitive into the swarm agent lifecycle so a
swarm agent can own an isolated ~162-byte Copy-On-Write branch of a shared
.rvf base instead of a full copy — the structural fix for the v3.14.4
worktree-bloat (3.3 GB from full-copy per-agent snapshots).

SwarmMemoryBranches service (src/services/swarm-memory-branches.ts):
  - branchForAgent(base, agentId): fork a 162-byte COW branch (nativeAnn),
    idempotent, persisted to .claude-flow/swarm/cow-branches.json
  - promoteAgent(agentId): merge the agent's edits back into base, then delete
    the branch (call on success)
  - discardAgent(agentId): drop the branch, pure-fs, works even degraded
    (call on failure)
  - Non-fatal + CLAUDE_FLOW_NO_COW_MEMORY kill switch; agenticow lazy-loaded so
    it stays off the CLI startup path (--help unchanged at ~0.09s).

Seam + wiring (src/mcp-tools/agent-tools.ts):
  agent_spawn stores agents as pure JSON metadata today — it never forks/copies
  an .rvf, so there is no full-copy to replace inline. Wired in opt-in: pass
  `memoryBase` to agent_spawn to fork a per-agent branch (recorded on the
  AgentRecord); agent_terminate promotes (promoteMemory:true) or discards it.
  Default behavior is unchanged.

Shared loader (src/mcp-tools/agenticow-loader.ts): extracted loadAgenticow /
resolveMemoryPath / manifestFor / validateLabel / openWithLineage from
agenticow-tools.ts so the MCP verbs and the service share one optional-dep
dance (behavior-preserving; existing agenticow-tools tests unchanged).

Tests (__tests__/swarm-memory-branches.test.ts): kill-switch + degraded
no-op contract (always-on) and a real branch -> ingest -> promote/discard
lifecycle against a temp .rvf (skipIf agenticow absent), following the
agenticow-tools.test.ts conventions.

Co-Authored-By: RuFlo <ruv@ruv.net>

* feat(agenticow): speculative branch-and-promote for A/B memory exploration (step 4)

Adds a self-contained SpeculativeExploration module + `agenticow_speculate`
MCP tool that composes on top of the existing agenticow COW verbs (fork /
promote). Fan out N candidate approaches, each on its own 162-byte COW branch
of a shared base .rvf; explore/score each independently; promote the winner's
branch into base and discard the losers by deleting their branch files. The
memory-state analogue of the worktree-per-agent pattern.

- src/agenticow/speculative-exploration.ts — generic `explore(base, {label,fn}[],
  score)` core + `exploreFromPath()` lifecycle wrapper. Returns
  {winner, scores, promoted, ...}.
- src/mcp-tools/_agenticow.ts — shared COW helpers (loadAgenticow / openWithLineage
  / manifestFor / resolveMemoryPath / validateLabel). ADDED alongside
  agenticow-tools.ts (kept byte-identical) to minimize merge conflict with a
  sibling branch.
- src/mcp-tools/agenticow-speculate-tools.ts — `agenticow_speculate` MCP tool
  (declarative candidates + probe-query / count scoring). Optional-dep graceful
  degradation per ADR-150.
- Registered in mcp-client.ts + mcp-tools/index.ts.
- __tests__/agenticow-speculate-tools.test.ts — 3-candidate deterministic-scorer
  test asserting winner promoted into base and loser branch files discarded
  (skipIf agenticow absent), following agenticow-tools.test.ts conventions.

Co-Authored-By: RuFlo <ruv@ruv.net>

* feat(autopilot): checkpoint/rollback gate for loop ticks (agenticow step 3)

Add CheckpointGate — an agenticow-backed checkpoint/rollback bracket for
autopilot loops and long-horizon workflows. Before a risky tick that mutates
.rvf memory, take an O(1) agenticow checkpoint (162 bytes); if the tick
regresses (throws, or the caller's verdict says the outcome is worse), roll
memory back to the checkpoint — O(edits-since-checkpoint), not an O(N) rebuild.

- New src/services/checkpoint-gate.ts: CheckpointGate.guard(memPath, label, fn)
  plus low-level checkpoint()/rollback(). Lazy-loads agenticow (optional dep,
  ADR-150) — zero startup cost, graceful degradation to unguarded pass-through
  when the package is absent, the CLAUDE_FLOW_AGENTICOW_DISABLE kill-switch is
  set, or no memory path is configured. Non-fatal: an agenticow failure never
  masks the tick's own result.
- Opt-in wiring into the autopilot loop (commands/autopilot.ts autopilotCheck):
  checkpoint the configured .rvf right before a re-engaged tick; roll back on
  the loop's regression signal (stall auto-disable). Activates only when
  CLAUDE_FLOW_AUTOPILOT_CHECKPOINT_MEM points at an .rvf the loop mutates.
  The loop's memory mutation is out-of-process, so this brackets across ticks
  rather than wrapping an in-process fn; guard() is available for callers that
  do have an in-process fn.
- Unit test (__tests__/checkpoint-gate.test.ts): 10 cases — real temp .rvf
  checkpoint→mutate→rollback restore, success-keeps, throw-rollback-rethrow,
  and degraded/kill-switch pass-through. skipIf(agenticow absent), following
  agenticow-tools.test.ts conventions.

Co-Authored-By: RuFlo <ruv@ruv.net>

* docs(adr): ADR-170..173 — agenticow substrate, provenance oracle, Fable harness, remote distillation

Records the architectural decisions of the agenticow + distillation
governance integration:
- ADR-170: agenticow COW memory as agent-scoped workspace substrate
  (9 verbs, per-agent branches, speculative, checkpoint gate, one
  shared optional-dep loader; honest 'no full-copy to replace' scope)
- ADR-171: provenance-tiered evaluation oracle (execution → fable →
  proxy, each label tagged; promote-gated on real clearance; causal
  failure receipts; never proxy-as-gold — ADR-169 applied to labels)
- ADR-172: cost-disciplined Fable advisor harness via claude -p
  (clean-cwd + batching = $1.56→~$0.02/item; judge + GEPA reflector)
- ADR-173: remote GPU distillation via weight-eft over SSH
  (dry-run default, --execute --yes gate, parameterized host, hard
  honesty rule — no false 'trains a model' claim)

Co-Authored-By: RuFlo <ruv@ruv.net>

* feat(neural): tiered `resolved` oracle for distill/weight-EFT SFT data

Replace the single structural-confidence proxy for the per-trajectory
`resolved` boolean with a 3-tier labeler where every label carries honest
provenance (ADR-169 — never present a proxy as ground truth).

- TIER 1 `oracle:test-exec` (distill-oracle.ts): real mechanical eval for
  trajectories with a SWE-bench FAIL_TO_PASS spec or a metaharness/darwin
  bench-suite case. Runs over SSH on a PARAMETERIZED remote (--remote / env
  RUFLO_DISTILL_REMOTE, never hard-coded). DRY-RUN by default: emits the
  ssh/darwin-bench/eval command plan + preflight and touches nothing; only
  execute:true runs the real eval, behind a wrapped non-fatal probe.
- TIER 2 `judge:fable` (fable-harness.ts): cost-disciplined headless Fable
  LLM-as-judge for trajectories with no mechanical spec. Runs `claude -p`
  from a FRESH EMPTY temp cwd (no CLAUDE.md — the 5x cost driver), carries
  the judge role via --append-system-prompt, BATCHES 20 items/call (~$0.02
  /item vs ~$1.56 from the project dir), and enforces --max-budget-usd.
  Off by default; second entry point reflectFailures() feeds GEPA/evolve.
- TIER 3 `proxy:structural`: existing output-verifier confidence, weakest,
  clearly labeled.

Interface consumed by distill:
  labelResolved(trajectories, opts) -> [{...traj, resolved, resolvedBy,
  resolvedConfidence?, resolvedReason?}]. Default (no opts) = dry-run oracle
  preflight + proxy fallback: ZERO spend, no SSH exec, no Fable call.
  Everything ADR-150 optional/graceful.

Tests mock `claude -p` (spawn) and the SSH exec — assert command
construction, batching, provenance tagging, tier fallback order, and the
zero-external-call default. One opt-in live smoke behind RUFLO_FABLE_LIVE=1.
37 passing, 1 skipped (live), $0.

Co-Authored-By: RuFlo <ruv@ruv.net>

* feat(neural): weight-eft $0 export slice + opt-in run-transcript capture path

Ship the ADR-150 weight-eft slice: turn ruflo run history into AUDITED
training DATA + a cost-Pareto measurement + a GPU plan. This does NOT train a
model and does NOT reduce escalation — weight-eft's own `train` never spawns
and no GPU tune has run. `resolved` is an explicitly-marked PROXY (ruflo has
no SWE-bench gold oracle).

Capture path (the blocker):
- NEW src/ruvector/run-transcript-recorder.ts — opt-in (CLAUDE_FLOW_RUN_TRANSCRIPTS=1,
  off by default; PII/retention surface mirroring the router trajectory recorder)
  recorder that persists full run records {instance_id, model, tier, resolved,
  resolved_source, messages[], model_patch} to .swarm/run-transcripts.jsonl.
- Seam wired in src/mcp-tools/agent-execute-core.ts next to the existing
  trajectory-outcome hook — the one place a run's transcript + model + tier +
  resolved-proxy are all known. resolved_source='api-success' (weakest proxy),
  model_patch='' (single-shot execute produces no diff) — both stamped honestly.
- router-trajectory.ts documents the companion split + adds unifiedRecorderStatus().

Distill command + service:
- NEW src/services/weight-eft.ts — optional-dep wrapper (dynamic import, graceful
  {degraded:true}; local type mirrors, no static @metaharness import). Archive
  builder (records → DarwinTrajectory[]), runExport/runPlan/runEval, plus a
  spend-gated remote-GPU train path (buildRemoteTrainInvocation + runRemoteTrain:
  DRY-RUN default, real compute only behind --execute --yes; ssh/rsync/ruvllm;
  host parameterized via --remote / $RUFLO_DISTILL_REMOTE, never hard-coded).
- `neural distill export|plan|eval|train` in neural.ts.
- @metaharness/weight-eft ~0.1.1 added to optionalDependencies.

Tests (18 in __tests__/weft-export.test.ts): archive-builder mapping, export E2E
(real weight-eft → sft.jsonl/dpo.jsonl/report), degraded path (absent dep →
{degraded:true}), remote-train command construction + dry-run/refused/preflight
gates, and capture-module opt-in gating. tsc build clean; CLI startup unaffected.

Co-Authored-By: RuFlo <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_01S7GYqnVUVxBfZ5W8znqry3

* merge: consolidate step-2/step-4 loaders into agenticow-loader.ts

* merge: step 3 (checkpoint gate) + consolidate 3rd loader into agenticow-loader.ts

All agenticow consumers (verbs, swarm branches, speculative, checkpoint
gate) now import one canonical loader — the first-class compat layer.
checkpoint-gate's inline copies removed; delegates to shared loadAgenticow.

* merge: oracle + weft + wire ADR-171 promote-gate & causal receipts (#23)

explore() and agenticow_speculate now enforce the ADR-171 promotion gate:
- a winning branch is promote-INELIGIBLE unless cleared by oracle:test-exec
  or an explicitly-accepted judge:fable — proxy:structural can NEVER clear,
  even when it claims cleared:true (fail-closed)
- requireClearance:true fail-closes without a gate — score alone never
  graduates work into base
- every discarded loser + an ineligible/failed winner emits a causal
  receipt {label, score, diff, outcome, provenance, reason}
- result carries promotedBy + promotionDecision for auditability

Back-compat: no clearance gate → legacy score-only promotion, honestly
tagged 'unverified' (never masquerading as ground truth). Tests 9/9.

Co-Authored-By: RuFlo <ruv@ruv.net>

* security(distill): bare dry-run is fully offline — gate ssh preflight behind --preflight

Adversarial RC finding: the remote-train dry-run ran a read-only 'ssh
host true' reachability probe, contacting the host without --execute.
Now the bare dry-run contacts NOTHING (prints commands only); the
read-only reachability/GPU probes require an explicit --preflight (or
--execute). Closes the 'no implicit remote execution' gate.

Co-Authored-By: RuFlo <ruv@ruv.net>

* fix(ci): regenerate v3/pnpm-lock.yaml for agenticow ^0.2.4 + @metaharness/weight-eft

Lock drifted from cli/package.json — had agenticow ~0.2.4 (pkg says
^0.2.4) and was MISSING @metaharness/weight-eft entirely, failing
--frozen-lockfile → cascading 34 CI job failures (the #2540→#2552
lesson). Regenerated; frozen-lockfile now clean.

Co-Authored-By: RuFlo <ruv@ruv.net>

* fix(ci): ADR-112 guidance clause on the 5 new agenticow verb descriptions

ingest/query/diff/lineage/status said 'Use to/after/before'; the ADR-112
audit requires the literal 'Use when' guidance signal. Reworded; audit
360/360 pass.

Co-Authored-By: RuFlo <ruv@ruv.net>

* fix(ci): register CLAUDE_FLOW_RUN_TRANSCRIPTS* as escape-hatch env vars (ADR-125/130)

The ADR-173 run-transcript capture path adds opt-in, off-by-default
background-recorder env toggles (PII/retention surface, mirroring the
existing CLAUDE_FLOW_ROUTER_TRAJECTORY* recorder). They have no
user-facing command, so they're env-only by design — registered as
known escape hatches like their router-trajectory analog. Audit passes.

Co-Authored-By: RuFlo <ruv@ruv.net>

* fix(ci): tilde-pin @metaharness/weight-eft + agenticow (anti-caret guard 17z73)

The metaharness smoke's iter-110 anti-caret regression guard requires all
@metaharness/* optional deps to be tilde-pinned (~) for reproducibility
(ADR-150 review-round-1). My earlier npm-lock revert accidentally flipped
@metaharness/weight-eft (and agenticow) to caret (^). Restored to tilde;
lock regenerated, frozen-lockfile clean.

Co-Authored-By: RuFlo <ruv@ruv.net>
2026-07-04 09:17:01 -04:00

203 lines
7.9 KiB
TypeScript
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
/**
* Tests for agenticow MCP tools.
*
* Two surfaces under test:
* 1. The HAPPY PATH — agenticow is installed (real package). The tool round-trips
* through actual COW branch / checkpoint / rollback / promote semantics on
* a temp .rvf file.
* 2. The DEGRADED PATH — agenticow is NOT installed (we vi.mock() it as
* ERR_MODULE_NOT_FOUND). Every tool must return `{degraded: true}` with no
* throw, matching the metaharness-tools / testgen-tools contract.
*
* Why both: agenticow is in `optionalDependencies`. CI runs with deps installed
* (happy path) but production users may not have it — the degraded path is
* the load-bearing architectural rule (zero hard runtime dep).
*/
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
import { mkdtempSync, rmSync, existsSync, statSync, readdirSync } from 'node:fs';
import { join } from 'node:path';
import { tmpdir } from 'node:os';
import { agenticowTools } from '../src/mcp-tools/agenticow-tools.js';
function findTool(name: string) {
const t = agenticowTools.find(t => t.name === name);
if (!t) throw new Error(`tool not found: ${name}`);
return t;
}
function dirBytes(p: string): number {
if (!existsSync(p)) return 0;
const st = statSync(p);
if (st.isFile()) return st.size;
let total = 0;
for (const e of readdirSync(p)) total += dirBytes(join(p, e));
return total;
}
// Detect agenticow availability at module scope (top-level await is allowed
// in vitest ESM modules). We need this before the `it.skipIf(...)` calls,
// which evaluate at definition time.
let havePkg = false;
try { await import('agenticow'); havePkg = true; } catch { havePkg = false; }
describe('agenticow MCP tools — happy path (real package)', () => {
let workdir: string;
let basePath: string;
beforeAll(() => {
workdir = mkdtempSync(join(tmpdir(), 'agenticow-tools-'));
basePath = join(workdir, 'base.rvf');
});
afterAll(() => {
try { rmSync(workdir, { recursive: true, force: true }); } catch { /* ignore */ }
});
it('exposes 9 tools (lifecycle + read/write verbs)', () => {
const names = agenticowTools.map(t => t.name).sort();
expect(names).toEqual([
'agenticow_branch',
'agenticow_checkpoint',
'agenticow_diff',
'agenticow_ingest',
'agenticow_lineage',
'agenticow_promote',
'agenticow_query',
'agenticow_rollback',
'agenticow_status',
]);
});
it.skipIf(!havePkg)('ingest → query round-trips vectors + text across a COW branch (issue: text persistence)', async () => {
const ingest = findTool('agenticow_ingest');
const query = findTool('agenticow_query');
const branch = findTool('agenticow_branch');
const v = (s: number) => Array.from({ length: 8 }, (_, i) => Math.sin(s + i));
const base = join(workdir, 'rw-base.rvf');
const br = join(workdir, 'rw-branch.rvf');
const ing = await ingest.handler({ path: base, records: [
{ id: 1, vector: v(1), text: 'alpha' }, { id: 2, vector: v(2), text: 'beta' },
], dimension: 8 }) as any;
expect(ing.success).toBe(true);
expect(ing.ingested.accepted).toBe(2);
// branch with the native ANN fast path, then ingest a branch-only vector
const b = await branch.handler({ basePath: base, branchPath: br, label: 'rw', nativeAnn: true }) as any;
expect(b.nativeAnn).toBe(true);
await ingest.handler({ path: br, records: [{ id: 3, vector: v(3), text: 'gamma' }] });
// query the branch: sees base ∪ branch, and text survives (0.2.4 fix)
const q = await query.handler({ path: br, vector: v(1), k: 3 }) as any;
expect(q.success).toBe(true);
const hit = q.hits.find((h: any) => h.id === 1);
expect(hit).toBeDefined();
expect(hit.text).toBe('alpha');
});
it.skipIf(!havePkg)('diff / lineage / status report branch state', async () => {
const ingest = findTool('agenticow_ingest');
const diff = findTool('agenticow_diff');
const lineage = findTool('agenticow_lineage');
const status = findTool('agenticow_status');
const v = (s: number) => Array.from({ length: 8 }, (_, i) => Math.cos(s + i));
const p = join(workdir, 'introspect.rvf');
await ingest.handler({ path: p, records: [{ id: 10, vector: v(1), text: 'x' }], dimension: 8 });
const d = await diff.handler({ path: p }) as any;
expect(d.success).toBe(true);
expect(Array.isArray(d.diff.added)).toBe(true);
const lin = await lineage.handler({ path: p }) as any;
expect(lin.success).toBe(true);
expect(lin.lineage.some((n: any) => n.role === 'working' || n.role === 'base')).toBe(true);
const st = await status.handler({ path: p }) as any;
expect(st.success).toBe(true);
expect(st.status.dimension).toBe(8);
});
it('new read/write verbs degrade + validate like the lifecycle verbs', async () => {
// path-traversal rejection is shared via resolveMemoryPath
const query = findTool('agenticow_query');
await expect(query.handler({ path: '../../etc/passwd', vector: [1, 2] })).rejects.toThrow(/disallowed/);
const ingest = findTool('agenticow_ingest');
await expect(ingest.handler({ path: 'ok.rvf', records: [] })).rejects.toThrow(/non-empty/);
});
it('every tool has a JSON-schema input + handler', () => {
for (const t of agenticowTools) {
expect(t.inputSchema.type).toBe('object');
expect(t.inputSchema.properties).toBeDefined();
expect(typeof t.handler).toBe('function');
}
});
it.skipIf(!havePkg)('agenticow_branch creates a tiny COW branch on a real base', async () => {
const branch = findTool('agenticow_branch');
// Use a small dim for test speed
const branchPath = join(workdir, 'b1.rvf');
const result: any = await branch.handler({
basePath, branchPath,
label: 'test-branch',
dimension: 32,
});
expect(result.success).toBe(true);
expect(result.degraded).toBeUndefined();
expect(result.label).toBe('test-branch');
expect(existsSync(branchPath)).toBe(true);
// The size win is the load-bearing claim — assert it's tiny
expect(dirBytes(branchPath)).toBeLessThan(2048);
});
it.skipIf(!havePkg)('agenticow_checkpoint persists state across reopen, agenticow_rollback restores it', async () => {
const cp = findTool('agenticow_checkpoint');
const rb = findTool('agenticow_rollback');
// checkpoint must persist via the .agenticow.json manifest. The base file
// was created earlier by the branch test (which writes the manifest).
const cpResult: any = await cp.handler({ path: basePath, label: 'cp-1' });
expect(cpResult.success).toBe(true);
expect(cpResult.label).toBe('cp-1');
// Manifest file should now exist next to the .rvf
expect(existsSync(`${basePath}.agenticow.json`)).toBe(true);
// Rollback now finds the checkpoint via the persisted manifest
const rbResult: any = await rb.handler({ path: basePath });
expect(rbResult.success).toBe(true);
expect(rbResult.rolledBack).toBe(true);
});
it('agenticow_branch rejects path traversal in basePath', async () => {
const branch = findTool('agenticow_branch');
await expect(branch.handler({
basePath: '../../../../etc/passwd',
branchPath: join(workdir, 'evil.rvf'),
label: 'evil',
dimension: 32,
})).rejects.toThrow(/disallowed characters/);
});
it('agenticow_branch rejects a malformed label', async () => {
const branch = findTool('agenticow_branch');
await expect(branch.handler({
basePath,
branchPath: join(workdir, 'b2.rvf'),
label: 'evil; rm -rf /',
dimension: 32,
})).rejects.toThrow(/may only contain/);
});
it('agenticow_branch requires dimension when creating a new memory file', async () => {
const branch = findTool('agenticow_branch');
const freshBase = join(workdir, 'nonexistent.rvf');
await expect(branch.handler({
basePath: freshBase,
branchPath: join(workdir, 'b3.rvf'),
label: 'fresh',
// no dimension
})).rejects.toThrow(/dimension is required/);
});
});