1 Commits

Author	SHA1	Message	Date
rUv	d759af6d6b	feat(agenticow): COW memory substrate + governed distillation loop (ADR-170..173) (#2562) ... * feat(agenticow): complete verb surface (ingest/query/diff/lineage/status) + nativeAnn + 0.2.4 Step 1 of the agenticow integration: the lifecycle verbs (branch/ checkpoint/rollback/promote) existed but the read/write verbs that make a branch usable did not — you could create a COW branch via MCP but couldn't populate or read it. Adds: - agenticow_ingest — write {id?,vector,text?} records into a branch/base - agenticow_query — k-NN across the full COW lineage (parent ∪ edits) - agenticow_diff / _lineage / _status — introspection - nativeAnn option on agenticow_branch (Rust dual-graph ANN fast path) - targeted rollback via optional checkpointId Floor bumped agenticow ~0.2.3 → ~0.2.4 (the upstream fix — text payloads now survive the save/load round-trip our wrapper does on every call; ruvnet/agenticow#3, published 0.2.4). 9 tools total; tests 10/10. Co-Authored-By: RuFlo <ruv@ruv.net> * feat(swarm): per-agent COW memory branches via agenticow (agenticow step 2) Wire the agenticow COW-memory primitive into the swarm agent lifecycle so a swarm agent can own an isolated ~162-byte Copy-On-Write branch of a shared .rvf base instead of a full copy — the structural fix for the v3.14.4 worktree-bloat (3.3 GB from full-copy per-agent snapshots). SwarmMemoryBranches service (src/services/swarm-memory-branches.ts): - branchForAgent(base, agentId): fork a 162-byte COW branch (nativeAnn), idempotent, persisted to .claude-flow/swarm/cow-branches.json - promoteAgent(agentId): merge the agent's edits back into base, then delete the branch (call on success) - discardAgent(agentId): drop the branch, pure-fs, works even degraded (call on failure) - Non-fatal + CLAUDE_FLOW_NO_COW_MEMORY kill switch; agenticow lazy-loaded so it stays off the CLI startup path (--help unchanged at ~0.09s). Seam + wiring (src/mcp-tools/agent-tools.ts): agent_spawn stores agents as pure JSON metadata today — it never forks/copies an .rvf, so there is no full-copy to replace inline. Wired in opt-in: pass `memoryBase` to agent_spawn to fork a per-agent branch (recorded on the AgentRecord); agent_terminate promotes (promoteMemory:true) or discards it. Default behavior is unchanged. Shared loader (src/mcp-tools/agenticow-loader.ts): extracted loadAgenticow / resolveMemoryPath / manifestFor / validateLabel / openWithLineage from agenticow-tools.ts so the MCP verbs and the service share one optional-dep dance (behavior-preserving; existing agenticow-tools tests unchanged). Tests (__tests__/swarm-memory-branches.test.ts): kill-switch + degraded no-op contract (always-on) and a real branch -> ingest -> promote/discard lifecycle against a temp .rvf (skipIf agenticow absent), following the agenticow-tools.test.ts conventions. Co-Authored-By: RuFlo <ruv@ruv.net> * feat(agenticow): speculative branch-and-promote for A/B memory exploration (step 4) Adds a self-contained SpeculativeExploration module + `agenticow_speculate` MCP tool that composes on top of the existing agenticow COW verbs (fork / promote). Fan out N candidate approaches, each on its own 162-byte COW branch of a shared base .rvf; explore/score each independently; promote the winner's branch into base and discard the losers by deleting their branch files. The memory-state analogue of the worktree-per-agent pattern. - src/agenticow/speculative-exploration.ts — generic `explore(base, {label,fn}[], score)` core + `exploreFromPath()` lifecycle wrapper. Returns {winner, scores, promoted, ...}. - src/mcp-tools/_agenticow.ts — shared COW helpers (loadAgenticow / openWithLineage / manifestFor / resolveMemoryPath / validateLabel). ADDED alongside agenticow-tools.ts (kept byte-identical) to minimize merge conflict with a sibling branch. - src/mcp-tools/agenticow-speculate-tools.ts — `agenticow_speculate` MCP tool (declarative candidates + probe-query / count scoring). Optional-dep graceful degradation per ADR-150. - Registered in mcp-client.ts + mcp-tools/index.ts. - __tests__/agenticow-speculate-tools.test.ts — 3-candidate deterministic-scorer test asserting winner promoted into base and loser branch files discarded (skipIf agenticow absent), following agenticow-tools.test.ts conventions. Co-Authored-By: RuFlo <ruv@ruv.net> * feat(autopilot): checkpoint/rollback gate for loop ticks (agenticow step 3) Add CheckpointGate — an agenticow-backed checkpoint/rollback bracket for autopilot loops and long-horizon workflows. Before a risky tick that mutates .rvf memory, take an O(1) agenticow checkpoint (162 bytes); if the tick regresses (throws, or the caller's verdict says the outcome is worse), roll memory back to the checkpoint — O(edits-since-checkpoint), not an O(N) rebuild. - New src/services/checkpoint-gate.ts: CheckpointGate.guard(memPath, label, fn) plus low-level checkpoint()/rollback(). Lazy-loads agenticow (optional dep, ADR-150) — zero startup cost, graceful degradation to unguarded pass-through when the package is absent, the CLAUDE_FLOW_AGENTICOW_DISABLE kill-switch is set, or no memory path is configured. Non-fatal: an agenticow failure never masks the tick's own result. - Opt-in wiring into the autopilot loop (commands/autopilot.ts autopilotCheck): checkpoint the configured .rvf right before a re-engaged tick; roll back on the loop's regression signal (stall auto-disable). Activates only when CLAUDE_FLOW_AUTOPILOT_CHECKPOINT_MEM points at an .rvf the loop mutates. The loop's memory mutation is out-of-process, so this brackets across ticks rather than wrapping an in-process fn; guard() is available for callers that do have an in-process fn. - Unit test (__tests__/checkpoint-gate.test.ts): 10 cases — real temp .rvf checkpoint→mutate→rollback restore, success-keeps, throw-rollback-rethrow, and degraded/kill-switch pass-through. skipIf(agenticow absent), following agenticow-tools.test.ts conventions. Co-Authored-By: RuFlo <ruv@ruv.net> * docs(adr): ADR-170..173 — agenticow substrate, provenance oracle, Fable harness, remote distillation Records the architectural decisions of the agenticow + distillation governance integration: - ADR-170: agenticow COW memory as agent-scoped workspace substrate (9 verbs, per-agent branches, speculative, checkpoint gate, one shared optional-dep loader; honest 'no full-copy to replace' scope) - ADR-171: provenance-tiered evaluation oracle (execution → fable → proxy, each label tagged; promote-gated on real clearance; causal failure receipts; never proxy-as-gold — ADR-169 applied to labels) - ADR-172: cost-disciplined Fable advisor harness via claude -p (clean-cwd + batching = $1.56→~$0.02/item; judge + GEPA reflector) - ADR-173: remote GPU distillation via weight-eft over SSH (dry-run default, --execute --yes gate, parameterized host, hard honesty rule — no false 'trains a model' claim) Co-Authored-By: RuFlo <ruv@ruv.net> * feat(neural): tiered `resolved` oracle for distill/weight-EFT SFT data Replace the single structural-confidence proxy for the per-trajectory `resolved` boolean with a 3-tier labeler where every label carries honest provenance (ADR-169 — never present a proxy as ground truth). - TIER 1 `oracle:test-exec` (distill-oracle.ts): real mechanical eval for trajectories with a SWE-bench FAIL_TO_PASS spec or a metaharness/darwin bench-suite case. Runs over SSH on a PARAMETERIZED remote (--remote / env RUFLO_DISTILL_REMOTE, never hard-coded). DRY-RUN by default: emits the ssh/darwin-bench/eval command plan + preflight and touches nothing; only execute:true runs the real eval, behind a wrapped non-fatal probe. - TIER 2 `judge:fable` (fable-harness.ts): cost-disciplined headless Fable LLM-as-judge for trajectories with no mechanical spec. Runs `claude -p` from a FRESH EMPTY temp cwd (no CLAUDE.md — the 5x cost driver), carries the judge role via --append-system-prompt, BATCHES 20 items/call (~$0.02 /item vs ~$1.56 from the project dir), and enforces --max-budget-usd. Off by default; second entry point reflectFailures() feeds GEPA/evolve. - TIER 3 `proxy:structural`: existing output-verifier confidence, weakest, clearly labeled. Interface consumed by distill: labelResolved(trajectories, opts) -> [{...traj, resolved, resolvedBy, resolvedConfidence?, resolvedReason?}]. Default (no opts) = dry-run oracle preflight + proxy fallback: ZERO spend, no SSH exec, no Fable call. Everything ADR-150 optional/graceful. Tests mock `claude -p` (spawn) and the SSH exec — assert command construction, batching, provenance tagging, tier fallback order, and the zero-external-call default. One opt-in live smoke behind RUFLO_FABLE_LIVE=1. 37 passing, 1 skipped (live), $0. Co-Authored-By: RuFlo <ruv@ruv.net> * feat(neural): weight-eft $0 export slice + opt-in run-transcript capture path Ship the ADR-150 weight-eft slice: turn ruflo run history into AUDITED training DATA + a cost-Pareto measurement + a GPU plan. This does NOT train a model and does NOT reduce escalation — weight-eft's own `train` never spawns and no GPU tune has run. `resolved` is an explicitly-marked PROXY (ruflo has no SWE-bench gold oracle). Capture path (the blocker): - NEW src/ruvector/run-transcript-recorder.ts — opt-in (CLAUDE_FLOW_RUN_TRANSCRIPTS=1, off by default; PII/retention surface mirroring the router trajectory recorder) recorder that persists full run records {instance_id, model, tier, resolved, resolved_source, messages[], model_patch} to .swarm/run-transcripts.jsonl. - Seam wired in src/mcp-tools/agent-execute-core.ts next to the existing trajectory-outcome hook — the one place a run's transcript + model + tier + resolved-proxy are all known. resolved_source='api-success' (weakest proxy), model_patch='' (single-shot execute produces no diff) — both stamped honestly. - router-trajectory.ts documents the companion split + adds unifiedRecorderStatus(). Distill command + service: - NEW src/services/weight-eft.ts — optional-dep wrapper (dynamic import, graceful {degraded:true}; local type mirrors, no static @metaharness import). Archive builder (records → DarwinTrajectory[]), runExport/runPlan/runEval, plus a spend-gated remote-GPU train path (buildRemoteTrainInvocation + runRemoteTrain: DRY-RUN default, real compute only behind --execute --yes; ssh/rsync/ruvllm; host parameterized via --remote / $RUFLO_DISTILL_REMOTE, never hard-coded). - `neural distill export|plan|eval|train` in neural.ts. - @metaharness/weight-eft ~0.1.1 added to optionalDependencies. Tests (18 in __tests__/weft-export.test.ts): archive-builder mapping, export E2E (real weight-eft → sft.jsonl/dpo.jsonl/report), degraded path (absent dep → {degraded:true}), remote-train command construction + dry-run/refused/preflight gates, and capture-module opt-in gating. tsc build clean; CLI startup unaffected. Co-Authored-By: RuFlo <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_01S7GYqnVUVxBfZ5W8znqry3 * merge: consolidate step-2/step-4 loaders into agenticow-loader.ts * merge: step 3 (checkpoint gate) + consolidate 3rd loader into agenticow-loader.ts All agenticow consumers (verbs, swarm branches, speculative, checkpoint gate) now import one canonical loader — the first-class compat layer. checkpoint-gate's inline copies removed; delegates to shared loadAgenticow. * merge: oracle + weft + wire ADR-171 promote-gate & causal receipts (#23) explore() and agenticow_speculate now enforce the ADR-171 promotion gate: - a winning branch is promote-INELIGIBLE unless cleared by oracle:test-exec or an explicitly-accepted judge:fable — proxy:structural can NEVER clear, even when it claims cleared:true (fail-closed) - requireClearance:true fail-closes without a gate — score alone never graduates work into base - every discarded loser + an ineligible/failed winner emits a causal receipt {label, score, diff, outcome, provenance, reason} - result carries promotedBy + promotionDecision for auditability Back-compat: no clearance gate → legacy score-only promotion, honestly tagged 'unverified' (never masquerading as ground truth). Tests 9/9. Co-Authored-By: RuFlo <ruv@ruv.net> * security(distill): bare dry-run is fully offline — gate ssh preflight behind --preflight Adversarial RC finding: the remote-train dry-run ran a read-only 'ssh host true' reachability probe, contacting the host without --execute. Now the bare dry-run contacts NOTHING (prints commands only); the read-only reachability/GPU probes require an explicit --preflight (or --execute). Closes the 'no implicit remote execution' gate. Co-Authored-By: RuFlo <ruv@ruv.net> * fix(ci): regenerate v3/pnpm-lock.yaml for agenticow ^0.2.4 + @metaharness/weight-eft Lock drifted from cli/package.json — had agenticow ~0.2.4 (pkg says ^0.2.4) and was MISSING @metaharness/weight-eft entirely, failing --frozen-lockfile → cascading 34 CI job failures (the #2540→#2552 lesson). Regenerated; frozen-lockfile now clean. Co-Authored-By: RuFlo <ruv@ruv.net> * fix(ci): ADR-112 guidance clause on the 5 new agenticow verb descriptions ingest/query/diff/lineage/status said 'Use to/after/before'; the ADR-112 audit requires the literal 'Use when' guidance signal. Reworded; audit 360/360 pass. Co-Authored-By: RuFlo <ruv@ruv.net> * fix(ci): register CLAUDE_FLOW_RUN_TRANSCRIPTS* as escape-hatch env vars (ADR-125/130) The ADR-173 run-transcript capture path adds opt-in, off-by-default background-recorder env toggles (PII/retention surface, mirroring the existing CLAUDE_FLOW_ROUTER_TRAJECTORY* recorder). They have no user-facing command, so they're env-only by design — registered as known escape hatches like their router-trajectory analog. Audit passes. Co-Authored-By: RuFlo <ruv@ruv.net> * fix(ci): tilde-pin @metaharness/weight-eft + agenticow (anti-caret guard 17z73) The metaharness smoke's iter-110 anti-caret regression guard requires all @metaharness/* optional deps to be tilde-pinned (~) for reproducibility (ADR-150 review-round-1). My earlier npm-lock revert accidentally flipped @metaharness/weight-eft (and agenticow) to caret (^). Restored to tilde; lock regenerated, frozen-lockfile clean. Co-Authored-By: RuFlo <ruv@ruv.net>	2026-07-04 09:17:01 -04:00