mirror of
https://github.com/proffesor-for-testing/agentic-qe.git
synced 2026-09-19 08:45:47 +08:00
Dragan Spiridonov
70154c1b72
Closes the self-learning loop: the Q-value Bellman updates that #487 populates now have a routing-side reader; structurally-aware exploration (via the mincut infrastructure from ADR-047 / ADR-068) gathers Q-table signal beyond the deterministic greedy winner; the catalog diversifies without sacrificing safety on bottleneck decisions. Three signals blended at the agent-routing decision point: 1. Static score (existing): domain match + capability match + performance prior + language boost + pattern similarity. Computed by `calculateAgentScores` in src/learning/agent-routing.ts. 2. Q-value blend (NEW — reads from rl_q_values): effectiveScore = staticScore * (1 - qWeight) + sigmoid(qValue) * qWeight qWeight ramps from 0 (no Q-data) to MAX_Q_WEIGHT=0.4 over QWEIGHT_RAMP_VISITS=20 visits per (state_key, action_key). Cold-start behavior is identical to pre-ADR-095 static-only routing; the learned signal contributes meaningfully only after the Q-table has matured for that state. 3. Mincut-gated ε-greedy exploration (NEW): baseEpsilon = AQE_ROUTER_EXPLORATION_RATE env or 0.05 default safetyMultiplier = getSharedMinCutMonitor().isCritical() ? 0.2 : 1.0 epsilon = baseEpsilon * safetyMultiplier With probability ε, swap the greedy winner with a uniformly-random pick from the top-3 alternatives. Uses crypto.randomInt (NOT Math.random) to match the codebase convention (randomUUID is the established crypto entry-point in the same modules). Maintainer decisions captured in ADR-095 v2: 1. Default ON (Option A): mincut-modulated exploration ships enabled. AQE_ROUTER_EXPLORATION_RATE=0 is the documented escape hatch. 2. Telemetry surface: extended `aqe learning loop-health` with a new "Routing diversification" section showing 7-day exploit/explore counts, avg quality_score per bucket, avg criticality, avg q_weight. 3. crypto.randomInt over Math.random — codebase convention preserved. 4. Q-value ramp constants are best-guess (MAX_Q_WEIGHT=0.4, QWEIGHT_RAMP_VISITS=20); tune post-deploy from telemetry. Source-level findings worth noting: - `rl_q_values` was populated by `updateHookRouterQValue` since #487 but NEVER read during routing. Verified via grep across src/learning/, src/routing/, src/cli/commands/hooks-handlers/. This commit adds the reader; closes that loop. - Mincut graph is populated by Queen coordinator swarm activity, NOT per-task. So the "criticality" signal is graph-level safety (the `isCritical()` boolean from MinCutHealthMonitor) rather than per-task structural weight. Honest framing reflected in the ADR. What landed: src/kernel/unified-memory-schemas.ts - routing_outcomes gains exploration (INTEGER NOT NULL DEFAULT 0), criticality (REAL), q_weight (REAL) columns + an exploration index for the loop-health bucket queries src/routing/routing-outcomes-migration.ts (NEW) - ensureRoutingOutcomesAdr095Columns(db) — idempotent ALTER TABLE for pre-existing databases. Process-local flag avoids repeat work. src/routing/routing-feedback.ts - hooks the migration helper into the existing schema-migration block src/learning/agent-routing.ts - new helpers: blendStaticAndQValue, resolveExplorationRate, applyExplorationPolicy (uses crypto.randomInt), sigmoid normalization - new helpers: deriveTaskType, buildRoutingStateKey, deriveComplexityBucket — moved from task-hooks.ts so the routing path and the post-task Q-update path share the same state_key derivation - calculateAgentScores accepts optional qValueLookup; backward-compat when undefined (identical pre-ADR-095 behavior) - ScoredAgent gains optional telemetry fields: staticScore, qWeight, qValue, qVisits, exploration src/learning/qe-reasoning-bank-types.ts - QERoutingResult gains optional exploration / criticality / qWeight src/learning/qe-reasoning-bank.ts - routeTask builds the Q-value lookup from rl_q_values keyed by stateKey = deriveTaskType(task) | normal | domain | complexityBucket - applies exploration policy after sort; reports avgQWeight + the mincut safetyMultiplier in the QERoutingResult - new private buildQValueLookup helper with table-exists guard so the method is a graceful no-op when the schema isn't applied yet (tests, fresh-init scenarios) src/cli/commands/hooks-handlers/task-hooks.ts - private deriveTaskType removed (DRY: now imported from agent-routing) - pre-task sentinel INSERT writes the three new columns - migration helper invoked before write src/cli/commands/hooks-handlers/routing-hooks.ts - JSON output passes exploration / criticality / qWeight through - INSERT writes the three new columns; migration helper invoked src/cli/commands/learning.ts - `aqe learning loop-health` queries routing_outcomes for 7-day rolling exploration stats and avg quality_score per bucket. Skipped silently when the columns don't exist (pre-ADR-095 databases). Tests: - tests/unit/learning/agent-routing-exploration.test.ts: 35 tests cover sigmoid blend math, qWeight ramp, env override priority, mincut safety multiplier, exploration rate statistical tolerance, only-top-3 alternative constraint, deriveTaskType regex coverage, stateKey format, complexityBucket scaling, end-to-end Q-blend - tests/unit/routing/routing-outcomes-migration.test.ts: 5 tests cover fresh-DB migration, idempotent re-run, missing-table tolerance, exploration index creation, column type/default checks 634/634 tests pass across CLI commands, workers, bridge, learning, init/phases, architecture, kernel, and routing suites; build clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>