Commit Graph

5 Commits

Author SHA1 Message Date
Dragan Spiridonov 1aac5206ba fix(adr-093): address devil's-advocate C1-C4 critical findings
Four merge-blocking issues surfaced by the qe-devils-advocate audit.

C1 — effort-resolver YAML loading was silently broken in ESM
  src/shared/llm/effort-resolver.ts: replaced
  const { readFileSync } = require('fs') with a top-level
  import { readFileSync } from 'fs'. The require() call threw
  ReferenceError under ESM ("type": "module" in package.json),
  caught by the bare try/catch, so the YAML priority tier in the
  resolver chain was dead — callers always fell through to the
  hardcoded xhigh default. config/fleet-defaults.yaml was a
  decorative file. Proven fixed by runtime check:
    YAML set to 'medium' → resolveEffortLevel() now returns 'medium'.

C2 — body.effort on every /v1/messages call was unverified shape
  src/shared/llm/providers/claude.ts: the previous code
  unconditionally set body.effort on every request, regardless of
  whether the target model supported it, and used a top-level shape
  that is not documented in Anthropic's public Messages API schema.
  New behavior: effort is only set when the registry marks the
  model supportsEffortXHigh=true (Opus 4.7 today) and is nested
  under body.thinking = { type: 'adaptive', effort: <level> },
  matching the Anthropic SDK's beta.messages thinking shape.
  Non-xhigh models receive no thinking/effort field at all,
  eliminating the risk of 400-ing Sonnet 4.6 / Haiku 4.5 calls.
  Rationale inline at the call site.

C3 — CYBER_PINNED_AGENTS was missing qe-security-reviewer
  qe-security-reviewer lives at .claude/agents/v3/subagents/
  (not in .claude/agents/v3/ top level) and was present in the
  submitted Cyber Verification application. Without it in the pin
  list, that agent could route to Opus 4.7 freely. Added.
  Cross-check test now parses the application file and asserts
  every CYBER_PINNED_AGENTS entry is named in it — future edits
  to either side cannot silently diverge.

C4 — cyber pin only fired in MultiModelExecutor.consult()
  Direct HybridRouter.chat() calls from security agents bypassed
  the pin. Extracted the pin helpers into src/routing/security/
  cyber-pin.ts (applyCyberPin, shouldCyberPin, isOpus47,
  CYBER_PINNED_AGENTS, CYBER_PIN_ADVISOR_FALLBACK,
  CYBER_PIN_CHAT_FALLBACK). HybridRouter.chat() now applies the
  pin post-selectProvider, before executeWithFallback, on both
  decision.model and decision.providerModelId — covering
  canonical, OpenRouter (dotted), and Bedrock ARN forms.
  MultiModelExecutor.consult() imports from the same module so
  both paths share one source of truth.

Tests moved tests/routing/advisor/cyber-pin.test.ts →
tests/routing/security/cyber-pin.test.ts and expanded: 19 tests
covering all 4 pinned agents × multiple model-id forms, the
application cross-check, env-flag semantics, non-security
passthrough, and non-4.7 model passthrough.

Verification:
* 0 tsc errors
* 128/128 targeted tests pass (effort-resolver + cyber-pin + advisor + providers)
* 2927/2927 broader regression pass (shared/llm + coordination + routing)
* Runtime proof of C1 fix: YAML 'medium' → resolveEffortLevel() returns 'medium'

Still outstanding from the devil's-advocate audit (deferred to
follow-up, not merge-blocking):
* C5 — Bedrock/OpenRouter model IDs need empirical verification
* C6 — no integration test hits real /v1/messages with 4.7
* H1–H5, M1–M5, L1–L3

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 16:31:24 +00:00
Dragan Spiridonov 4a641e7ef1 fix(adr-093): post-audit sweep — CI workflow, consensus provider, MCP sampling, 12 skill evals, 9 test files
Fifth post-audit sweep. 24 files, +66/-60.

CRITICAL (user-facing):
* .github/workflows/skill-validation.yml:422: CI eval runner
  --model claude-sonnet-4 → claude-sonnet-4-6. Previously CI would
  target a retiring model.
* src/coordination/consensus/providers/claude-provider.ts: Claude 3.x
  consensus provider modernized — type union extended with
  claude-sonnet-4-6 / claude-opus-4-7 / claude-haiku-4-5 as first
  entries, default model updated from claude-3-5-sonnet-20241022
  to claude-sonnet-4-6, availableModels list prepends new models.
  Legacy 3.x entries retained for backward compat.
* src/mcp/security/sampling-server.ts: default model
  claude-3-sonnet-20240229 (retired 2026-02-19) → claude-sonnet-4-6.

Skill evals (12 files, 6 + 6 mirrors):
* .claude/skills/ and assets/skills/ mirrors for:
  performance-testing, chaos-engineering-resilience, quality-metrics,
  localization-testing, accessibility-testing,
  n8n-trigger-testing-strategies. models_to_test list updated to
  claude-sonnet-4-6 / claude-haiku-4-5.

Test files (9):
* tests/unit/coordination/consensus/providers/claude-provider.test.ts:
  default model, test descriptions, and references updated to 4.6/4.7.
* tests/unit/routing/economic-routing.test.ts: claude-3-opus-20240229
  usage → claude-opus-4-7.
* tests/unit/learning/skill-validation-learner.test.ts: claude-3-haiku
  / claude-3.5-sonnet / claude-3-opus → new canonicals.
* tests/unit/mcp/handlers/handler-test-utils.ts: claude-3-haiku →
  claude-haiku-4-5.
* tests/integration/llm/{real-provider-integration,multi-provider}.test.ts:
  bare 'claude-sonnet-4' canonical → 'claude-sonnet-4-6'.
* tests/integration/validation/parallel-eval-runner.test.ts:
  claude-3.5-sonnet / claude-3-haiku → new canonicals.
* tests/routing/advisor/multi-model-executor.test.ts:
  CRITICAL BUG — 'claude-opus-4-6' (non-existent model ID, typo) ×2
  → 'claude-opus-4-7'. Stale test name 'anthropic/claude-opus-4
  when no provider' → '-4.7'.
* tests/unit/opencode/degradation.test.ts: CRITICAL BUG —
  modelId 'claude-opus-4-6' (typo) → 'claude-opus-4-7'.

Left intentionally untouched:
* tests/unit/coordination/consensus/providers/openrouter-provider.test.ts:
  tests that OpenRouter provider lists claude-3-haiku / claude-3.5-sonnet
  / claude-3-opus as supported. These assertions reflect actual
  OpenRouter provider data, which hasn't changed. Adding new-model
  assertions would be an enhancement, not a fix.

Verification:
* 0 tsc errors
* 174/174 tests across touched paths pass

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 14:26:23 +00:00
Dragan Spiridonov 6010017661 feat(adr-093): pin cyber-sensitive agents to Sonnet 4.6 pending approval
Commit 6 of 6 for ADR-093 Opus 4.7 migration. Opt-out gate behind
AQE_CYBER_VERIFIED env var.

* src/routing/advisor/multi-model-executor.ts: new applyCyberPin()
  helper + CYBER_PINNED_AGENTS list (qe-pentest-validator,
  qe-security-auditor, qe-security-scanner). When the advisor model
  resolves to Opus 4.7 for any of these agents, the pin downgrades
  to anthropic/claude-sonnet-4.6 and logs a warning naming the env
  var to flip after Cyber Verification Program approval lands.
  Non-security agents pass through unchanged. Non-4.7 models pass
  through unchanged. Only AQE_CYBER_VERIFIED === 'true' lifts the
  pin — '1', 'yes', truthy-but-non-"true" values keep the pin.
* tests/routing/advisor/cyber-pin.test.ts (new): 10 tests covering
  all 3 pinned agents with pin-on and pin-off, non-security passthrough,
  non-Opus-4.7 passthrough, canonical claude-opus-4-7 form, and
  strict "true" semantics.

Fleet inventory note: 3 qe-* security/pentest agents exist. The
Cyber Verification application draft listed 4 (including a
qe-security-reviewer that does not exist in .claude/agents/v3/).

Verification:
* 0 tsc errors
* 75/75 advisor tests pass (+10 new)
* 16555/16564 unit + routing tests pass (9 pre-existing skips)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 12:57:41 +00:00
Dragan Spiridonov b055560e81 feat(adr-093): set Opus 4.7 as default ADR-092 advisor model
Commit 5 of 6 for ADR-093 (Commit 4 skipped — no thinking-config
callers to shim; documented in the implementation plan).

* src/routing/advisor/multi-model-executor.ts: DEFAULT_ADVISOR_MODEL
  upgraded from 'anthropic/claude-opus-4' to 'anthropic/claude-opus-4.7'.
  When TinyDancerRouter sets triggerMultiModel=true (confidence < 0.80,
  or < 0.85 for security tasks), MultiModelExecutor.consult() now
  dispatches to Opus 4.7 by default via HybridRouter.chat() +
  OpenRouter provider.
* tests/routing/advisor/multi-model-executor.test.ts,
  tiny-dancer-router-advisor.test.ts: update expected provider model IDs.

SWE-bench Verified 87.6% (vs 80.8% Opus 4.6), SWE-bench Pro 64.3%
(vs 53.4%), CursorBench 70% (vs 58%). Vendor-independence preserved —
consultations still flow through HybridRouter's 7 providers per
ADR-043.

Verification:
* 0 tsc errors
* 65/65 advisor unit tests pass

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 12:46:42 +00:00
Dragan Spiridonov 24e30e0584 feat(advisor): implement ADR-092 provider-agnostic advisor strategy
Multi-model routing layer with circuit breaking, PII redaction, and
domain-specific prompts for the Queen coordinator.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 07:40:20 +00:00