Dragan Spiridonov 2fe2300c68 feat(adr-091): make qe-browser.yaml a runnable eval + pin Vibium ...

Closes GAP-01, GAP-02, GAP-04 from the ADR-091 completion plan.

- Adds CommandEvalRunner (src/validation/command-eval-runner.ts) as a
  sibling to ParallelEvalRunner. Detects shell-based eval suites by
  shape (first test case has input.command), runs setup/main commands
  sequentially, and evaluates exit_code, json_fields (dotted JSONPath),
  severity_at_least (none < low < medium < high < critical), and
  candidate_count_at_least assertions against script JSON envelopes.
- Wires aqe eval run to dispatch to CommandEvalRunner when the suite is
  command-mode. LLM-prompt suites keep the pre-existing path unchanged.
- Activates .claude/skills/qe-browser/evals/qe-browser.yaml (status:
  active, was design-spec) and removes the honesty note now that the
  runner supports its structured assertions.
- Pins vibium@^26.3.18 via DEFAULT_VIBIUM_SPEC in browser-engine-
  installer.ts so npm install -g can't silently upgrade past a version
  we've verified against the 5 primitives + smoke tests.
- Drops qe-browser SKILL.md trust_tier from 3 to 2 pending eval
  integration in CI; will re-promote once the full verification pass
  runs green.

Tests: 31 new cases (pure helpers, runner integration with injected
runner, happy-path against the shipped qe-browser.yaml).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-21 12:12:40 +00:00

13 KiB

Raw Permalink Blame History

View Raw