mirror of
https://github.com/proffesor-for-testing/agentic-qe.git
synced 2026-09-19 08:45:47 +08:00
Dragan Spiridonov
cda9fb3c05
Single commit covering the 5 concrete findings from the qe-devils-advocate review. Does not change the underlying CommandEvalRunner design or the 5 primitives — only the claims around them and the fixture-server hack the yaml had silently inherited. 1. Revert promotion (G4): trust_tier 3 -> 2 in SKILL.md; ADR-091 status Implemented -> In Progress (2026-04-21); v3-adrs.md index row updated to reflect the same; counts 80->79 Implemented, 6->7 In Progress. The promotion gate is now documented on ADR-091: first green CI run of test-qe-browser.yml (unit + smoke + eval against real Vibium) before trust_tier 3 / Implemented are restored. 2. Fix yaml fixtures (G3): tc006/tc007 were referencing http://localhost:8088/qe-browser/SKILL.md.html which required an unstarted fixture server, so `aqe eval run --skill qe-browser` would have failed on a fresh machine. Rewrote both to use https://httpbin.org/html (matching scripts/smoke-test.sh tc006/tc007). Dropped tc009 (poisoned-page check-injection) from the yaml — it needs the local serve-skills.js fixture which is out of scope for CommandEvalRunner. Documented as an explicit GAP comment in the yaml, noted that check-injection.js's severity path is unit-tested at tests/unit/scripts/qe-browser-check-injection.test.ts. yaml test_cases now 11 -> 10. 3. Correct pin comment wording (G1): the previous comment claimed ^26.3.18 was a "major.minor-line pin" — that's factually wrong under npm semver (^26.3.18 accepts 26.99.x). Rewritten to accurately describe the intent: major-line pin that blocks 27.0+ while allowing auto-uptake of 26.x security patches and additive features, with scripts/smoke-test.sh as the belt-and-suspenders API contract gate. The spec itself stays ^26.3.18 — the correct choice for an end-user tool under semver assumptions. 4. Add eval CI job (G2): third job in test-qe-browser.yml runs `aqe eval run --skill qe-browser --model claude-3.5-sonnet` against a real Vibium install after `npm run build`. This is the end-to-end proof that CommandEvalRunner dispatch works that the previous unit tests (all mocked CommandRunner) could not provide. Uploads the eval JSON as a CI artifact for post-run inspection. actionlint clean. 5. Scrub stray files (G5): git rm the coherence-comparison artifacts from docs/reports/ that were swept in by a prior `git add -A`, and added a focused pattern to .gitignore (not a blanket docs/reports/ ignore) so future sessions can't repeat the mistake while intentional reports can still be committed. Tests: 51 relevant tests still pass (3 files: command-eval-runner, command-eval-runner-qe-browser, browser-engine-installer). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>