mirror of
https://github.com/proffesor-for-testing/agentic-qe.git
synced 2026-09-19 08:45:47 +08:00
Profa
2253d05618
Remove the nested v3/ subdirectory by moving all active source code, tests, configs, and assets directly to the project root. Archive v2/ (dead code, 847MB) to v2-archive.tar.gz. Key changes: - Move v3/src/ → src/, v3/tests/ → tests/, v3/packages/ → packages/ - Move v3/tsconfig.json, v3/vitest.config.ts to root - Merge v3/package.json into root package.json (ESM, scripts, deps, exports) - Merge v3/scripts/ into scripts/, v3/implementation/ into docs/implementation/ - Update all CI workflows to remove "cd v3 &&" and v3/ path prefixes - Fix all init installer path resolution (3-level-up → 2-level-up) - Fix hooks to use "node ./dist/cli/bundle.js" instead of bare "aqe" - Update settings.json ADR/DDD directories to docs/implementation/ - Fix helper scripts (statusline, adr-compliance, ddd-tracker, guidance-hooks) - Update 58 agent definitions: aqe/v3/ → aqe/ memory namespaces - Update 15 guidance shards: v3/src/ → src/ path references - Fix sync interfaces: remove v3/ and v2/ path references - Fix test path calculations (process.cwd() + '..' no longer needed) - Fix flaky perf test threshold (500ms → 1000ms for CI environments) - Fix complexity-analyzer test to match ADR-051 keyword-based eligibility Build passes, 17,901 tests pass, CLI and MCP bundles verified. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
140 lines
3.8 KiB
Handlebars
140 lines
3.8 KiB
Handlebars
# Validation Report
|
|
|
|
> {{statusEmoji}} Generated: {{timestamp}}
|
|
> Run ID: {{runId}}
|
|
> Policy: ADR-056 Trust But Verify
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
| Metric | Value |
|
|
|--------|-------|
|
|
| Total Skills | {{summary.totalSkills}} |
|
|
| Passed Skills | {{summary.passedSkills}} ({{passedPercent}}%) |
|
|
| Failed Skills | {{summary.failedSkills}} |
|
|
| Average Pass Rate | {{avgPassRatePercent}}% |
|
|
| Total Tests | {{summary.totalTests}} |
|
|
| Passed Tests | {{summary.passedTests}} |
|
|
| Failed Tests | {{summary.failedTests}} |
|
|
| Total Duration | {{durationSeconds}}s |
|
|
| Models Used | {{#each summary.modelsUsed}}{{this}}{{#unless @last}}, {{/unless}}{{/each}} |
|
|
|
|
---
|
|
|
|
## Recommendations
|
|
|
|
{{#each recommendations}}
|
|
{{add @index 1}}. {{this}}
|
|
{{/each}}
|
|
|
|
---
|
|
|
|
## Cross-Model Analysis
|
|
|
|
**Variance:** {{variancePercent}}%
|
|
**Standard Deviation:** {{stdDeviationPercent}}%
|
|
|
|
### Model Performance
|
|
|
|
| Model | Avg Pass Rate | Skills Tested | Total Tests |
|
|
|-------|---------------|---------------|-------------|
|
|
{{#each modelPerformance}}
|
|
| {{@key}} | {{passRatePercent this.avgPassRate}}% | {{this.skillCount}} | {{this.totalTests}} |
|
|
{{/each}}
|
|
|
|
{{#if crossModelAnalysis.anomalies.length}}
|
|
### Anomalies Detected
|
|
|
|
| Model | Type | Description | Deviation |
|
|
|-------|------|-------------|-----------|
|
|
{{#each crossModelAnalysis.anomalies}}
|
|
| {{this.model}} | {{this.type}} | {{this.description}} | {{deviationPercent this.deviation}}% |
|
|
{{/each}}
|
|
{{/if}}
|
|
|
|
{{#if crossModelAnalysis.inconsistentSkills.length}}
|
|
### Inconsistent Skills
|
|
|
|
These skills show high variance across models:
|
|
|
|
{{#each crossModelAnalysis.inconsistentSkills}}
|
|
- {{this}}
|
|
{{/each}}
|
|
{{/if}}
|
|
|
|
{{#if regressions.length}}
|
|
---
|
|
|
|
## Regressions Detected
|
|
|
|
| Skill | Model | Previous | Current | Drop | Severity |
|
|
|-------|-------|----------|---------|------|----------|
|
|
{{#each regressions}}
|
|
| {{this.skill}} | {{this.model}} | {{passRatePercent this.previousPassRate}}% | {{passRatePercent this.currentPassRate}}% | {{passRatePercent this.regressionAmount}}% | {{this.severity}} |
|
|
{{/each}}
|
|
|
|
{{#each criticalRegressions}}
|
|
### {{this.skill}} ({{this.model}})
|
|
|
|
**Severity:** {{upper this.severity}}
|
|
**Trend:** {{this.trend}}
|
|
|
|
**Possible Causes:**
|
|
{{#each this.possibleCauses}}
|
|
- {{this}}
|
|
{{/each}}
|
|
|
|
{{/each}}
|
|
{{/if}}
|
|
|
|
---
|
|
|
|
## Skill Results
|
|
|
|
### Passing Skills ({{summary.passedSkills}})
|
|
|
|
| Skill | Trust Tier | Pass Rate | Tests | Issues |
|
|
|-------|------------|-----------|-------|--------|
|
|
{{#each passingSkills}}
|
|
| {{this.skill}} | T{{this.trustTier}} | {{passRatePercent this.avgPassRate}}% | {{this.passedTests}}/{{this.testCount}} | {{this.issues.length}} |
|
|
{{/each}}
|
|
|
|
{{#if failingSkills.length}}
|
|
### Failing Skills ({{summary.failedSkills}})
|
|
|
|
| Skill | Trust Tier | Pass Rate | Tests | Critical Issues |
|
|
|-------|------------|-----------|-------|-----------------|
|
|
{{#each failingSkills}}
|
|
| {{this.skill}} | T{{this.trustTier}} | {{passRatePercent this.avgPassRate}}% | {{this.passedTests}}/{{this.testCount}} | {{criticalIssueCount this.issues}} |
|
|
{{/each}}
|
|
|
|
{{#each topFailingSkills}}
|
|
#### {{this.skill}} Issues
|
|
|
|
{{#each (take this.issues 5)}}
|
|
- **[{{this.severity}}]** {{this.type}}: {{this.message}}{{#if this.testId}} ({{this.testId}}){{/if}}
|
|
{{/each}}
|
|
{{#if (gt this.issues.length 5)}}
|
|
- ... and {{subtract this.issues.length 5}} more issues
|
|
{{/if}}
|
|
|
|
{{/each}}
|
|
{{/if}}
|
|
|
|
---
|
|
|
|
## Metadata
|
|
|
|
- **Version:** {{metadata.version}}
|
|
- **Generated By:** {{metadata.generatedBy}}
|
|
- **Input Runs:** {{#each metadata.inputs.runIds}}{{this}}{{#unless @last}}, {{/unless}}{{/each}}
|
|
- **Models Tested:** {{#each metadata.inputs.models}}{{this}}{{#unless @last}}, {{/unless}}{{/each}}
|
|
{{#if metadata.environment}}
|
|
- **Environment:** {{metadata.environment}}
|
|
{{/if}}
|
|
|
|
---
|
|
|
|
*Generated by ValidationResultAggregator per ADR-056*
|