96 lines
4.8 KiB
Markdown
96 lines
4.8 KiB
Markdown
# Eval Trending — Metrics Over Time
|
||
|
||
Track evaluation metrics across multiple runs and versions to visualize improvement trends and detect regressions. This addresses the gap of understanding how agent quality changes over time.
|
||
|
||
## Prerequisites
|
||
|
||
- At least 2 evaluation runs in the same evaluation group (same `evaluationId` when created)
|
||
- Project endpoint and selected environment resolved from azd or the selected metadata overlay
|
||
|
||
> ⚠️ **Eval-group immutability:** Trend a group only when its evaluator set and thresholds stayed fixed across runs. If either changed, start a new evaluation group and track that history separately.
|
||
|
||
## Step 1 — Retrieve Evaluation History
|
||
|
||
Use **`evaluation_get`** to list all evaluation groups:
|
||
|
||
| Parameter | Required | Description |
|
||
|-----------|----------|-------------|
|
||
| `projectEndpoint` | ✅ | Azure AI Project endpoint |
|
||
| `isRequestForRuns` | | `false` (default) to list evaluation groups |
|
||
|
||
Then retrieve all runs within the target evaluation group:
|
||
|
||
| Parameter | Required | Description |
|
||
|-----------|----------|-------------|
|
||
| `projectEndpoint` | ✅ | Azure AI Project endpoint |
|
||
| `evalId` | ✅ | Evaluation group ID |
|
||
| `isRequestForRuns` | ✅ | `true` to list runs |
|
||
|
||
> ⚠️ **Parameter guardrail:** evaluation_get expects `evalId`, not `evaluationId`, even if the runs were grouped earlier with `evaluationId`.
|
||
|
||
## Step 2 — Build Metrics Timeline
|
||
|
||
For each run, extract per-evaluator scores and build a timeline:
|
||
|
||
| Run | Agent Version | Date | Coherence | Fluency | Relevance | Intent Resolution | Task Adherence | Safety |
|
||
|-----|--------------|------|-----------|---------|-----------|-------------------|----------------|--------|
|
||
| run-001 | v1 | 2025-01-15 | 3.2 | 4.1 | 2.8 | 3.0 | 2.5 | 0.95 |
|
||
| run-002 | v2 | 2025-01-22 | 3.8 | 4.3 | 3.5 | 3.7 | 3.2 | 0.97 |
|
||
| run-003 | v3 | 2025-02-01 | 4.1 | 4.4 | 4.0 | 4.2 | 3.8 | 0.96 |
|
||
| run-004 | v4 | 2025-02-08 | 4.0 | 4.5 | 3.6 | 4.1 | 3.9 | 0.98 |
|
||
|
||
## Step 3 — Trend Analysis
|
||
|
||
Calculate trends for each evaluator:
|
||
|
||
| Evaluator | v1 → v4 Change | Trend | Status |
|
||
|-----------|----------------|-------|--------|
|
||
| Coherence | +0.8 (+25%) | ↑ Improving | ✅ |
|
||
| Fluency | +0.4 (+10%) | ↑ Improving | ✅ |
|
||
| Relevance | +0.8 (+29%) | ↑ Improving (dip at v4) | ⚠️ |
|
||
| Intent Resolution | +1.1 (+37%) | ↑ Improving | ✅ |
|
||
| Task Adherence | +1.4 (+56%) | ↑ Improving | ✅ |
|
||
| Safety | +0.03 (+3%) | → Stable | ✅ |
|
||
|
||
### Detecting Regressions
|
||
|
||
Flag any evaluator where the latest run scored **lower** than the previous run:
|
||
|
||
| Evaluator | Previous (v3) | Latest (v4) | Delta | Alert |
|
||
|-----------|--------------|-------------|-------|-------|
|
||
| Relevance | 4.0 | 3.6 | -0.4 (-10%) | ⚠️ **REGRESSION** |
|
||
|
||
> ⚠️ **Regression detected:** Relevance dropped 10% from v3 to v4. Investigate prompt changes or dataset drift. See [Eval Regression](eval-regression.md) for automated analysis.
|
||
|
||
### Trend Visualization (Text-based)
|
||
|
||
```
|
||
Coherence ████████████████████████████████░░░░░░ 4.0/5.0 ↑ +25%
|
||
Fluency █████████████████████████████████████░░ 4.5/5.0 ↑ +10%
|
||
Relevance ████████████████████████████░░░░░░░░░░ 3.6/5.0 ↑ +29% ⚠️ dip
|
||
Intent Res. █████████████████████████████████░░░░░░ 4.1/5.0 ↑ +37%
|
||
Task Adh. ████████████████████████████████░░░░░░░ 3.9/5.0 ↑ +56%
|
||
Safety ████████████████████████████████████████ 0.98 → Stable
|
||
```
|
||
|
||
## Step 4 — Cross-Version Summary
|
||
|
||
Present an executive summary:
|
||
|
||
*"Over 4 agent versions (v1→v4), your agent has improved significantly across all quality metrics. The biggest gain is Task Adherence (+56%). However, Relevance showed a 10% regression from v3 to v4 — recommend investigating recent prompt changes. Safety remains stable at 98%."*
|
||
|
||
## Recommended Thresholds
|
||
|
||
| Severity | Threshold | Action |
|
||
|----------|-----------|--------|
|
||
| ✅ Healthy | ≤ 2% drop from previous run | No action needed |
|
||
| ⚠️ Warning | 2–5% drop from previous run | Review recent changes |
|
||
| 🔴 Regression | > 5% drop from previous run | Block deployment, investigate |
|
||
| 🔴 Critical | Below baseline (v1) on any metric | Rollback to last known good version |
|
||
|
||
## Next Steps
|
||
|
||
- **Investigate regression** → [Eval Regression](eval-regression.md)
|
||
- **Compare specific versions** → [Dataset Comparison](dataset-comparison.md)
|
||
- **Set up automated monitoring** → [observe skill CI/CD](../../observe/references/cicd-monitoring.md)
|