128 lines
4.0 KiB
Markdown
128 lines
4.0 KiB
Markdown
# Eval Lineage — Full Traceability from Production to Deployment
|
|
|
|
Track the complete chain from production traces through dataset creation, evaluation runs, comparisons, and deployment decisions. Enables "why was this deployed?" audit queries and compliance reporting.
|
|
|
|
## Lineage Chain
|
|
|
|
```
|
|
Production Trace (App Insights)
|
|
│ conversationId, responseId
|
|
▼
|
|
Dataset Version (.foundry/datasets/*.jsonl, environment-scoped)
|
|
│ metadata.conversationId, metadata.harvestRule
|
|
▼
|
|
Evaluation Run (evaluation_agent_batch_eval_create)
|
|
│ evaluationId when creating, evalId when querying, evalRunId
|
|
▼
|
|
Comparison (evaluation_comparison_create)
|
|
│ insightId, baselineRunId, treatmentRunIds
|
|
▼
|
|
Deployment Decision (agent_update)
|
|
│ agentVersion
|
|
▼
|
|
Production Trace (cycle repeats)
|
|
```
|
|
|
|
## Lineage Manifest
|
|
|
|
Track lineage in `.foundry/datasets/manifest.json`:
|
|
|
|
```json
|
|
{
|
|
"datasets": [
|
|
{
|
|
"name": "support-bot-prod-traces",
|
|
"file": "support-bot-prod-traces-v3.jsonl",
|
|
"version": "v3",
|
|
"tag": "prod",
|
|
"source": "trace-harvest",
|
|
"harvestRule": "error+latency",
|
|
"timeRange": "2025-02-01 to 2025-02-07",
|
|
"exampleCount": 63,
|
|
"createdAt": "2025-02-08T10:00:00Z",
|
|
"evalRuns": [
|
|
{
|
|
"evalId": "eval-group-001",
|
|
"runId": "run-abc-123",
|
|
"agentVersion": "3",
|
|
"date": "2025-02-08T12:00:00Z",
|
|
"status": "completed"
|
|
},
|
|
{
|
|
"evalId": "eval-group-001",
|
|
"runId": "run-def-456",
|
|
"agentVersion": "4",
|
|
"date": "2025-02-10T09:00:00Z",
|
|
"status": "completed"
|
|
}
|
|
],
|
|
"comparisons": [
|
|
{
|
|
"insightId": "insight-xyz-789",
|
|
"baselineRunId": "run-abc-123",
|
|
"treatmentRunIds": ["run-def-456"],
|
|
"result": "v4 improved on 3/5 metrics",
|
|
"date": "2025-02-10T10:00:00Z"
|
|
}
|
|
],
|
|
"deployments": [
|
|
{
|
|
"agentVersion": "4",
|
|
"deployedAt": "2025-02-10T14:00:00Z",
|
|
"reason": "v4 improved coherence +25%, relevance +10% vs v3"
|
|
}
|
|
]
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
## Audit Queries
|
|
|
|
### "Why was version X deployed?"
|
|
|
|
1. Read `.foundry/datasets/manifest.json`
|
|
2. Find entries where `deployments[].agentVersion == X`
|
|
3. Show the comparison that justified the deployment
|
|
4. Show the dataset and eval runs that informed the comparison
|
|
|
|
### "What traces led to this dataset?"
|
|
|
|
1. Read the dataset JSONL file
|
|
2. Extract `metadata.conversationId` from each example
|
|
3. Look up each conversation in App Insights using the [trace skill](../../trace/trace.md)
|
|
|
|
### "What evaluation history does this agent have?"
|
|
|
|
1. Use **`evaluation_get`** to list all evaluation groups
|
|
2. For each group, list runs with `isRequestForRuns=true`
|
|
3. Build the timeline from [Eval Trending](eval-trending.md)
|
|
4. Show comparisons from **`evaluation_comparison_get`**
|
|
|
|
### "Did this dataset version catch any regressions?"
|
|
|
|
1. Find the dataset version in the manifest
|
|
2. Check `evalRuns` for runs that used this dataset
|
|
3. Check `comparisons` for any regression results
|
|
4. Cross-reference with `tag == "regression-<date>"` entries
|
|
|
|
## Maintaining Lineage
|
|
|
|
Update `.foundry/datasets/manifest.json` at each step:
|
|
|
|
| Event | Fields to Update |
|
|
|-------|-----------------|
|
|
| Dataset created | Add new entry with `name`, `version`, `source`, `exampleCount` |
|
|
| Evaluation run | Append to `evalRuns[]` with `evalId`, `runId`, `agentVersion` |
|
|
| Comparison | Append to `comparisons[]` with `insightId`, `result` |
|
|
| Deployment | Append to `deployments[]` with `agentVersion`, `reason` |
|
|
| Tag change | Update `tag` field |
|
|
|
|
> 💡 **Tip:** Store the evaluation group identifier as `evalId` in lineage/manifest records, even if the create call used the parameter name `evaluationId`.
|
|
|
|
## Next Steps
|
|
|
|
- **View metric trends** → [Eval Trending](eval-trending.md)
|
|
- **Check for regressions** → [Eval Regression](eval-regression.md)
|
|
- **Harvest new traces** → [Trace-to-Dataset](trace-to-dataset.md) (start the next cycle)
|