Tracing Insights API

Automatically detect quality regressions and anomalies in agent traces using changepoint detection on evaluation scores stored in App Insights.

When to Use

Use this instead of manual KQL queries when you want automated anomaly detection across evaluation dimensions (task adherence, intent resolution, fluency, latency, token usage). The API finds statistical changepoints in score distributions — no manual threshold tuning needed.

Prerequisites:

App Insights connected to the Foundry project (with gen_ai.evaluation.result custom events)
Evaluation data from portal playground sessions or batch evals (raw traces alone are not enough)

Endpoint

POST https://{region}.api.azureml.ms/notification/v1-beta2/subscriptions/{sub}/resourceGroups/{rg}/providers/microsoft.insights/components/{component}/:insights

The API is region-agnostic — any regional endpoint can serve requests for any project. For lowest latency, use the same region as the Foundry project (e.g., eastus2, westus2, westcentralus). If the project region is unknown, use eastus2 as the default.

Query parameters:

Parameter	Required	Description
`startDateTimeUtc`	Yes	ISO 8601 start of analysis window
`endDateTimeUtc`	Yes	ISO 8601 end of analysis window
`agent`	Yes	Agent name (URL-encoded)
`projectId`	Yes	ARM resource ID of the Foundry project (URL-encoded — contains slashes)
`top`	No	Max insights to return (default 50)

Auth: az account get-access-token --resource https://ai.azure.com

Body: Must send {} (empty JSON object) — POST with no body returns 400.

Example

$token = az account get-access-token --resource https://ai.azure.com --query accessToken -o tsv
$encodedAgent = [uri]::EscapeDataString("my-agent")
$encodedProjectId = [uri]::EscapeDataString("/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.CognitiveServices/accounts/{account}/projects/{project}")

$uri = "https://{region}.api.azureml.ms/notification/v1-beta2/subscriptions/{sub}/resourceGroups/{rg}/providers/microsoft.insights/components/{component}/:insights?startDateTimeUtc=2025-01-01T00:00:00Z&endDateTimeUtc=2025-01-18T00:00:00Z&agent=$encodedAgent&projectId=$encodedProjectId&top=50"

$response = Invoke-RestMethod -Uri $uri -Method POST -Headers @{
    "Authorization" = "Bearer $token"
    "Content-Type" = "application/json"
} -Body "{}"

Response Structure (v1-beta2)

Response is grouped by agent version. Each insight includes relatedSpans with operationId (App Insights trace ID) for querying full trace content.

{
  "agents": [{
    "agent": "my-agent:1",
    "insights": [{
      "id": "anomaly-token-shift-<hash>",
      "type": "Token",
      "severity": "Critical",
      "message": "Token usage increased by 137%",
      "agentVersion": "1",
      "metadata": { "meanBefore": 2041, "meanAfter": 4831, "confidence": 0.91 },
      "relatedSpans": {
        "totalCount": 13,
        "spans": [
          { "responseId": "resp_...", "operationId": "<trace-id>", "evaluationRunId": null }
        ]
      }
    }],
    "insightCount": 3
  }],
  "totalCount": 3, "criticalCount": 1, "warningCount": 1, "improvementCount": 1
}

Querying Traces from relatedSpans

Use operationId from relatedSpans to fetch full trace content from App Insights:

dependencies
| where operation_Id == "<operationId>"
| where customDimensions has "invoke_agent"
| project input = customDimensions["gen_ai.input.messages"],
          output = customDimensions["gen_ai.output.messages"],
          tokens = toint(customDimensions["gen_ai.usage.output_tokens"])

This returns the user query and agent response for the specific trace flagged by the insight.

How Changepoint Detection Works

The API finds statistical inflection points within the queried time window. meanBefore/meanAfter represent averages on either side of the detected shift — not comparisons to a historical baseline.

10+ data points give better signal for changepoint detection
confidence close to 1.0 = statistically significant shift

Next Steps

After receiving insights with Warning or Critical severity:

Use relatedSpans.operationId values to query full trace content from App Insights (see KQL above)
Present the insights summary to the user with severity, type, evaluator name, and shift magnitude
Offer to drill into specific traces for detailed analysis using the trace analysis skill

4.8 KiB Raw Permalink Blame History