Step 1 - Auto-Setup Evaluation Suite

This step runs automatically after deployment. If the agent was deployed via the deploy skill, .foundry cache and metadata may already be configured. Check .foundry/evaluators/, .foundry/datasets/, and the selected metadata file under the selected agent root before re-creating them.

Auto-Generate Suite

After deployment, immediately prepare a Foundry evaluation suite and local references for the selected environment without waiting for the user to request it.

1. Resolve Context

Use Common Project Context Resolution to compute effective context. In azd projects, prefer azd env get-values for deployment context and use the selected .foundry/agent-metadata*.yaml file only as an overlay/cache. Use agent_get, the local azure.yaml service block, and matching eval.yaml as needed to resolve:

Value	Source
`projectEndpoint`	azd env, then metadata override
`agentName` / `agentVersion`	azd agent vars, then metadata/`agent_get`
`suiteName`	verified `eval.yaml` name or `<agent-name>-smoke` unless user provided one
generation deployment	`model_deployment_get`; choose a chat-completions deployment

suiteName must start with a letter (A-Z or a-z). If a derived name starts with a number, prefix it with an alphabetic label such as suite-.

Do not assume gpt-4o exists.

2. Reuse or Refresh Cache

Inspect .foundry/suites/, .foundry/evaluators/, .foundry/datasets/, matching eval.yaml, and the selected environment's evaluationSuites[] in the selected agent root only. Do not merge sibling agent folders.

Suite metadata has suiteName and current cache -> call evaluation_suite_get to verify the remote suite, then reuse it.
eval.yaml exists and matches the selected agent -> verify its dataset.local_uri or registered dataset.name/dataset.version, evaluators[], and optional name remotely or register them before persisting a synced suite entry. Normalize legacy dataset_file in memory only.
Cache is missing/stale or user asks refresh -> generate a new suite after confirming any overwrite.
Legacy entry without suiteName -> keep it as legacy fallback metadata unless the user approves generating a new suite.

3. Generate Suite

Read Evaluation Suite Generation. If the user selected existing eval.yaml, follow the local eval.yaml verification/registration path there before creating a generated suite. Otherwise call:

evaluation_suite_generation_job_create(
  projectEndpoint,
  suiteName,
  agentName,
  generationModelDeploymentName,
  dataGenerationType,
  maxSamples
)

For trace-informed suites, include traceAgentName or traceAgentId, traceAgentVersion, traceStartTime, traceEndTime, and maxTraces. Start background polling with evaluation_suite_generation_job_get, suppress intermediate in_progress output, then verify the generated suite with evaluation_suite_get after terminal success.

When refining an existing dataset, include datasetName and datasetVersion.

4. Persist Local References

Cache generated artifacts inside the selected root:

.foundry/
  agent-metadata.yaml
  agent-metadata.prod.yaml
  suites/<suite-name>-v<version>.json
  evaluators/<evaluator-name>-v<version>.json
  datasets/<agent-name>-<dataset-name>-v<version>.ref.json
  datasets/<dataset-name>-v<version>/<blob-name>
  results/

If the job result exposes only remote names/versions, fetch metadata with evaluation_suite_get(projectEndpoint, suiteName, suiteVersion), evaluation_dataset_get, evaluation_dataset_sas_url_get, and evaluator_catalog_get, then materialize the full suite JSON, full evaluator JSON, dataset .ref.json, and downloaded dataset blobs. Never overwrite user-edited cache files without confirmation; deterministic re-fetch of the same immutable remote <name>-v<version> may replace the generated cache artifact for that exact version.

5. Update Metadata

Write only the selected metadata file and selected environment. In azd projects, persist only non-derivable overlay/cache state; do not copy azd-owned project endpoint, agent name/version, ACR, or observability values. Persist evaluation suites with:

id, tags, suiteName, suiteVersion
generationJobId, generationSource (synthetic, traces, or manual-fallback)
dataset, datasetVersion, datasetFile, datasetUri
evaluator name, version, threshold, definitionFile (full cached JSON)

Use tags such as tier: smoke, purpose: baseline, and stage: generated. If metadata still uses older testSuites[] or legacy testCases[], replace that list with evaluationSuites[] on write and map priority to tags.tier only when tags.tier is missing.

6. Fallback

If suite generation fails, is unavailable, or returns incomplete artifacts, explain the failure and fall back to the existing manual path: evaluator_catalog_get, local seed JSONL generation via Generate Seed Evaluation Dataset, evaluation_dataset_create, and evaluationSuites[] metadata with generationSource: manual-fallback.

7. Prompt User

Ask: "Your agent is deployed and the selected environment has evaluation-suite metadata plus local dataset/evaluator references. Would you like to run an evaluation to identify optimization opportunities?"

If yes -> proceed to Step 2: Evaluate. If no -> stop.

5.5 KiB Raw Permalink Blame History