skills/dotnet__skills
1
0
Fork 0
mirror of https://github.com/dotnet/skills.git synced 2026-09-20 09:49:54 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
eval-verify/nuget-comment
dotnet__skills/.github/workflows/evaluation.yml
T
Abhitej John 78054c1161 Migrate LLM evals to the Vally harness (#877) 
* Make Vally the sole LLM eval engine, retiring skill-validator evaluate

Collapses the parallel skill-validator + Vally eval pipeline into a single
Vally-only path while preserving every capability skill-validator provided:
PR gate, PR comment, dotnet/skills-data historical push, and the dashboard.
The skill-validator `check` linter is retained (skill-check.yml) pending
microsoft/vally #463.

- evaluation.yml: delete build-validator and the skill-validator `evaluate`
  job; single `evaluate:` job now uses vally-evaluation.yml. Forward the
  COPILOT_PAT_0..9 pool onto it (from #911) so the reusable workflow's token
  gate is satisfied. Rewire comment-on-pr / report-status / publish-* /
  deploy-dashboard onto `evaluate` and vally-results-* artifacts.
- vally-evaluation.yml: pin @microsoft/vally-cli@0.9.0; keep the #911 prepare
  guard; write the generated per-plugin experiment to $GITHUB_WORKSPACE so
  relative eval globs resolve.
- adapt.mjs: reconcile #887's runCompareWithRetry + conclusive/unmatched
  hardening with the consolidation helpers (nonActivation, compareByStim,
  roleToDashboard, etc.). Add consolidate.mjs and gen-experiment.mjs.
- build-replay-sessions.ps1: read Vally executor-session-logs/events.jsonl
  instead of skill-validator session output.
- Rename all tests/*/*/eval.vally.yaml to eval.yaml (Vally format is now the
  only eval format); author Vally evals for the four skills that lacked one.
- CONTRIBUTING.md + InvestigatingResults: point contributors at the Vally
  harness; drop non-public links.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix setup-local-sdk eval for Vally schema

The hand-authored setup-local-sdk eval used skill-validator grader/environment
schema that Vally rejects at validation (no stimuli executed):
- file-contains/file-not-contains graders require 'value:' not 'substring:'
  (output-contains correctly keeps 'substring:').
- environment.files entries require 'src:'/'dest:' (copy a fixture) rather than
  inline 'path:'/'content:'. Moved the global.json body into fixtures/global.json.

Verified with 'vally lint --eval-spec' (0 errors) across all 97 eval.yaml.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix markdownlint violations in migration docs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review feedback on eval migration

- Consolidate run-vally-evals.sh into a single neutral entrypoint
  eng/run-skill-evals.sh (deletes the thin wrapper; adapt.mjs stays in
  eng/vally-adapter/ since CI invokes it directly)
- Rename reusable workflow vally-evaluation.yml -> evaluation-run.yml and
  update all references/self-trigger regexes
- Rename local output dir vally-results -> eval-results
- Bake prerequisite preflight (Node 20+, GITHUB_TOKEN/gh) into the runner
- Simplify CONTRIBUTING "Running tests locally" prerequisites
- Scrub harness name from consumer-facing surfaces

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Reinstate overfitting detection under the Vally harness

Overfitting detection previously ran inside skill-validator's `evaluate`
command, which the Vally migration replaced, so it silently stopped running.
Reinstate it as a standalone step wired into the Vally pipeline:

- Add `skill-validator overfitting` command that discovers skills, parses
  each eval.yaml, runs the existing OverfittingJudge, and emits
  [{plugin, skill, overfittingResult}] JSON (per-skill failures non-fatal,
  bounded parallelism).
- Add a Vally-format parser bridge (ParseEvalConfigFlexible) so the judge
  handles the current stimuli/graders eval.yaml format; the legacy
  scenarios-only parser rejected all 98 evals, so the judge never ran.
- adapt.mjs: merge overfitting results onto each verdict via
  --overfitting <file> (keyed by plugin/skill); output is byte-identical
  when the flag is absent.
- evaluation-run.yml: build skill-validator (shared cache with check),
  run the judge per leg reading GITHUB_TOKEN, and pass --overfitting to
  adapt.mjs so the dashboard/skills-data consume it unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Reconcile #919 Vally eval configs into the sole eval.yaml convention

PR #919 landed on main adding fresh Vally specs for three coverage-gap
skills (setup-local-sdk, convert-blazor-server-to-webapp, dotnet-webapi)
under the interim eval.vally.yaml filename. This branch collapses to a
single Vally spec named eval.yaml, so the merge left each skill with both
my migrated eval.yaml and #919's newer eval.vally.yaml (a git-clean but
semantic duplicate the pipeline would not discover).

Adopt #919's reviewed Vally content as the canonical eval.yaml for all
three skills and drop the redundant eval.vally.yaml files, preserving the
'sole eval.yaml, zero eval.vally.yaml' invariant. Content is byte-identical
to #919's blobs (b14121c6 / 47d43823 / 18a9362f), which already passed
vally-evaluate on main.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-07-21 15:22:31 -07:00

1454 lines
68 KiB
YAML
Raw Permalink Blame History

# Unified evaluation workflow for all PRs (same-repo and fork) and scheduled runs.
#
# IMPORTANT: The /evaluate command uses the `issue_comment` trigger, which
# ALWAYS runs the workflow YAML from the default branch (main), NOT from the
# PR branch. Changes to this file in a PR will not take effect until merged.
# The Vally harness runs the skills content checked out from the PR branch, so
# skill/test/eval.yaml changes are evaluated before merge.
#
# For PRs (same-repo and fork):
# - On PR open/sync, the `pr-status` job posts an initial commit status:
# - "success" if no skills changed (required check passes immediately)
# - "pending" if skills changed (maintainer must post /evaluate to trigger)
# - When a maintainer posts "/evaluate" on the PR, the `gate` job validates
# permissions and triggers the full evaluation pipeline.
#
# For scheduled runs:
# - Runs daily, evaluates all plugins with skills and tests.
#
# Security model for fork PRs:
# - Workflow YAML: always from the default branch (enforced by issue_comment
# and pull_request_target triggers)
# - Skill/test content: checked out from the fork PR (untrusted data, read-only)
# - Secret access: only users with write+ permission can trigger evaluation
name: evaluation
# For a triage/manual single-PR dispatch, surface the PR + head in the run name.
# This run name also doubles as the idempotency key the PR-triage worker matches
# on (eval_run_exists_for_head in pr-triage-act.sh): a workflow_dispatch run's
# head_sha is the default branch — not the PR head — so the run name is how the
# worker recognises that an evaluation has already been dispatched for this head.
# For all other events the expression yields '' and GitHub uses the default name.
run-name: "${{ inputs.pr_number != '' && format('Evaluate PR #{0} @ {1}', inputs.pr_number, inputs.head_sha) || '' }}"
on:
# Manual trigger for one-off deploys (e.g., AGENTVIZ SPA update) and
# targeted re-runs of a single plugin without committing to main.
# The pr_number/head_sha inputs are also the PR-triage worker's entry point:
# the worker dispatches this workflow directly (workflow_dispatch is exempt
# from GitHub's GITHUB_TOKEN recursion guard) instead of relying on the
# `evaluate-now` label, which never fires when applied by the bot.
workflow_dispatch:
inputs:
plugin:
description: "Specific plugin to evaluate (leave blank for all)"
type: string
required: false
pr_number:
description: "PR number to evaluate. Routes the dispatch through the gate/PR pipeline instead of a full/plugin run."
type: string
required: false
head_sha:
description: "Short head SHA of the PR (informational; used for the run name and triage idempotency). The gate re-fetches the live head."
type: string
required: false
# Same-repo PRs: post initial status
pull_request:
# Fork PRs: post initial status (runs from base branch for security).
# Also receives `labeled` events: a human applying the `evaluate-now` label
# is one entry point into this pipeline (alongside `/evaluate`). The triage
# worker runs as github-actions[bot] and cannot use the label (label events
# emitted by GITHUB_TOKEN do not start workflows), so it uses the
# workflow_dispatch pr_number input above instead.
pull_request_target:
types: [opened, synchronize, reopened, labeled]
# /evaluate command trigger
issue_comment:
types: [created]
# Daily scheduled evaluation
schedule:
- cron: '0 0 * * *' # Once daily at midnight UTC
concurrency:
# /evaluate (issue_comment), the human-applied `evaluate-now` label
# (pull_request_target labeled), and the triage worker's workflow_dispatch
# (pr_number input) all share the same `eval-<pr_number>` group, so a race
# between any of them collapses to a single run for the PR.
group: ${{ github.workflow }}-${{ (github.event_name == 'issue_comment' && startsWith(github.event.comment.body, '/evaluate')) && format('eval-{0}', github.event.issue.number) || (github.event_name == 'pull_request_target' && github.event.action == 'labeled' && github.event.label.name == 'evaluate-now') && format('eval-{0}', github.event.pull_request.number) || (github.event_name == 'workflow_dispatch' && inputs.pr_number != '') && format('eval-{0}', inputs.pr_number) || (github.event_name == 'issue_comment' && format('eval-noop-{0}-{1}', github.event.issue.number, github.event.comment.id)) || (github.event_name == 'pull_request' && format('eval-status-{0}', github.event.pull_request.number)) || (github.event_name == 'pull_request_target' && format('eval-fork-status-{0}', github.event.pull_request.number)) || github.run_id }}
cancel-in-progress: true
env:
DASHBOARD_RETENTION_DAYS: 14
MODEL: claude-opus-4.6
JUDGE_MODEL: claude-opus-4.6
permissions:
contents: write
pull-requests: write
statuses: write
jobs:
# ==========================================================================
# PR STATUS JOBS
# Post initial commit status so the required check is never stuck as "Expected".
# Posts success (no skills) or pending (needs /evaluate).
# ==========================================================================
# Same-repo PRs: use pull_request trigger (has direct access to PR content)
pr-status:
if: >-
github.event_name == 'pull_request' &&
github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
permissions:
contents: read
statuses: write
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v6
with:
fetch-depth: 0
persist-credentials: false
- name: Discover changes requiring evaluation
id: discover
shell: pwsh
run: |
$base = "${{ github.event.pull_request.base.sha }}"
$head = "${{ github.event.pull_request.head.sha }}"
$mergeBase = git merge-base $base $head
$changedFiles = git diff --name-only --diff-filter=ACMR $mergeBase $head
$hasSkillChanges = $changedFiles |
Where-Object { $_ -match '^(plugins/[^/]+/skills|tests/[^/]+)/[^/]+/' } |
Select-Object -First 1
# Evaluation pipeline changes need a re-eval. The linter (skill-validator)
# is a separate workflow, so eng/skill-validator/ changes are NOT infra
# changes here. Documentation files don't affect evaluation.
$hasInfraChanges = $changedFiles |
Where-Object {
($_ -match '^eng/vally-adapter/') -or
($_ -match '^dotnet-skills\.experiment\.yaml$') -or
$_ -match '^\.github/workflows/(evaluation|evaluation-run)\.yml$'
} |
Select-Object -First 1
if ($hasSkillChanges -or $hasInfraChanges) {
echo "needs_eval=true" >> $env:GITHUB_OUTPUT
} else {
echo "needs_eval=false" >> $env:GITHUB_OUTPUT
}
- name: Post evaluation commit status
env:
GH_TOKEN: ${{ github.token }}
run: |
if [[ "${{ steps.discover.outputs.needs_eval }}" == "true" ]]; then
STATE="pending"
DESC="Post /evaluate to trigger evaluation"
else
STATE="success"
DESC="No skills to evaluate"
fi
gh api "repos/${{ github.repository }}/statuses/${{ github.event.pull_request.head.sha }}" \
-f state="$STATE" \
-f context="evaluation-status" \
-f description="$DESC" \
-f target_url="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
# Fork PRs: use pull_request_target (runs from base branch, fetches PR metadata safely)
fork-pr-status:
if: >-
github.event_name == 'pull_request_target' &&
github.event.action != 'labeled' &&
github.event.action != 'unlabeled' &&
github.event.pull_request.head.repo.full_name != github.repository
runs-on: ubuntu-latest
permissions:
contents: read
statuses: write
steps:
- name: Checkout base branch
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v6
with:
ref: ${{ github.event.pull_request.base.sha }}
fetch-depth: 0
persist-credentials: false
- name: Fetch PR head for diff
run: git fetch origin +refs/pull/${{ github.event.pull_request.number }}/head
- name: Discover changes requiring evaluation
id: discover
shell: pwsh
run: |
$base = "${{ github.event.pull_request.base.sha }}"
$head = "FETCH_HEAD"
$mergeBase = git merge-base $base $head
$changedFiles = git diff --name-only --diff-filter=ACMR $mergeBase $head
$hasSkillChanges = $changedFiles |
Where-Object { $_ -match '^(plugins/[^/]+/skills|tests/[^/]+)/[^/]+/' } |
Select-Object -First 1
$hasInfraChanges = $changedFiles |
Where-Object {
($_ -match '^eng/vally-adapter/') -or
($_ -match '^dotnet-skills\.experiment\.yaml$') -or
$_ -match '^\.github/workflows/(evaluation|evaluation-run)\.yml$'
} |
Select-Object -First 1
if ($hasSkillChanges -or $hasInfraChanges) {
echo "needs_eval=true" >> $env:GITHUB_OUTPUT
} else {
echo "needs_eval=false" >> $env:GITHUB_OUTPUT
}
- name: Post evaluation commit status
env:
GH_TOKEN: ${{ github.token }}
run: |
if [[ "${{ steps.discover.outputs.needs_eval }}" == "true" ]]; then
STATE="pending"
DESC="Fork PR — post /evaluate to trigger evaluation"
else
STATE="success"
DESC="No skills to evaluate"
fi
gh api "repos/${{ github.repository }}/statuses/${{ github.event.pull_request.head.sha }}" \
-f state="$STATE" \
-f context="evaluation-status" \
-f description="$DESC" \
-f target_url="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
# ==========================================================================
# GATE JOB
# Validate evaluation trigger. Three entry points:
# 1. /evaluate comment (issue_comment) — original human path
# 2. evaluate-now label (pull_request_target labeled) — a human applying
# the label (the bot cannot: GITHUB_TOKEN label events don't start runs)
# 3. workflow_dispatch with pr_number — the pr-triage worker's path. It runs
# as github-actions[bot] and dispatches this workflow directly, since
# workflow_dispatch is exempt from the GITHUB_TOKEN recursion guard.
# All must come from a trusted actor (write+ on the repo, or the workflow's
# own github-actions[bot] identity for paths 2 and 3).
# ==========================================================================
gate:
if: >-
(github.event.issue.pull_request &&
startsWith(github.event.comment.body, '/evaluate'))
||
(github.event_name == 'pull_request_target' &&
github.event.action == 'labeled' &&
github.event.label.name == 'evaluate-now')
||
(github.event_name == 'workflow_dispatch' &&
inputs.pr_number != '')
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
statuses: write
issues: write
outputs:
head_sha: ${{ steps.pr.outputs.head_sha }}
base_sha: ${{ steps.pr.outputs.base_sha }}
pr_number: ${{ steps.pr.outputs.pr_number }}
is_fork: ${{ steps.pr.outputs.is_fork }}
steps:
- name: Check actor permissions
id: perms
env:
GH_TOKEN: ${{ github.token }}
run: |
if [[ "${{ github.event_name }}" == "issue_comment" ]]; then
ACTOR='${{ github.event.comment.user.login }}'
else
ACTOR='${{ github.event.sender.login }}'
fi
# The triage worker dispatches this workflow as github-actions[bot]
# (workflow_dispatch). A human may instead apply the evaluate-now label,
# which arrives under that human's identity. The bot identity has no
# entry in /collaborators/* but is implicitly trusted because only this
# repository's own GITHUB_TOKEN can act under it.
if [[ "$ACTOR" == "github-actions[bot]" ]]; then
echo "Actor is github-actions[bot] — trusted by construction"
exit 0
fi
PERMISSION=$(gh api "repos/${{ github.repository }}/collaborators/${ACTOR}/permission" --jq '.permission')
echo "Actor ${ACTOR} has permission: $PERMISSION"
if [[ "$PERMISSION" != "admin" && "$PERMISSION" != "write" && "$PERMISSION" != "maintain" ]]; then
echo "::error::Actor does not have write access"
exit 1
fi
- name: Get PR details
id: pr
env:
GH_TOKEN: ${{ github.token }}
# Read the dispatch input via env (not direct interpolation) so the
# numeric validation below runs before the value reaches any gh api
# call — a crafted input cannot inject shell or path traversal.
PR_NUMBER_INPUT: ${{ inputs.pr_number }}
run: |
if [[ "${{ github.event_name }}" == "issue_comment" ]]; then
PR_NUMBER='${{ github.event.issue.number }}'
elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
PR_NUMBER="$PR_NUMBER_INPUT"
if [[ ! "$PR_NUMBER" =~ ^[0-9]+$ ]]; then
echo "::error::workflow_dispatch input pr_number='$PR_NUMBER' must be a positive integer"
exit 1
fi
else
PR_NUMBER='${{ github.event.pull_request.number }}'
fi
PR_DATA=$(gh api "repos/${{ github.repository }}/pulls/${PR_NUMBER}")
HEAD_SHA=$(echo "$PR_DATA" | jq -r '.head.sha')
HEAD_REPO=$(echo "$PR_DATA" | jq -r '.head.repo.full_name')
BASE_REPO=$(echo "$PR_DATA" | jq -r '.base.repo.full_name')
BASE_SHA=$(echo "$PR_DATA" | jq -r '.base.sha')
if [[ "$HEAD_REPO" != "$BASE_REPO" ]]; then
echo "is_fork=true" >> $GITHUB_OUTPUT
else
echo "is_fork=false" >> $GITHUB_OUTPUT
fi
echo "PR #${PR_NUMBER}: head=${HEAD_SHA} base=${BASE_SHA} fork=$([[ "$HEAD_REPO" != "$BASE_REPO" ]] && echo true || echo false)"
echo "head_sha=${HEAD_SHA}" >> $GITHUB_OUTPUT
echo "base_sha=${BASE_SHA}" >> $GITHUB_OUTPUT
echo "pr_number=${PR_NUMBER}" >> $GITHUB_OUTPUT
- name: Add reaction to comment
if: github.event_name == 'issue_comment'
env:
GH_TOKEN: ${{ github.token }}
run: |
gh api "repos/${{ github.repository }}/issues/comments/${{ github.event.comment.id }}/reactions" \
-X POST -f content='eyes' || true
- name: Remove evaluate-now label
if: github.event_name == 'pull_request_target' || github.event_name == 'workflow_dispatch'
continue-on-error: true
env:
GH_TOKEN: ${{ github.token }}
run: |
# For the label entry point, removing the label consumes the trigger so
# re-applying re-fires. For the workflow_dispatch entry point there is no
# label to consume, but we still strip any stale human-applied
# evaluate-now label so it doesn't linger. Removal uses the workflow's
# own GITHUB_TOKEN; per GitHub's recursion rules, label events emitted by
# GITHUB_TOKEN do not start new workflow runs.
gh pr edit "${{ steps.pr.outputs.pr_number }}" \
--repo "${{ github.repository }}" \
--remove-label "evaluate-now" || true
- name: Set pending commit status
continue-on-error: true
env:
GH_TOKEN: ${{ github.token }}
run: |
gh api "repos/${{ github.repository }}/statuses/${{ steps.pr.outputs.head_sha }}" \
-f state=pending \
-f context="evaluation-status" \
-f description="Evaluation in progress..." \
-f target_url="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
# ==========================================================================
# DISCOVER JOB
# Find skills to evaluate based on changed files.
# ==========================================================================
discover:
needs: gate
if: >-
always() &&
(needs.gate.result == 'success' || github.event_name == 'schedule' || (github.event_name == 'workflow_dispatch' && inputs.pr_number == '')) &&
(github.event_name != 'schedule' || github.repository == 'dotnet/skills')
runs-on: ubuntu-latest
permissions:
actions: read
contents: read
outputs:
entries: ${{ steps.find.outputs.entries }}
has_entries: ${{ steps.find.outputs.has_entries }}
is_infra: ${{ steps.find.outputs.is_infra }}
plugins: ${{ steps.find.outputs.plugins }}
has_plugins: ${{ steps.find.outputs.has_plugins }}
steps:
- name: Check for new commits since last evaluation
if: github.event_name == 'schedule'
id: check-changes
env:
GH_TOKEN: ${{ github.token }}
run: |
# Bypass the skip guard on manual reruns so "Re-run jobs" always executes.
if [ "${GITHUB_RUN_ATTEMPT}" != "1" ]; then
echo "Manual rerun (attempt ${GITHUB_RUN_ATTEMPT}) — bypassing skip guard"
echo "has_changes=true" >> $GITHUB_OUTPUT
exit 0
fi
# Determine whether a new evaluation is needed by inspecting the most
# recent completed scheduled run.
LATEST=$(gh api "repos/${{ github.repository }}/actions/workflows/evaluation.yml/runs?event=schedule&status=completed&per_page=1" \
--jq '(.workflow_runs[0] // empty) | "\(.head_sha) \(.conclusion)"' 2>/dev/null) || LATEST=""
if [ -z "$LATEST" ]; then
echo "No previous completed scheduled run found — proceeding"
echo "has_changes=true" >> $GITHUB_OUTPUT
exit 0
fi
LAST_SHA="${LATEST%% *}"
LAST_CONCLUSION="${LATEST##* }"
CURRENT_SHA="${{ github.sha }}"
if [ "$LAST_SHA" = "$CURRENT_SHA" ] && [ "$LAST_CONCLUSION" = "success" ]; then
echo "Last scheduled evaluation at $LAST_SHA succeeded — skipping"
echo "has_changes=false" >> $GITHUB_OUTPUT
else
if [ "$LAST_SHA" != "$CURRENT_SHA" ]; then
COUNT=$(gh api "repos/${{ github.repository }}/compare/${LAST_SHA}...${CURRENT_SHA}" --jq '.total_commits' 2>/dev/null) || COUNT="unknown"
echo "$COUNT new commit(s) since last evaluation ($LAST_SHA)"
else
echo "Last scheduled evaluation at $LAST_SHA concluded with '$LAST_CONCLUSION' — retrying"
fi
echo "has_changes=true" >> $GITHUB_OUTPUT
fi
- name: Checkout repository
if: github.event_name != 'schedule' || steps.check-changes.outputs.has_changes == 'true' || github.event_name == 'workflow_dispatch'
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v6
with:
fetch-depth: 0
persist-credentials: false
- name: Fetch PR head
if: needs.gate.outputs.pr_number != ''
run: git fetch origin +refs/pull/${{ needs.gate.outputs.pr_number }}/head:refs/remotes/origin/pr-head
- name: Find skills to evaluate
if: github.event_name != 'schedule' || steps.check-changes.outputs.has_changes == 'true' || github.event_name == 'workflow_dispatch'
id: find
run: |
$entries = @()
$plugins = @()
# Build matrix entries for a full-plugin evaluation, sharding the
# plugin's skills by the optional `executionShard:` top-level tag
# in tests/<plugin>/<skill>/eval.yaml. Untagged skills fall into a
# synthetic "default" bucket. Plugins with all skills in one bucket
# produce a single entry (unchanged behavior); plugins with multiple
# buckets fan out into one matrix entry per shard so each shard
# runs on a fresh GitHub-hosted runner. This is the mitigation for
# MCP-heavy plugins like dotnet-msbuild, where cumulative runner
# resource usage across ~90 min of continuous MCP traffic
# consistently triggered host-level termination.
function Get-PluginShardEntries {
param(
[string]$plugin,
[string]$contentRoot = ".",
[string[]]$selectedSkills = @()
)
$skillsDir = "plugins/$plugin/skills"
$skillsRoot = Join-Path $contentRoot $skillsDir
$singleEntry = @{ name = $plugin; plugin = $plugin; skills_path = $skillsDir }
if (-not (Test-Path $skillsRoot)) {
if ($selectedSkills.Count -gt 0) { return @() }
return @($singleEntry)
}
$skillDirs = @(Get-ChildItem -Path $skillsRoot -Directory -ErrorAction SilentlyContinue |
Sort-Object Name | Select-Object -ExpandProperty Name)
if ($selectedSkills.Count -gt 0) {
$selected = @{}
foreach ($skill in $selectedSkills) { $selected[$skill] = $true }
$skillDirs = @($skillDirs | Where-Object { $selected.ContainsKey($_) })
}
if ($skillDirs.Count -eq 0) {
if ($selectedSkills.Count -gt 0) { return @() }
return @($singleEntry)
}
$shardGroups = @{}
foreach ($skill in $skillDirs) {
$evalPath = Join-Path $contentRoot "tests" $plugin $skill "eval.yaml"
$shard = "default"
if (Test-Path $evalPath) {
# Allow optional leading whitespace so an accidentally indented
# executionShard: key still groups correctly (rather than silently
# collapsing back to the default bucket).
$m = Select-String -Path $evalPath -Pattern '^\s*executionShard:\s*[\x27"]?([\w.\-]+)' -List
if ($m) { $shard = $m.Matches[0].Groups[1].Value }
}
if (-not $shardGroups.ContainsKey($shard)) { $shardGroups[$shard] = @() }
$shardGroups[$shard] += $skill
}
if ($shardGroups.Count -le 1) {
if ($selectedSkills.Count -gt 0) {
$paths = ($skillDirs | ForEach-Object { "$skillsDir/$_" }) -join ' '
$name = if ($skillDirs.Count -eq 1) { "$plugin--$($skillDirs[0])" } else { $plugin }
return @(@{ name = $name; plugin = $plugin; skills_path = $paths })
}
return @($singleEntry)
}
return @($shardGroups.Keys | Sort-Object | ForEach-Object {
$shardName = $_
$paths = ($shardGroups[$shardName] | ForEach-Object { "$skillsDir/$_" }) -join ' '
@{ name = "$plugin--shard-$shardName"; plugin = $plugin; skills_path = $paths }
})
}
if ("${{ needs.gate.outputs.pr_number }}" -ne "") {
# Single-PR run (either /evaluate or evaluate-now label):
# detect individual changed skills using gate outputs
$base = "${{ needs.gate.outputs.base_sha }}"
$head = (git rev-parse origin/pr-head)
# Use a worktree so Test-Path checks are against PR content
git worktree add /tmp/pr-content origin/pr-head 2>$null
$contentRoot = "/tmp/pr-content"
$mergeBase = git merge-base $base $head
$changedFiles = git diff --name-only --diff-filter=ACMR $mergeBase $head
# Check if any changed files are in infrastructure paths. The linter
# (skill-validator) is a separate workflow, so eng/skill-validator/
# changes are NOT infra changes here.
$hasInfraChanges = $changedFiles |
Where-Object {
($_ -match '^eng/vally-adapter/') -or
($_ -match '^dotnet-skills\.experiment\.yaml$') -or
$_ -match '^\.github/workflows/(evaluation|evaluation-run)\.yml$'
} |
Select-Object -First 1
# Also check for skill/test changes so we don't lose them
$hasSkillChanges = $changedFiles |
Where-Object { $_ -match '^(?:plugins/([^/]+)/skills|tests/([^/]+))/([^/]+)/' } |
Select-Object -First 1
if ($hasInfraChanges -and -not $hasSkillChanges) {
echo "is_infra=true" >> $env:GITHUB_OUTPUT
# Infra-only: evaluate a small random subset of plugins to keep
# the smoke-test fast while still catching regressions.
$allPlugins = @(Get-ChildItem -Path (Join-Path $contentRoot "plugins") -Directory -ErrorAction SilentlyContinue |
Where-Object {
(Test-Path (Join-Path $_.FullName "skills")) -and
(Test-Path (Join-Path $contentRoot "tests" $_.Name))
} |
Select-Object -ExpandProperty Name)
$plugins = @($allPlugins | Get-Random -Count ([Math]::Min(2, $allPlugins.Count)))
Write-Host "Infrastructure changes detected, evaluating random subset: $($plugins -join ', ')"
$entries = @($plugins | ForEach-Object { Get-PluginShardEntries -plugin $_ -contentRoot $contentRoot })
} else {
# Extract unique plugin/skill pairs from changed files
$changedPairs = @($changedFiles |
Where-Object { $_ -match '^(?:plugins/([^/]+)/skills|tests/([^/]+))/([^/]+)/' } |
ForEach-Object {
$p = if ($Matches[1]) { $Matches[1] } else { $Matches[2] }
"$p/$($Matches[3])"
} |
Sort-Object -Unique)
# Filter to skills that have a SKILL.md and a tests directory,
# then group the changed skills by plugin and apply the same
# executionShard bucketing used by schedule/full-plugin runs.
$changedSkillsByPlugin = @{}
$changedPairs | ForEach-Object {
$parts = $_ -split '/'
$plugin = $parts[0]
$skill = $parts[1]
$skillMd = Join-Path $contentRoot "plugins" $plugin "skills" $skill "SKILL.md"
$testsDir = Join-Path $contentRoot "tests" $plugin
if ((Test-Path $skillMd) -and (Test-Path $testsDir)) {
if (-not $changedSkillsByPlugin.ContainsKey($plugin)) { $changedSkillsByPlugin[$plugin] = @() }
$changedSkillsByPlugin[$plugin] += $skill
}
}
$entries = @($changedSkillsByPlugin.Keys | Sort-Object | ForEach-Object {
$plugin = $_
Get-PluginShardEntries -plugin $plugin -contentRoot $contentRoot -selectedSkills ($changedSkillsByPlugin[$plugin] | Sort-Object -Unique)
})
$plugins = @($entries | ForEach-Object { $_.plugin } | Sort-Object -Unique)
}
git worktree remove /tmp/pr-content --force 2>$null
} else {
# Schedule and workflow_dispatch: evaluate full plugins.
# Schedule covers everything; workflow_dispatch can optionally
# narrow to a single plugin via the `plugin` input for ad-hoc
# smoke tests without committing to main.
# Exclude experimental plugins from scheduled runs — they are
# evaluated only on-demand via /evaluate on PRs.
$excludeFromSchedule = @('dotnet-experimental')
$dispatchPlugin = "${{ inputs.plugin }}"
if ("${{ github.event_name }}" -eq "workflow_dispatch" -and $dispatchPlugin) {
# Restrict to a simple directory-name shape before any path
# construction, so values like '../foo' or 'foo/bar' cannot
# traverse outside plugins/ or tests/.
if ($dispatchPlugin -notmatch '^[a-zA-Z0-9._-]+$') {
throw "workflow_dispatch input plugin='$dispatchPlugin' must match ^[a-zA-Z0-9._-]+$ (single directory name, no path separators)"
}
if (-not (Test-Path (Join-Path "plugins" $dispatchPlugin "skills")) -or
-not (Test-Path (Join-Path "tests" $dispatchPlugin))) {
throw "workflow_dispatch input plugin='$dispatchPlugin' is not a valid plugin (must have plugins/<name>/skills and tests/<name>)"
}
$plugins = @($dispatchPlugin)
Write-Host "workflow_dispatch: evaluating only $dispatchPlugin"
} else {
$plugins = @(Get-ChildItem -Path "plugins" -Directory |
Where-Object { (Test-Path (Join-Path $_.FullName "skills")) -and (Test-Path (Join-Path "tests" $_.Name)) -and ($_.Name -notin $excludeFromSchedule) } |
Select-Object -ExpandProperty Name)
}
$entries = @($plugins | ForEach-Object { Get-PluginShardEntries -plugin $_ })
}
# Validate every entry against a strict allowlist before it enters
# the matrix consumed by the vally evaluation. A lone-dot
# component ('.') matches the charset and holds no '..', but it is a
# path-normalization token, so reject it for names and path segments.
$namePattern = '\A[A-Za-z0-9._-]+\z'
$pathPattern = '\Aplugins/[A-Za-z0-9._-]+/skills(/[A-Za-z0-9._-]+)?\z'
foreach ($e in $entries) {
if ("$($e.plugin)" -notmatch $namePattern -or "$($e.plugin)" -match '\.\.' -or "$($e.plugin)" -eq '.') {
throw "Refusing unsafe matrix entry: plugin '$($e.plugin)' must match $namePattern, not be '.', and not contain '..'"
}
if ("$($e.name)" -notmatch $namePattern -or "$($e.name)" -match '\.\.' -or "$($e.name)" -eq '.') {
throw "Refusing unsafe matrix entry: name '$($e.name)' must match $namePattern, not be '.', and not contain '..'"
}
# skills_path is a space-separated list of one or more path segments.
# An empty or whitespace-only value must hard-fail: it is later used
# to build CLI arguments, so a blank entry is a configuration error,
# not a no-op. Filter on the trimmed value so whitespace-only
# segments count as empty (matching the bash validation step).
$spSegments = @("$($e.skills_path)" -split ' ' | Where-Object { $_.Trim() })
if ($spSegments.Count -eq 0) {
throw "Refusing unsafe matrix entry: skills_path for entry '$($e.name)' is empty"
}
# Each segment must also belong to THIS entry's plugin: $pathPattern
# only checks the generic shape, so bind the prefix to $e.plugin via
# ordinal string comparison. This blocks a mismatched entry such as
# plugin=foo + skills_path=plugins/bar/skills, which downstream steps
# assume shares the entry's plugin prefix.
$skillsPrefix = "plugins/$($e.plugin)/skills"
foreach ($sp in $spSegments) {
if ($sp -notmatch $pathPattern -or $sp -match '\.\.') {
throw "Refusing unsafe matrix entry: skills_path segment '$sp' must match $pathPattern and not contain '..'"
}
# Reject a lone-dot path component ('plugins/./skills' or '.../skills/.').
if ($sp -match '/\./' -or $sp -match '/\.$') {
throw "Refusing unsafe matrix entry: skills_path segment '$sp' contains a '.' path component"
}
if ($sp -cne $skillsPrefix -and -not $sp.StartsWith("$skillsPrefix/", [System.StringComparison]::Ordinal)) {
throw "Refusing unsafe matrix entry: skills_path segment '$sp' does not belong to plugin '$($e.plugin)'"
}
}
}
# Output entries for evaluate matrix
if (-not $entries -or $entries.Count -eq 0) {
Write-Host "No entries to evaluate"
echo "entries=[]" >> $env:GITHUB_OUTPUT
echo "has_entries=false" >> $env:GITHUB_OUTPUT
} else {
$json = $entries | ConvertTo-Json -Compress -AsArray
Write-Host "Entries to evaluate: $json"
echo "entries=$json" >> $env:GITHUB_OUTPUT
echo "has_entries=true" >> $env:GITHUB_OUTPUT
}
# Output plugins for publish jobs
if (-not $plugins -or $plugins.Count -eq 0) {
echo "plugins=[]" >> $env:GITHUB_OUTPUT
echo "has_plugins=false" >> $env:GITHUB_OUTPUT
} else {
$cjson = $plugins | ConvertTo-Json -Compress -AsArray
echo "plugins=$cjson" >> $env:GITHUB_OUTPUT
echo "has_plugins=true" >> $env:GITHUB_OUTPUT
}
shell: pwsh
# ==========================================================================
# EVALUATE
# Run the eval harness across changed skills via the reusable
# evaluation-run.yml workflow. This is the sole LLM eval engine — it
# produces the vally-results-* artifacts every downstream job consumes.
# (The static-analysis linter is a separate workflow, skill-check.yml.)
# ==========================================================================
evaluate:
needs: [gate, discover]
if: >-
always() &&
needs.discover.outputs.has_entries == 'true' &&
needs.discover.result == 'success' &&
(needs.gate.result == 'success' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch')
uses: ./.github/workflows/evaluation-run.yml
with:
entries: ${{ needs.discover.outputs.entries }}
head_sha: ${{ needs.gate.outputs.head_sha }}
# Forward ONLY the Copilot PAT pool across the workflow_call boundary. A
# reusable workflow does not inherit the caller's org/repo secrets, so the
# pool would read empty on this path unless it is passed explicitly. We map
# the 10 named tokens rather than using `secrets: inherit` — inherit would
# forward every repo/org secret into the reusable workflow and reverse the
# blast-radius reduction from #868; named mapping crosses only these tokens.
# This job is a top-level caller, so `secrets.COPILOT_PAT_*` resolves from the
# existing ORG secrets — the same pool the scheduled validate-pat-pool workflow
# reads — so no `copilot-pat-pool` environment-secret configuration is needed.
secrets:
COPILOT_PAT_0: ${{ secrets.COPILOT_PAT_0 }}
COPILOT_PAT_1: ${{ secrets.COPILOT_PAT_1 }}
COPILOT_PAT_2: ${{ secrets.COPILOT_PAT_2 }}
COPILOT_PAT_3: ${{ secrets.COPILOT_PAT_3 }}
COPILOT_PAT_4: ${{ secrets.COPILOT_PAT_4 }}
COPILOT_PAT_5: ${{ secrets.COPILOT_PAT_5 }}
COPILOT_PAT_6: ${{ secrets.COPILOT_PAT_6 }}
COPILOT_PAT_7: ${{ secrets.COPILOT_PAT_7 }}
COPILOT_PAT_8: ${{ secrets.COPILOT_PAT_8 }}
COPILOT_PAT_9: ${{ secrets.COPILOT_PAT_9 }}
# ==========================================================================
# COMMENT ON PR
# Post consolidated evaluation results as a PR comment.
# ==========================================================================
comment-on-pr:
needs: [gate, discover, evaluate, publish-session-data]
if: >-
always() &&
needs.evaluate.result != 'cancelled' &&
needs.gate.outputs.pr_number != '' &&
needs.discover.outputs.has_entries == 'true'
runs-on: ubuntu-latest
permissions:
pull-requests: write
steps:
- name: Checkout adapter tooling
if: needs.evaluate.result != 'skipped'
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v6
with:
persist-credentials: false
sparse-checkout: |
eng/vally-adapter
- name: Download all result artifacts
if: needs.evaluate.result != 'skipped'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
pattern: vally-results-*
path: all-results/
merge-multiple: false
continue-on-error: ${{ needs.evaluate.result != 'success' }}
- name: Consolidate and post results
continue-on-error: true
env:
GH_TOKEN: ${{ github.token }}
run: |
PR_NUMBER=${{ needs.gate.outputs.pr_number }}
RUN_URL="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
# If results exist, consolidate them into a summary comment
JSON_FILES=$(find all-results/ -name results.json 2>/dev/null || true)
if [ -n "$JSON_FILES" ]; then
# Full table (all columns incl. Quality (Plugin)) for the step summary,
# simplified table (Quality (Plugin) dropped) for the PR comment.
node eng/vally-adapter/consolidate.mjs --format full --root all-results/ --output summary-body.md
node eng/vally-adapter/consolidate.mjs --format simple --root all-results/ --output simplified-body.md
INVESTIGATE_PROMPT=""
# If any skill failed, build a copy-paste prompt for AI-assisted investigation
if jq -se '[.[].verdicts[] | select(.passed == false)] | length > 0' $JSON_FILES > /dev/null 2>&1; then
RUN_ID="${{ github.run_id }}"
INVESTIGATE_PROMPT=$(printf '\n> **To investigate failures**, paste this to your AI coding agent:\n>\n> _For PR %s in %s, download eval artifacts with `gh run download %s --repo %s --pattern "vally-results-*" --dir ./eval-results`, then fetch https://raw.githubusercontent.com/%s/%s/eng/vally-adapter/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first._' \
"${PR_NUMBER}" "${{ github.repository }}" "${RUN_ID}" "${{ github.repository }}" "${{ github.repository }}" "${{ needs.gate.outputs.head_sha }}")
fi
# PR comment: simplified table + "Full results" link + investigation prompt
{
cat simplified-body.md
echo ""
echo "[🔍 Full Results - additional metrics and failure investigation steps]($RUN_URL)"
if [ -n "$INVESTIGATE_PROMPT" ]; then
echo "$INVESTIGATE_PROMPT"
fi
} > consolidated-comment.md
# Append AGENTVIZ replay link if session data was published
if [[ "${{ needs.publish-session-data.result }}" == "success" ]]; then
# Session data lives in the standalone dotnet/skills-data repo.
MANIFEST_URL="https://raw.githubusercontent.com/dotnet/skills-data/dashboard-session-data/data/manifest.json"
# Derive the Pages base URL from the repository (org.github.io/repo)
ORG=$(echo "${{ github.repository }}" | cut -d/ -f1)
REPO=$(echo "${{ github.repository }}" | cut -d/ -f2)
REPLAY_URL="https://${ORG}.github.io/${REPO}/replay/index.html"
MANIFEST_ENCODED=$(printf '%s' "$MANIFEST_URL" | jq -sRr @uri)
FULL_URL="${REPLAY_URL}?manifest=${MANIFEST_ENCODED}&tag=pr-${PR_NUMBER}"
echo "" >> consolidated-comment.md
echo "**[▶ Sessions Visualisation](${FULL_URL})** -- interactive replay of all evaluation sessions" >> consolidated-comment.md
ANALYTICS_BASE_URL="https://server.mangowater-996ff7b2.swedencentral.azurecontainerapps.io/"
FACET_ENCODED=$(printf '%s' "label:pr-${PR_NUMBER}" | jq -sRr @uri)
ANALYTICS_URL="${ANALYTICS_BASE_URL}?facet=${FACET_ENCODED}"
echo "**[📊 Session Analytics (preview)](${ANALYTICS_URL})** -- aggregated metrics across evaluation sessions" >> consolidated-comment.md
fi
# Action summary: full table (all columns) + investigation prompt
{
cat summary-body.md
if [ -n "$INVESTIGATE_PROMPT" ]; then
echo "$INVESTIGATE_PROMPT"
fi
} >> "$GITHUB_STEP_SUMMARY"
gh api "repos/${{ github.repository }}/issues/${PR_NUMBER}/comments" \
-X POST -F "body=@consolidated-comment.md"
else
# No results — evaluate was skipped or failed before producing artifacts
if [[ "${{ needs.evaluate.result }}" == "skipped" ]]; then
BODY="❌ Evaluation did not complete (upstream job failed or was skipped). [View workflow run](${RUN_URL})"
else
BODY="❌ Evaluation failed. [View workflow run](${RUN_URL})"
fi
gh api "repos/${{ github.repository }}/issues/${PR_NUMBER}/comments" -X POST -f body="$BODY"
fi
# ==========================================================================
# REPORT STATUS
# Post final evaluation status via commit status API.
# ==========================================================================
report-status:
needs: [gate, discover, evaluate]
if: always() && needs.gate.result == 'success'
runs-on: ubuntu-latest
permissions:
statuses: write
pull-requests: write
issues: write
steps:
- name: Remove eyes reaction from trigger comment
if: github.event_name == 'issue_comment'
env:
GH_TOKEN: ${{ github.token }}
run: |
REACTION_ID=$(gh api "repos/${{ github.repository }}/issues/comments/${{ github.event.comment.id }}/reactions" \
--jq '.[] | select(.content == "eyes" and .user.login == "github-actions[bot]") | .id' | head -1 || echo "")
if [[ -n "$REACTION_ID" && "$REACTION_ID" != "null" ]]; then
gh api "repos/${{ github.repository }}/issues/comments/${{ github.event.comment.id }}/reactions/${REACTION_ID}" \
-X DELETE || true
fi
- name: Set final commit status
env:
GH_TOKEN: ${{ github.token }}
run: |
if [[ "${{ needs.evaluate.result }}" == "success" ]]; then
STATE="success"
DESC="Evaluation passed"
elif [[ "${{ needs.evaluate.result }}" == "skipped" && "${{ needs.discover.result }}" == "success" && "${{ needs.discover.outputs.has_entries }}" != "true" ]]; then
STATE="success"
DESC="No skills to evaluate"
elif [[ "${{ needs.evaluate.result }}" == "failure" ]]; then
STATE="failure"
DESC="Evaluation failed"
else
STATE="error"
DESC="Evaluation did not complete (evaluate: ${{ needs.evaluate.result }}, discover: ${{ needs.discover.result }})"
fi
gh api "repos/${{ github.repository }}/statuses/${{ needs.gate.outputs.head_sha }}" \
-f state="$STATE" \
-f context="evaluation-status" \
-f description="$DESC" \
-f target_url="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
- name: Post completion comment for skipped evaluation
if: needs.evaluate.result == 'skipped'
continue-on-error: true
env:
GH_TOKEN: ${{ github.token }}
run: |
PR_NUMBER=${{ needs.gate.outputs.pr_number }}
RUN_URL="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
if [[ "${{ needs.discover.result }}" == "success" && "${{ needs.discover.outputs.has_entries }}" != "true" ]]; then
BODY="⏭️ No skills to evaluate — no changed skills with tests were found in this PR. [View workflow run](${RUN_URL})"
else
BODY="❌ Evaluation did not complete (upstream job failed or was skipped). [View workflow run](${RUN_URL})"
fi
gh api "repos/${{ github.repository }}/issues/${PR_NUMBER}/comments" -X POST -f body="$BODY"
# ==========================================================================
# PUBLISH TOKEN DATA
# Both scheduled and PR runs: generate token-usage data → dashboard-token-data
# ==========================================================================
publish-token-data:
needs: [gate, discover, evaluate]
if: >-
always() &&
!cancelled() &&
needs.discover.result == 'success' &&
needs.discover.outputs.has_plugins == 'true' &&
(
github.ref == 'refs/heads/main' ||
needs.gate.result == 'success'
)
concurrency:
group: publish-token-data
cancel-in-progress: false
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v6
with:
persist-credentials: false
- name: Download evaluation artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
pattern: vally-results-*
path: all-results/
merge-multiple: false
continue-on-error: ${{ needs.evaluate.result != 'success' }}
- name: Fetch existing token data from dashboard-token-data
run: |
git fetch origin dashboard-token-data:dashboard-token-data 2>/dev/null || true
mkdir -p /tmp/token-data/data
git checkout dashboard-token-data -- data/token-usage.json 2>/dev/null && \
cp data/token-usage.json /tmp/token-data/data/ && \
git checkout HEAD -- . || true
- name: Get PR title
if: needs.gate.outputs.pr_number != ''
id: pr-info
env:
GH_TOKEN: ${{ github.token }}
run: |
PR_TITLE=$(gh api "repos/${{ github.repository }}/pulls/${{ needs.gate.outputs.pr_number }}" --jq '.title' 2>/dev/null) || PR_TITLE=""
echo "pr_title=${PR_TITLE}" >> $GITHUB_OUTPUT
- name: Generate token usage data
env:
PR_TITLE: ${{ steps.pr-info.outputs.pr_title }}
run: |
$source = if ("${{ needs.gate.outputs.pr_number }}" -ne "") { "pr" } else { "scheduled" }
$plugins = '${{ needs.discover.outputs.plugins }}' | ConvertFrom-Json
New-Item -ItemType Directory -Force -Path "/tmp/combined" | Out-Null
foreach ($plugin in $plugins) {
# Vally artifacts are named vally-results-$plugin (scheduled/infra) or
# vally-results-${plugin}--${skill} (individual skill-change PRs). The
# adapter writes one results.json per skill at <plugin>/<skill>/results.json
# inside each artifact, so gather every results.json for this plugin and
# merge their verdicts into a single file (generate-benchmark-data.ps1
# expects one file whose verdicts cover the whole plugin).
$artifactDirs = @(Get-ChildItem -Path "all-results" -Directory -ErrorAction SilentlyContinue |
Where-Object { $_.Name -eq "vally-results-$plugin" -or $_.Name -like "vally-results-$plugin--*" })
if ($artifactDirs.Count -eq 0) {
Write-Warning "No artifacts found for $plugin, skipping"
continue
}
$resultFiles = @($artifactDirs | ForEach-Object {
Get-ChildItem -Path $_.FullName -Recurse -Filter "results.json" -File -ErrorAction SilentlyContinue
})
if ($resultFiles.Count -eq 0) {
Write-Warning "No results.json found for $plugin, skipping"
continue
}
$parsed = @($resultFiles | ForEach-Object { Get-Content $_.FullName -Raw | ConvertFrom-Json })
$verdicts = @($parsed | ForEach-Object { $_.verdicts })
if ($verdicts.Count -eq 0) {
Write-Warning "No verdicts found for $plugin, skipping"
continue
}
$combined = [ordered]@{
model = $parsed[0].model
judgeModel = $parsed[0].judgeModel
timestamp = $parsed[0].timestamp
verdicts = $verdicts
}
$combinedFile = "/tmp/combined/$plugin.results.json"
$combined | ConvertTo-Json -Depth 30 | Out-File -FilePath $combinedFile -Encoding utf8
Write-Host "`n=== Collecting token usage for: $plugin (from $($resultFiles.Count) results.json across $($artifactDirs.Count) artifact(s)) ==="
$params = @{
ResultsFile = $combinedFile
PluginName = $plugin
OutputDir = "/tmp/token-data/data"
Source = $source
RetentionDays = $env:DASHBOARD_RETENTION_DAYS
}
if ($source -eq "pr") {
$params.PRNumber = ${{ needs.gate.outputs.pr_number }}
$params.PRTitle = $env:PR_TITLE
}
$params.SkipBenchmarkData = $true
& ./eng/dashboard/generate-benchmark-data.ps1 @params
}
shell: pwsh
- name: Push to dashboard-token-data branch
run: |
cd /tmp
REPO_URL="https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}.git"
if git ls-remote --exit-code --heads "$REPO_URL" dashboard-token-data > /dev/null 2>&1; then
git clone --branch dashboard-token-data --single-branch "$REPO_URL" token-deploy
else
mkdir token-deploy && cd token-deploy && git init && git checkout -b dashboard-token-data
git remote add origin "$REPO_URL"
cd /tmp
fi
cd /tmp/token-deploy
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
# Remove any stale files so this branch stays token-only
git rm -rf data/ --ignore-unmatch --quiet 2>/dev/null || true
mkdir -p data
# Ensure token-usage.json exists; create an empty fallback if missing
if [ ! -f /tmp/token-data/data/token-usage.json ]; then
mkdir -p /tmp/token-data/data
printf '{ "entries": [] }\n' > /tmp/token-data/data/token-usage.json
fi
cp /tmp/token-data/data/token-usage.json data/
git add data/token-usage.json
git diff --cached --quiet && echo "No changes to deploy" && exit 0
if [[ -n "${{ needs.gate.outputs.pr_number }}" ]]; then
git commit -m "Update PR token usage data (PR #${{ needs.gate.outputs.pr_number }})"
else
git commit -m "Update scheduled token usage data"
fi
git push origin dashboard-token-data
# ==========================================================================
# PUBLISH SESSION DATA
# Both scheduled and PR runs: flatten JSONL sessions → dashboard-session-data
# branch on dotnet/skills-data (separate repo to keep this one small).
# ==========================================================================
publish-session-data:
needs: [gate, discover, evaluate]
if: >-
always() && !cancelled() &&
needs.discover.result == 'success' &&
needs.discover.outputs.has_plugins == 'true' &&
(
github.ref == 'refs/heads/main' ||
needs.gate.result == 'success'
)
runs-on: ubuntu-latest
concurrency:
group: publish-session-data
cancel-in-progress: false
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v6
with:
persist-credentials: false
- name: Download evaluation artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
pattern: vally-results-*
path: all-results/
merge-multiple: false
continue-on-error: ${{ needs.evaluate.result != 'success' }}
- name: Inspect downloaded artifacts
if: always()
run: |
echo "=== all-results/ directory listing (3 levels) ==="
if [ -d all-results ]; then
find all-results -maxdepth 3 -ls 2>/dev/null | head -80
echo "---"
echo "executor-session-logs dirs:"
find all-results -type d -name 'executor-session-logs' 2>/dev/null
echo "session metadata.json files:"
find all-results -path '*executor-session-logs*' -name 'metadata.json' 2>/dev/null | head -20
echo "events.jsonl files:"
find all-results -path '*executor-session-logs*' -name 'events.jsonl' 2>/dev/null | head -20
else
echo "all-results/ directory does not exist!"
fi
- name: Determine source metadata
id: meta
run: |
if [ -n "${{ needs.gate.outputs.pr_number }}" ]; then
echo "source=pr" >> "$GITHUB_OUTPUT"
echo "pr_number=${{ needs.gate.outputs.pr_number }}" >> "$GITHUB_OUTPUT"
echo "subdir=pr/${{ needs.gate.outputs.pr_number }}" >> "$GITHUB_OUTPUT"
else
echo "source=scheduled" >> "$GITHUB_OUTPUT"
echo "pr_number=" >> "$GITHUB_OUTPUT"
echo "subdir=scheduled/$(date -u +%Y-%m-%d)" >> "$GITHUB_OUTPUT"
fi
- name: Build session manifest
shell: pwsh
run: |
./eng/dashboard/build-replay-sessions.ps1 `
-ResultsDir all-results `
-OutputDir staging `
-Source ${{ steps.meta.outputs.source }} `
-PrNumber "${{ steps.meta.outputs.pr_number }}"
- name: Clone existing session data branch
env:
# Cross-repo PAT with contents:write on dotnet/skills-data.
# GITHUB_TOKEN cannot push to a different repo, so a fine-grained PAT is required.
SKILLS_DATA_TOKEN: ${{ secrets.SKILLS_DATA_TOKEN }}
run: |
cd /tmp
REPO_URL="https://x-access-token:${SKILLS_DATA_TOKEN}@github.com/dotnet/skills-data.git"
if git ls-remote --exit-code --heads "$REPO_URL" dashboard-session-data > /dev/null 2>&1; then
git clone --branch dashboard-session-data --single-branch --depth 1 "$REPO_URL" session-deploy
else
mkdir session-deploy && cd session-deploy && git init && git checkout -b dashboard-session-data
git remote add origin "$REPO_URL"
cd /tmp
fi
cd /tmp/session-deploy
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
- name: Merge and purge old sessions
shell: pwsh
run: |
./eng/dashboard/purge-replay-sessions.ps1 `
-ExistingDir /tmp/session-deploy/data `
-NewDir staging `
-OutputDir /tmp/session-deploy/data `
-RetentionDays 7
- name: Push to dashboard-session-data branch (dotnet/skills-data)
run: |
cd /tmp/session-deploy
git add data/
git diff --cached --quiet && echo "No changes to deploy" && exit 0
if [[ -n "${{ needs.gate.outputs.pr_number }}" ]]; then
git commit -m "Update session data (PR #${{ needs.gate.outputs.pr_number }})"
else
git commit -m "Update scheduled session data"
fi
git push origin dashboard-session-data
# ==========================================================================
# PUBLISH EVAL DATA
# Scheduled runs only: generate benchmark data → dashboard-eval-data
# ==========================================================================
publish-eval-data:
needs: [gate, discover, evaluate]
if: >-
always() &&
!cancelled() &&
needs.discover.result == 'success' &&
needs.discover.outputs.has_plugins == 'true' &&
github.event_name == 'schedule'
concurrency:
group: publish-eval-data
cancel-in-progress: false
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v6
with:
persist-credentials: false
- name: Download evaluation artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
pattern: vally-results-*
path: all-results/
merge-multiple: false
continue-on-error: ${{ needs.evaluate.result != 'success' }}
- name: Fetch existing eval data from dashboard-eval-data
run: |
git fetch origin dashboard-eval-data:dashboard-eval-data 2>/dev/null || true
mkdir -p /tmp/eval-data/data
git checkout dashboard-eval-data -- data/ 2>/dev/null && \
cp -r data/* /tmp/eval-data/data/ && \
git checkout HEAD -- . || true
- name: Generate benchmark data
run: |
$sha = "${{ github.sha }}"
$commitMsg = git log -1 --format='%s' $sha
$commitTimestamp = git log -1 --format='%aI' $sha
$commitAuthor = git log -1 --format='%an' $sha
$commitJson = @{
id = $sha
message = $commitMsg
timestamp = $commitTimestamp
url = "https://github.com/${{ github.repository }}/commit/$sha"
author = @{ name = $commitAuthor; username = "${{ github.actor }}" }
} | ConvertTo-Json -Compress
$plugins = '${{ needs.discover.outputs.plugins }}' | ConvertFrom-Json
foreach ($plugin in $plugins) {
# A plugin may have been fanned out into multiple shards by the
# discover job (artifact names "vally-results-<plugin>--shard-<tag>"),
# or it may be a single artifact ("vally-results-<plugin>"). The
# adapter writes one results.json per skill at <plugin>/<skill>/results.json
# inside each artifact, so gather every results.json across all matching
# artifacts and merge their verdicts into a single results.json — the
# dashboard records ONE datapoint per plugin per scheduled run.
$artifactDirs = @(Get-ChildItem -Path "all-results" -Directory -ErrorAction SilentlyContinue |
Where-Object { $_.Name -eq "vally-results-$plugin" -or $_.Name -like "vally-results-$plugin--*" })
if ($artifactDirs.Count -eq 0) {
Write-Warning "No run results found for $plugin, skipping"
continue
}
$resultsFiles = @($artifactDirs | ForEach-Object {
Get-ChildItem -Path $_.FullName -Recurse -Filter "results.json" -File -ErrorAction SilentlyContinue |
ForEach-Object { $_.FullName }
})
if ($resultsFiles.Count -eq 0) {
Write-Warning "No results.json found in any artifact for $plugin, skipping"
continue
}
if ($resultsFiles.Count -eq 1) {
$resultsFile = $resultsFiles[0]
} else {
# Merge per-skill/per-shard verdicts into a single synthetic
# results.json. Schema: { model, verdicts[], ... }. We concat the
# verdicts arrays and keep top-level scalars from the first file.
Write-Host "Merging $($resultsFiles.Count) results.json files for $plugin"
$merged = $null
$allVerdicts = [System.Collections.Generic.List[object]]::new()
foreach ($rf in ($resultsFiles | Sort-Object)) {
$r = Get-Content $rf -Raw | ConvertFrom-Json
if ($null -eq $merged) { $merged = $r }
if ($r.verdicts) { foreach ($v in $r.verdicts) { $allVerdicts.Add($v) } }
}
$merged.verdicts = $allVerdicts.ToArray()
$mergedDir = "all-results/_merged"
New-Item -ItemType Directory -Force -Path $mergedDir | Out-Null
$resultsFile = Join-Path $mergedDir "$plugin.results.json"
$merged | ConvertTo-Json -Depth 100 | Out-File -FilePath $resultsFile -Encoding utf8
Write-Host " Merged verdict count: $($allVerdicts.Count) -> $resultsFile"
}
Write-Host "`n=== Generating benchmark data for: $plugin ==="
$existingFile = "/tmp/eval-data/data/$plugin.json"
$params = @{
ResultsFile = $resultsFile
PluginName = $plugin
OutputDir = "/tmp/eval-data/data"
CommitJson = $commitJson
RetentionDays = $env:DASHBOARD_RETENTION_DAYS
Source = 'scheduled'
SkipTokenUsage = $true
}
if ((Test-Path $existingFile) -and (Get-Content $existingFile -Raw -ErrorAction SilentlyContinue)) {
$params.ExistingDataFile = $existingFile
}
& ./eng/dashboard/generate-benchmark-data.ps1 @params
}
# Purge entries older than retention window
& ./eng/dashboard/generate-benchmark-data.ps1 -PurgeStaleFiles -DataDir "/tmp/eval-data/data" -RetentionDays $env:DASHBOARD_RETENTION_DAYS
# Generate components.json manifest (exclude token-usage.json if present)
$plugins = Get-ChildItem -Path "/tmp/eval-data/data" -Filter "*.json" -File -ErrorAction SilentlyContinue |
Where-Object { $_.Name -notin @("components.json", "token-usage.json") } |
ForEach-Object { $_.BaseName }
@($plugins) | ConvertTo-Json -AsArray | Out-File -FilePath "/tmp/eval-data/data/components.json" -Encoding utf8
shell: pwsh
- name: Push to dashboard-eval-data branch
run: |
cd /tmp
REPO_URL="https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}.git"
if git ls-remote --exit-code --heads "$REPO_URL" dashboard-eval-data > /dev/null 2>&1; then
git clone --branch dashboard-eval-data --single-branch "$REPO_URL" eval-deploy
else
mkdir eval-deploy && cd eval-deploy && git init && git checkout -b dashboard-eval-data
git remote add origin "$REPO_URL"
cd /tmp
fi
cd /tmp/eval-deploy
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
# Remove stale files so this branch only contains current eval data
git rm -rf data/ --ignore-unmatch --quiet 2>/dev/null || true
mkdir -p data
# Copy only eval data files (not token-usage.json)
cp /tmp/eval-data/data/components.json data/
for f in /tmp/eval-data/data/*.json; do
fname=$(basename "$f")
[ "$fname" = "token-usage.json" ] && continue
cp "$f" "data/$fname"
done
git add data/
git diff --cached --quiet && echo "No changes to deploy" && exit 0
git commit -m "Update benchmark data"
git push origin dashboard-eval-data
# ==========================================================================
# DEPLOY DASHBOARD
# Scheduled runs + manual dispatch on main: assemble data from both
# branches + UI → gh-pages. No data generation — pure copy and deploy.
# ==========================================================================
deploy-dashboard:
needs: [discover, publish-token-data, publish-eval-data, publish-session-data]
if: >-
always() &&
!cancelled() &&
(
(needs.discover.result == 'success' && needs.discover.outputs.has_plugins == 'true' && github.event_name == 'schedule') ||
(github.event_name == 'workflow_dispatch' && github.ref == 'refs/heads/main')
)
concurrency:
group: deploy-dashboard
cancel-in-progress: false
runs-on: ubuntu-latest
steps:
- name: Checkout repository (for dashboard UI files)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v6
with:
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 20
- name: Fetch eval data from dashboard-eval-data branch
run: |
mkdir -p /tmp/gh-pages/data
git fetch origin dashboard-eval-data:dashboard-eval-data 2>/dev/null || true
git checkout dashboard-eval-data -- data/ 2>/dev/null && \
cp -r data/* /tmp/gh-pages/data/ && \
git checkout HEAD -- . || true
- name: Fetch token data from dashboard-token-data branch
run: |
git fetch origin dashboard-token-data:dashboard-token-data 2>/dev/null || true
git show dashboard-token-data:data/token-usage.json > /tmp/gh-pages/data/token-usage.json 2>/dev/null || \
echo '{"entries":[]}' > /tmp/gh-pages/data/token-usage.json
- name: Check if AGENTVIZ SPA needs update
id: check-replay
run: |
AGENTVIZ_REPO="https://github.com/jayparikh/agentviz.git"
AGENTVIZ_BRANCH="main"
# Resolve the latest commit on the AGENTVIZ branch (no clone needed)
TARGET_SHA=$(git ls-remote "$AGENTVIZ_REPO" "refs/heads/$AGENTVIZ_BRANCH" | cut -f1)
if [ -z "$TARGET_SHA" ]; then
echo "::error::Could not resolve AGENTVIZ branch $AGENTVIZ_BRANCH"
exit 1
fi
echo "target_sha=$TARGET_SHA" >> "$GITHUB_OUTPUT"
echo "AGENTVIZ target commit: $TARGET_SHA"
# Read the currently deployed commit SHA from gh-pages (no clone needed)
DEPLOYED_SHA=""
DEPLOYED_SHA=$(curl -fsSL \
"https://raw.githubusercontent.com/${{ github.repository }}/gh-pages/replay/.agentviz-commit" \
2>/dev/null) || true
echo "deployed_sha=$DEPLOYED_SHA" >> "$GITHUB_OUTPUT"
if [ "$TARGET_SHA" = "$DEPLOYED_SHA" ]; then
echo "skip=true" >> "$GITHUB_OUTPUT"
echo "AGENTVIZ SPA is up-to-date (commit $TARGET_SHA), skipping build."
else
echo "skip=false" >> "$GITHUB_OUTPUT"
echo "AGENTVIZ SPA needs update: deployed=$DEPLOYED_SHA target=$TARGET_SHA"
fi
- name: Build AGENTVIZ SPA
if: steps.check-replay.outputs.skip != 'true'
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v4
id: agentviz-cache
with:
path: /tmp/agentviz-dist
key: agentviz-dist-${{ steps.check-replay.outputs.target_sha }}
- name: Build AGENTVIZ SPA (on cache miss)
if: steps.check-replay.outputs.skip != 'true' && steps.agentviz-cache.outputs.cache-hit != 'true'
run: |
TARGET_SHA="${{ steps.check-replay.outputs.target_sha }}"
git clone https://github.com/jayparikh/agentviz.git /tmp/agentviz-src
cd /tmp/agentviz-src
# Check out the exact resolved commit for deterministic builds
if ! git checkout "$TARGET_SHA"; then
echo "::error::Failed to check out AGENTVIZ commit $TARGET_SHA"
exit 1
fi
npm ci
npm run build
mkdir -p /tmp/agentviz-dist
cp -r dist/* /tmp/agentviz-dist/
- name: Deploy to GitHub Pages
run: |
cd /tmp
REPO_URL="https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}.git"
if git ls-remote --exit-code --heads "$REPO_URL" gh-pages > /dev/null 2>&1; then
git clone --branch gh-pages --single-branch "$REPO_URL" deploy
else
mkdir deploy && cd deploy && git init && git checkout -b gh-pages
git remote add origin "$REPO_URL"
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
cd /tmp
fi
cd /tmp/deploy
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
# Copy data from both branches
mkdir -p data
cp /tmp/gh-pages/data/*.json data/
# Copy dashboard UI from source tree
cp ${{ github.workspace }}/eng/dashboard/dashboard.html index.html
cp ${{ github.workspace }}/eng/dashboard/dashboard.js dashboard.js
cp ${{ github.workspace }}/eng/dashboard/token-usage.js token-usage.js
# Deploy AGENTVIZ SPA (skip if already present and unchanged)
if [ "${{ steps.check-replay.outputs.skip }}" != "true" ]; then
rm -rf replay
mkdir -p replay
if [ -d /tmp/agentviz-dist ]; then
cp -r /tmp/agentviz-dist/* replay/
else
echo "::error::No AGENTVIZ build artifacts found"
exit 1
fi
echo "${{ steps.check-replay.outputs.target_sha }}" > replay/.agentviz-commit
fi
git add .
git diff --cached --quiet && echo "No changes to deploy" && exit 0
git commit -m "Update dashboard"
git push origin gh-pages
Reference in New Issue View Git Blame Copy Permalink