Commit Graph

161 Commits

Author SHA1 Message Date
Jan Krivanek 0e0ec7115e Merge branch 'main' into dev/jankrivanek/eval-dispatch-plugin-input 2026-06-03 16:15:05 +02:00
hngye02 df4578950d Add session viz link to evaluation.yml (#701)
Co-authored-by: t-hungnguyen <t-hungnguyen@microsoft.com>
2026-06-01 10:25:14 +00:00
dependabot[bot] b2ccf6dcab Bump github/gh-aw-actions in the github-actions-dependencies group (#696)
Bumps the github-actions-dependencies group with 1 update: [github/gh-aw-actions](https://github.com/github/gh-aw-actions).


Updates `github/gh-aw-actions` from 0.68.3 to 0.76.1
- [Release notes](https://github.com/github/gh-aw-actions/releases)
- [Changelog](https://github.com/github/gh-aw-actions/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/gh-aw-actions/compare/abea67e08ee83539ea33aaae67bf0cddaa0b03b5...46d564922b082d0db93244972e8005ea6904ee5f)

---
updated-dependencies:
- dependency-name: github/gh-aw-actions
  dependency-version: 0.76.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-28 12:21:05 +02:00
Jan Krivanek 053792ce8e Aggregate sharded plugin results in publish-eval-data (#698)
The publish-eval-data job iterates discover.outputs.plugins and looks for
"all-results/skill-validator-results-<plugin>", but when the discover job fans a
plugin into multiple shards the artifacts are named
"skill-validator-results-<plugin>--shard-<tag>". The exact-match lookup misses
all of them and the step logs "No run results found for dotnet-msbuild,
skipping" -- which is exactly what the 2026-05-27 scheduled run did (confirmed
in the publish-eval-data logs of run 26484154365).

Fix: locate every matching artifact dir (exact name OR --shard-* suffix),
collect the latest run dir's results.json from each, and -- when there are
multiple -- concat their verdicts arrays into a single synthetic results.json
before calling generate-benchmark-data.ps1. Calling the script per-shard would
have emitted N separate datapoints with the same commit/timestamp and skewed
the dashboard.

publish-token-data and publish-session-data were already shard-tolerant and
need no changes.
2026-05-28 12:20:34 +02:00
Jan Krivanek bd5e7402db Restore investigation prompt to PR comment (#693)
PR #473 moved the failure investigation prompt out of the PR comment and
only into the workflow run summary. This made it harder for users to
discover — they had to click through to the run details to find it.

Move it back to the PR comment so the copy-paste prompt is visible
inline on the PR, while keeping it in the workflow summary as well.
2026-05-27 12:47:23 +00:00
dependabot[bot] a9c35454f2 Bump github/gh-aw-actions in the github-actions-dependencies group (#678)
Bumps the github-actions-dependencies group with 1 update: [github/gh-aw-actions](https://github.com/github/gh-aw-actions).


Updates `github/gh-aw-actions` from ba90f2186d7ad780ec640f364005fa24e797b360 to abea67e08ee83539ea33aaae67bf0cddaa0b03b5
- [Release notes](https://github.com/github/gh-aw-actions/releases)
- [Changelog](https://github.com/github/gh-aw-actions/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/gh-aw-actions/compare/ba90f2186d7ad780ec640f364005fa24e797b360...abea67e08ee83539ea33aaae67bf0cddaa0b03b5)

---
updated-dependencies:
- dependency-name: github/gh-aw-actions
  dependency-version: abea67e08ee83539ea33aaae67bf0cddaa0b03b5
  dependency-type: direct:production
  dependency-group: github-actions-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-26 11:04:16 +02:00
Jan Krivanek 14ee3c90f9 Rework evaluation to allow sharding of resource intensive evaluations (#692)
* Shard plugin evaluation via executionShard tag in eval.yaml

MCP-heavy plugins like dotnet-msbuild reliably exhaust the GitHub-hosted runner host after ~87 min of cumulative MCP traffic (Copilot CLI plus dotnet dnx Microsoft.AITools.BinlogMcp child processes), which has produced no dashboard data for dotnet-msbuild scheduled runs since 2026-05-20. Cutting per-runner concurrency alone was insufficient: at the default 18-way concurrency the runner OOMs in 17-30 min; at 3-way it survives longer but still hits a host-level termination at ~87 min.

Two mitigations, both required:

1. Sharding (this change). The discover step reads an optional top-level 'executionShard:' string from each tests/<plugin>/<skill>/eval.yaml and groups skills by tag, emitting one matrix entry per shard so each shard runs on a fresh runner. Untagged skills fall into a 'default' bucket. Plugins where all skills resolve to a single bucket emit one entry (unchanged behavior). The validator already calls IgnoreUnmatchedProperties so no Models.cs change is needed. Shard names follow '<plugin>--shard-<tag>' so the existing 'skill-validator-results-<plugin>--*' artifact glob keeps working.

2. Per-MCP-plugin concurrency caps (already in place). When plugin.json declares mcpServers, parallelism is forced to 1x3x1 with --judge-timeout 600 to avoid acute OOM within a shard.

Also adds a workflow_dispatch 'plugin' input for targeted re-runs of a single plugin, and tags 10 dotnet-msbuild eval.yamls (heavy/medium); the remaining 8 fall into 'default'. Empirically the three shards complete in parallel in ~30-40 min each, well inside the 180-min job timeout, and successfully upload dashboard data.

* Address PR review: lenient shard regex and validate dispatch plugin input

- executionShard regex now tolerates optional leading whitespace, so an accidentally indented key still groups correctly instead of silently falling back to the default bucket.

- workflow_dispatch 'plugin' input is now validated against ^[a-zA-Z0-9._-]+$ before any Join-Path/Test-Path use, preventing path-traversal style values (e.g. '../foo') from being injected into filesystem paths.
2026-05-26 08:22:40 +00:00
Jan Krivanek d3aaffae4f ci: parameterize evaluation workflow_dispatch with plugin input 2026-05-25 16:21:06 +02:00
Jan Krivanek a913aaab1e Soften MCP-plugin parallelism cap to 1x2x2=4 concurrent (was 1x1x1 serial) 2026-05-25 15:59:56 +02:00
Jan Krivanek 7c238299d5 TEMP: serial-execution for MCP plugins + workflow_dispatch path (dotnet-msbuild only)
Real fix: detect plugins declaring mcpServers and force PARALLEL_SKILLS/SCENARIOS/RUNS=1 to avoid runner OOM when dnx-launched MCP children compete for memory (root cause of dotnet-msbuild scheduled-run failures since 2026-05-20).

TEMP (revert before PR): enable workflow_dispatch through gate/discover/build-validator, hardcoded to evaluate only dotnet-msbuild, for CI validation without merging to main.
2026-05-25 14:38:37 +02:00
Amaury Levé 3c23523092 Fix vally eval isolation: pin baseline to 0 skills, skilled to 1 (#662)
* Fix vally eval isolation: pin baseline to 0 skills, skilled to 1

vally evaluations were not isolating skills properly. In the failing run
https://github.com/dotnet/skills/actions/runs/26021981688, the dotnet-test
`code-testing-agent` eval reported "Found 77 skill(s)" for baseline and
"Found 22 skill(s)" for the skilled run instead of the expected 0 and 1.

Root cause: `vally eval` derives skills via `discoverSkills(skillRoot)`
where `skillRoot = opts.skillDir ?? cwd`:

- Baseline omitted `--skill-dir`, so vally walked up from the repo root
  to `.vally.yaml` and loaded every skill under `plugins/` (77 total).
- The CI workflow's skilled run passed `--skill-dir` pointing at the
  plugin-level `plugins/<plugin>/skills` (from
  `matrix.entry.skills_path`). That directory has no `SKILL.md` at
  its root, so vally's Step-4 child scan picked up every sibling skill in
  the plugin (22 for dotnet-test).

Fix applied to both `eng/vally-adapter/run-vally-evals.sh` and
`.github/workflows/vally-evaluation.yml`:

- Baseline now passes `--skill-dir <empty mktemp dir>` so vally
  discovers 0 skills.
- Skilled now uses `plugins/<plugin>/skills/<eval-name>` (the specific
  skill directory, not the plugin-level one) so vally matches Step-1
  (SKILL.md at root) and discovers exactly 1 skill.
- Evals with no matching `SKILL.md` (the `agent.*` orchestrator evals
  that declare their own `environment.skills`) are explicitly skipped
  with a warning -- baseline-vs-skilled isolation is not meaningful for
  them, since they require multi-skill setup by design.
- `Find eval specs` now narrows to a single eval when
  `workflow_dispatch` is invoked with a specific skill, so single-skill
  dispatches do not accidentally run every eval in the plugin.
- Empty temp dirs are created per-eval and cleaned up after each run to
  avoid any cross-run contention under parallel execution.

After the fix the CI log should report `Found 0 skill(s)` for baseline
and `Found 1 skill(s): <skill-name>` for skilled.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review: trap-based cleanup for EMPTY_SKILL_DIR; fail-loud on missing dispatch skill

- run-vally-evals.sh: register a RETURN trap immediately after creating
  EMPTY_SKILL_DIR so it is removed on every exit path, including aborts
  triggered by set -euo pipefail (e.g. if
ode adapt.mjs exits
  non-zero inside the log-capture block). Removes the unconditional
  rm at the end of the function which would not run on errors.

- vally-evaluation.yml: when workflow_dispatch supplies a skill input
  but no 	ests/<plugin>/<skill>/eval.vally.yaml exists, emit an
  explicit ::error:: that includes the missing candidate path and fail
  the job, instead of silently falling through to the generic
  `No eval.vally.yaml files found for <plugin>` warning.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-18 12:30:02 +00:00
Amaury Levé 0f31228b58 Fix eval timeouts and skill activation for dotnet-test evals (#641)
* Fix evaluation timeouts and improve prompts for dotnet-test skills

- writing-mstest-tests: increase global vally timeout from 3m to 5m to match
  the 300s eval.yaml timeout for the complex 'Modernize legacy test patterns' stimulus
- code-testing-agent: increase timeout from 40m/2400s to 60m/3600s
- mtp-hot-reload: increase timeout from 6m to 10m and remove overly strict
  expect_tools constraint on launchSettings stimulus
- migrate-mstest-v1v2-to-v3: rephrase prompt for assertion overload errors to
  mention CS1501 and cover AreNotEqual/AreSame alongside AreEqual

* Increase eval timeouts for mtp-hot-reload and migrate-mstest-v1v2-to-v3 scenarios

* Strengthen activation prompts and add missing vally scenarios

- migrate-mstest-v1v2-to-v3: Add explicit 'MSTest v2 to v3' migration
  language in .testsettings and DataRow prompts (both eval.yaml and
  eval.vally.yaml)
- writing-mstest-tests: Add 'MSTest' keyword and skill-specific
  terminology to all prompts that lacked activation signal
- writing-mstest-tests: Add 5 missing vally scenarios (string
  assertions, comparison assertions, collection/null/reference
  assertions, conditional execution, parallelization)
- Add service-registry fixture files for the collection assertions
  scenario

* Improve skill activation via SKILL.md descriptions and prompt alignment

Three-pronged fix for systematic NOT ACTIVATED failures:

1. SKILL.md descriptions: Restructure writing-mstest-tests, migrate-
   mstest-v1v2-to-v3, and code-testing-agent descriptions to use the
   proven 'Use when user says' pattern with quoted trigger phrases
   (matching the working run-tests pattern)

2. Vally prompts: Each prompt now contains at least one exact quoted
   trigger phrase from its SKILL.md description to ensure strong
   description-to-prompt matching

3. Specific fixes:
   - writing-mstest-tests: All 15 prompts now include MSTest-specific
     trigger phrases (Assert.ThrowsExactly, sealed test class, DataRow,
     DynamicData, Assert.HasCount, Assert.IsInstanceOfType, etc.)
   - migrate-mstest-v1v2-to-v3: Add '.NET 5 dropped' and 'framework
     compatibility issues' to trigger phrases; strengthen dropped-TFM
     prompt
   - code-testing-agent: Add 'generate comprehensive unit tests' and
     'achieve high code coverage' to trigger phrases

* Revert SKILL.md description changes that caused activation regression

The description restructuring in 67056d8ec broke skill activation:
- migrate-mstest-v1v2-to-v3: went from 6/9 activating to 0/9
- writing-mstest-tests: lost isolated activation
- code-testing-agent: unnecessary change to working description

Restore all three SKILL.md files to their pre-regression state.
Keep the vally prompt improvements from 259727e.

* Fix writing-mstest-tests plugin-mode activation

- Add MSTest carve-out to code-testing-agent DO NOT USE FOR to prevent
  it from intercepting MSTest-specific prompts
- Add test parallelization/MSTest.Sdk keywords to writing-mstest-tests
  description (scenario 15 failed even in isolated mode)
- Strengthen prompts to be more MSTest-specific:
  - Scenario 1: emphasize MSTest patterns over generic 'write tests'
  - Scenario 4: remove vague 'something seems off', directly ask to
    fix swapped Assert.AreEqual arguments
  - Scenario 5: reframe as modernization request with specific APIs
  - Scenario 13: name specific MSTest assertion APIs in the prompt

* Fix writing-mstest-tests description over 1024 char limit

Shorten the USE FOR section by removing redundant 'help me write
comprehensive tests' and condensing the assertion API list.
Description is now 944 chars (limit: 1024).

* Increase vally job timeout from 60m to 180m

The code-testing-agent eval has a 60m per-stimulus timeout, and vally
runs multiple configurations (baseline, isolated, plugin). The 60m
job-level timeout was causing the job to be cancelled mid-evaluation.

Match the 180m timeout from the evaluate job in evaluation.yml.

* Address PR review comments

- Add IsPackable=false to writing-mstest-tests fixture csproj for
  consistency with other dotnet-test fixture projects
- Fix grammar in migrate-mstest-v1v2-to-v3 eval prompts: add comma
  after 'upgrade' and add 'file' for clarity (both eval.yaml and
  eval.vally.yaml)

* Trim dotnet-test aggregate description size under 15,000 limit

Consolidate duplicate filter keyword mentions in migrate-vstest-to-mtp
and shorten the cross-reference in migrate-xunit-to-xunit-v3 while
keeping the --filter-class/--filter-trait/--filter-query trigger
keywords for skill activation.

* Fix vally skill activation for 9 failing eval scenarios

Improve eval.vally.yaml prompts that were too specific, causing the model
to answer from general knowledge without activating the skill:

- migrate-mstest-v1v2-to-v3: Remove leading question that gives away the
  answer ('Is .NET 5 dropped?'); use multi-target fixture matching eval.yaml
- writing-mstest-tests (7 scenarios): Add 'following MSTest best practices'
  to trigger skill reading; make prompts more open-ended (don't name exact
  APIs like OSCondition, Assert.AreSame); match eval.yaml prompt style
- code-testing-agent: Clarify app complexity to encourage skill pipeline

The pattern: vally prompts that give away exact API names or answers let
the model skip reading the skill. More open-ended prompts referencing
'best practices' trigger skill activation.

* Improve skills and eval prompts for regression scenarios

migrate-xunit-to-xunit-v3 SKILL.md:
- Restructure from 16 steps to 13, merging redundant steps
- Move test platform selection from step 12 to step 4
- Mark steps 6-12 as conditional with prioritization note
- Remove external URL references that caused web_fetch detours

writing-mstest-tests SKILL.md:
- Add Response Guidelines for proportional responses
- Remove redundant Validation checklist and Common Pitfalls table
  (all items already covered in workflow steps)
- Reduces token count from 3,305 to 2,944

migrate-mstest-v1v2-to-v3 eval.vally.yaml:
- Relax DataRow rubric to account for MSTest 3.8.0 fixing the
  16-parameter constructor limit

writing-mstest-tests eval.vally.yaml:
- Make 3 prompts more open-ended by removing API name hints
  (Assert.HasCount, Assert.StartsWith, DynamicData to ValueTuples)
  so baseline model cannot answer confidently without the skill
2026-05-13 17:49:55 +02:00
github-actions[bot] 0521ae0ac6 chore: upgrade gh-aw runtime (#606)
* chore: upgrade gh-aw runtime

* chore: recompile gh-aw workflows

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jan Krivanek <jankrivanek@microsoft.com>
2026-05-13 08:43:12 +00:00
dependabot[bot] 3a323e4b9b Bump github/gh-aw-actions in the github-actions-dependencies group (#642)
Bumps the github-actions-dependencies group with 1 update: [github/gh-aw-actions](https://github.com/github/gh-aw-actions).


Updates `github/gh-aw-actions` from 0.72.1 to 0.74.1
- [Release notes](https://github.com/github/gh-aw-actions/releases)
- [Changelog](https://github.com/github/gh-aw-actions/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/gh-aw-actions/compare/bc56a0cad2f450c562810785ef38649c04db812a...b07cf98ac5874e8f51c34ba52099d8a6fac2ef93)

---
updated-dependencies:
- dependency-name: github/gh-aw-actions
  dependency-version: 0.74.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-13 10:07:46 +02:00
Jan Krivanek 205e9f54e2 fix(groom): prune resolved rows, add First Seen column, handle duplicate sections (#637)
* fix(groom): prune resolved rows, add First Seen column, handle duplicate sections

- Rename 'Status' column to 'Investigation' for clarity
- Add 'First Seen' column showing when finding was first detected
- Remove resolved investigation rows instead of keeping them forever
- Handle duplicate Investigation Results sections by merging into one
- Update health-check workflow to use matching column schema
- Drop resolved rows from health-check table preservation logic

* fix(health-check): remove duplicate Investigation Results section

The health-check and groom workflows both created a '## Investigation
Results' section, causing duplicates in issue #288. The groom owns this
section via its replace-island block — remove it from the health-check
template and update the guideline accordingly.

* fix(health-check): restore Investigation Results as placeholder section

The health-check replaces the entire issue body, so removing the section
entirely caused it to disappear. Instead, keep it as a placeholder that:
- Preserves existing rows from the groom's island markers
- Appends new 'Dispatched' rows for findings dispatched in this run
- Does NOT add island markers itself (the groom adds those)

This ensures the section survives health-check body rewrites while the
groom still owns the detailed content via replace-island.

* fix: address PR review comments on groom/health-check workflows

- Fix duplicate section guidance: clarify replace-island only updates
  first occurrence, doesn't auto-remove later duplicates
- Match example dispatched row text with health-check template
- Fix First Seen source: parse from summary line (first seen YYYY-MM-DD)
  instead of non-existent **First seen:** field
- Preserve all non-resolved rows (not just Done rows) in health-check
- Fix Step 4.2 to align with prune behavior in Step 4.3
- Add schema migration for legacy 4-column tables in health-check
- Add link to dispatched investigation rows
2026-05-11 20:31:24 +02:00
Jan Krivanek cf831597ff fix(groom): use MCP tools with min-integrity:none instead of unauthenticated gh CLI (#633)
* fix(groom): use MCP tools with min-integrity:none instead of unauthenticated gh CLI

The previous fix (PR #627) instructed the agent to use 'gh api' for reading
issue comments. However, the gh CLI is NOT authenticated inside the sandbox —
the framework deliberately strips credentials via the 'Clean git credentials'
step. The system prompt also explicitly states 'gh is not authenticated'.

Root cause analysis of run 25481332080 (pre-fix) shows the MCP issue_read
tool DID return comments — only 1 item out of many was filtered by integrity
policy. The agent overreacted to the [Filtered] warning and called
report_incomplete instead of working with the available data.

This fix:
- Sets min-integrity: none under tools.github so the integrity filter
  does not remove any comments. This is safe because the workflow only
  processes comments from github-actions[bot] (Step 2 filters by author).
- Reverts to MCP issue_read with get_comments (the proper tool for reading
  comments in the sandbox).
- Removes gh from bash tools (not usable without auth).
- Adds explicit guidance: if filtered items appear, continue working with
  available data — do NOT call report_incomplete or missing_tool.
- Documents that gh CLI is NOT authenticated in the sandbox.

Closes #625

* fix: address review comments - scope repos, bound comment fetch, fix jq guideline
2026-05-11 13:16:20 +00:00
dependabot[bot] 4a4c424ee4 Bump the github-actions-dependencies group across 1 directory with 3 updates (#631)
* Bump the github-actions-dependencies group across 1 directory with 3 updates

Bumps the github-actions-dependencies group with 3 updates in the / directory: [github/gh-aw-actions](https://github.com/github/gh-aw-actions), [actions/upload-artifact](https://github.com/actions/upload-artifact) and [actions/setup-node](https://github.com/actions/setup-node).


Updates `github/gh-aw-actions` from 2fe53acc038ba01c3bbdc767d4b25df31ca5bdfc to ea222e359276c0702a5f5203547ff9d88d0ddd76
- [Release notes](https://github.com/github/gh-aw-actions/releases)
- [Changelog](https://github.com/github/gh-aw-actions/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/gh-aw-actions/compare/2fe53acc038ba01c3bbdc767d4b25df31ca5bdfc...ea222e359276c0702a5f5203547ff9d88d0ddd76)

Updates `actions/upload-artifact` from 7.0.0 to 7.0.1
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v7...043fb46d1a93c77aae656e7c1c64a875d1fc6a0a)

Updates `actions/setup-node` from 4.4.0 to 6.4.0
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/v4.4.0...48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e)

---
updated-dependencies:
- dependency-name: github/gh-aw-actions
  dependency-version: ea222e359276c0702a5f5203547ff9d88d0ddd76
  dependency-type: direct:production
  dependency-group: github-actions-dependencies
- dependency-name: actions/upload-artifact
  dependency-version: 7.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions-dependencies
- dependency-name: actions/setup-node
  dependency-version: 6.4.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update gh-aw-actions from v0.68.1 to v0.72.1 in agentics-maintenance.yml

Agent-Logs-Url: https://github.com/dotnet/skills/sessions/2c61295e-713c-4b82-9681-339f0923989d

Co-authored-by: danmoseley <6385855+danmoseley@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: danmoseley <6385855+danmoseley@users.noreply.github.com>
2026-05-08 09:37:03 +00:00
Dan Moseley e7ec5a77a0 Fix markdownlint CI workflow (startup_failure since inception) (#629) 2026-05-08 08:33:08 +01:00
Jan Krivanek b306b4f11d Fix groom workflow: bypass MCP integrity filter with gh api (#627)
* Fix groom workflow: use gh api to bypass MCP integrity filter

The MCP gateway's integrity policy intermittently blocks access to issue
comments (report_incomplete: 'Resource issue:dotnet/skills#288 has lower
integrity than agent requires'). This caused issue #625.

Fix: Add 'gh' to allowed bash tools and instruct the agent to use
'gh api --paginate' for fetching comments. This bypasses the MCP gateway
entirely and also provides node_id fields required by hide-comment
(which the MCP issue_read tool did not return).

* Address PR review: fix --paginate JSON, add author filter, narrow guideline scope
2026-05-07 13:23:33 +02:00
Jan Krivanek 0bdf428da0 Fix triage workflow: separate label and assignee operations (#626)
* Fix triage workflow: separate label and assignee operations

- Use add_labels tool for labeling (Step 6) instead of update_issue
- Reserve update_issue exclusively for assignee assignment (Step 4)
- Increase update_issue max to 2 for assignment flexibility
- Add explicit workflow_dispatch guidance: agent must pass
  issue_number/item_number on all safe-output calls when there
  is no triggering issue context
- Fixes issues where owners were mentioned in comments but not
  actually assigned to the issue

* Apply suggestions from code review

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-05-07 10:42:56 +00:00
Aditya Mandaleeka 3cba22740b Include Vally job in parallel with existing eval (#622) 2026-05-06 15:14:17 -07:00
Aditya Mandaleeka 6a6340d82b fix: use 'vally' binary name in CI workflow (#621) 2026-05-06 14:04:05 -07:00
Aditya Mandaleeka 9433a31e67 Initial port of evals to Vally (#615) 2026-05-06 10:11:45 -07:00
Jan Krivanek 66628b651e Add triage workflows (#612) 2026-05-06 15:39:20 +02:00
Jan Krivanek b774eff859 Fix deploy-dashboard condition for workflow_dispatch (#591)
* Dashboards series fix (followup)

* Fix deploy-dashboard condition for workflow_dispatch

The deploy-dashboard job required needs.discover.result == 'success',
but the discover job's condition does not include workflow_dispatch,
so it is always skipped for manual triggers. This made workflow_dispatch
unable to deploy the dashboard.

Split the condition so workflow_dispatch from main bypasses the
discover dependency, while scheduled runs still require it.

* Update deploy-dashboard comment to reflect workflow_dispatch support
2026-05-05 14:01:05 +00:00
dependabot[bot] 3cbbaf2c3c Bump the github-actions-dependencies group with 2 updates (#573)
Bumps the github-actions-dependencies group with 2 updates: [actions/setup-node](https://github.com/actions/setup-node) and [tj-actions/changed-files](https://github.com/tj-actions/changed-files).


Updates `actions/setup-node` from 6.3.0 to 6.4.0
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/53b83947a5a98c8d113130e565377fae1a50d02f...48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e)

Updates `tj-actions/changed-files` from 47.0.0 to 47.0.6
- [Release notes](https://github.com/tj-actions/changed-files/releases)
- [Changelog](https://github.com/tj-actions/changed-files/blob/main/HISTORY.md)
- [Commits](https://github.com/tj-actions/changed-files/compare/24d32ffd492484c1d75e0c0b894501ddb9d30d62...9426d40962ed5378910ee2e21d5f8c6fcbf2dd96)

---
updated-dependencies:
- dependency-name: actions/setup-node
  dependency-version: 6.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions-dependencies
- dependency-name: tj-actions/changed-files
  dependency-version: 47.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-24 11:07:21 +00:00
Amaury Levé c6cf65c7c3 Add Markdown Linter agentic workflow (#582)
Adds a new agentic workflow (.github/workflows/markdown-linter.md) that
runs Super Linter on weekdays at 2 PM UTC and on manual dispatch. The
agent analyzes linter output and creates GitHub issues with categorized
findings, severity breakdown, and fix recommendations.

Adapted from https://github.com/githubnext/agentics/blob/main/workflows/markdown-linter.md
with repo-specific conventions: PAT pool rotation, fork protection,
pinned action SHAs, scoped bash tools, and safe-outputs.
2026-04-23 13:57:45 +00:00
Jan Krivanek c549fa1f18 Fix jsonl sessions attaching (#569) 2026-04-21 15:48:42 +02:00
github-actions[bot] 3aa1128ca9 chore: upgrade gh-aw runtime (#557)
* chore: upgrade gh-aw runtime

* chore: recompile gh-aw workflows

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jan Krivanek <jankrivanek@microsoft.com>
2026-04-20 17:34:47 +02:00
Jan Krivanek 1dc3d36f9b Fix gh-aw upgrade workflow to work with GITHUB_TOKEN (#554)
GITHUB_TOKEN cannot push commits that modify files under .github/workflows/.
This is a hard GitHub platform restriction (requires workflows scope, which
is only available via PAT or GitHub App tokens).

The workflow now:
- Runs gh aw upgrade (without --create-pull-request)
- Categorizes changes into pushable (outside .github/workflows/) and
  non-pushable (inside .github/workflows/)
- Creates a PR with only the pushable changes (e.g. actions-lock.json)
- Documents any workflow files that need manual recompilation in the PR body
  with instructions for running gh aw compile locally

Tested successfully on https://github.com/JanKrivanek/skills2/pull/1
2026-04-20 17:00:33 +02:00
Jan Krivanek ba3bcf122b Fix: add workflows permission, remove temp file creation (#545) 2026-04-20 09:35:32 +02:00
Amaury Levé 9792d18080 Add markdownlint analysis for local dev and CI (#534)
* Add markdownlint analysis for local dev and CI

- Add .markdownlint.jsonc with rule configuration
- Add .markdownlint-cli2.jsonc with ignore patterns
- Add package.json with markdownlint-cli2 and lint scripts
- Add GitHub Actions workflow running on PRs touching *.md files
- Add VS Code extension recommendation for vscode-markdownlint

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Address review comments: use markdownlint-cli2-action, merge configs, lint only changed files

- Replace manual Node.js + npm setup with DavidAnson/markdownlint-cli2-action (pinned to SHA)
- Use tj-actions/changed-files to lint only PR-changed markdown files
- Merge .markdownlint.jsonc into .markdownlint-cli2.jsonc (single config file)
- Remove package.json and package-lock.json (no longer needed)

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-17 20:17:33 +02:00
Jan Krivanek 05e2868b00 Reintroduce copilot sessions storing (#530) 2026-04-17 12:58:15 +00:00
Jan Krivanek e294085cf0 Automate GH AW updates (#536) 2026-04-17 14:45:11 +02:00
Amaury Levé 54cdb7ebca Add skill-coverage PR workflow with slash command support (#538)
* Add skill-coverage PR workflow with slash command support

- Auto-triggers on PRs that change plugins/** or tests/**
- Detects specific plugin/skill pairs from the PR diff
- Runs Measure-SkillCoverage.ps1 and posts a coverage report as a PR comment
- Supports /skill-coverage slash command on PR comments:
  /skill-coverage              (runs on changed skills in the PR)
  /skill-coverage <plugin>     (runs on all skills in the plugin)
  /skill-coverage <plugin> <skill>  (runs on a specific skill)

* Address Copilot review: security, permissions, and robustness fixes

- Add permission check: require write access before running slash command
- Tighten /skill-coverage matching to avoid false triggers on similar commands
- Add issues: write permission (PR comments use Issues API)
- Use refs/pull/<num>/head for checkout to support fork PRs
- Remove unused \ variable
- Fix multi-line JSON parsing with Out-String

* Address PR #538 review: fork PR handling, reactions, and sourcing note

- Skip PR comment posting for fork PRs (read-only token); coverage still runs in step summary
- Add continue-on-error to reaction step to avoid 403 failures
- Add header comment noting issue_comment trigger sources from default branch
- Add continue-on-error to no-changes comment step

* Split into analyze + comment jobs; fix pagination and dedup

Security:
- Split into two jobs: analyze (contents: read only) and comment (write only)
- Analyze job checks out base branch for trusted tooling, PR head into
  a separate git worktree - untrusted PR code is never executed
- Comment job only reads an uploaded artifact, no PR code checkout

Fixes:
- Use github.paginate() for listComments to handle PRs with >100 comments
- Deduplicate targets: plugin-wide changes replace per-skill targets for
  that plugin instead of producing duplicates
2026-04-17 12:55:33 +02:00
Jan Krivanek 74cf19f605 Update gh-aw (#532) 2026-04-17 09:52:44 +02:00
Jan Krivanek 9f2c53f68e Fix the DevOps health dashboard grooming (#529)
* Improve the groomer

* Cleanup the daily health issue
2026-04-15 19:01:40 +02:00
dependabot[bot] 7079f6c3ae Bump the github-actions-dependencies group with 3 updates (#526)
Bumps the github-actions-dependencies group with 3 updates: [actions/github-script](https://github.com/actions/github-script), [actions/cache](https://github.com/actions/cache) and [actions/upload-artifact](https://github.com/actions/upload-artifact).


Updates `actions/github-script` from 8.0.0 to 9.0.0
- [Release notes](https://github.com/actions/github-script/releases)
- [Commits](https://github.com/actions/github-script/compare/ed597411d8f924073f98dfc5c65a23a2325f34cd...3a2844b7e9c422d3c10d287c895573f7108da1b3)

Updates `actions/cache` from 5.0.4 to 5.0.5
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/668228422ae6a00e4ad889ee87cd7109ec5666a7...27d5ce7f107fe9357f9df03efb73ab90386fccae)

Updates `actions/upload-artifact` from 7.0.0 to 7.0.1
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/bbbca2ddaa5d8feaa63e36b76fdaad77386f024f...043fb46d1a93c77aae656e7c1c64a875d1fc6a0a)

---
updated-dependencies:
- dependency-name: actions/github-script
  dependency-version: 9.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-dependencies
- dependency-name: actions/cache
  dependency-version: 5.0.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions-dependencies
- dependency-name: actions/upload-artifact
  dependency-version: 7.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-15 09:46:36 +02:00
Jan Krivanek 70e2d3b333 Simplify the focus of devops health workflows (#521)
* Simplify the focus of devops health workflows

* Updated gh-aw and recompiled

* Fix scope wording to include resource usage per review feedback
2026-04-13 17:39:20 +02:00
Jan Krivanek a331d0994e Fix the devops health workflows after gh-aw update (#511) 2026-04-09 20:10:13 +02:00
Jan Krivanek ba4ff9569c Fix session visualisation filtering (#498) 2026-04-09 15:32:32 +00:00
dependabot[bot] 25e3b5fd0c Bump the github-actions-dependencies group with 2 updates (#503)
Bumps the github-actions-dependencies group with 2 updates: [actions/cache](https://github.com/actions/cache) and [actions/setup-node](https://github.com/actions/setup-node).


Updates `actions/cache` from 4.2.3 to 5.0.4
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v4.2.3...668228422ae6a00e4ad889ee87cd7109ec5666a7)

Updates `actions/setup-node` from 4.4.0 to 6.3.0
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/49933ea5288caeca8642d1e84afbd3f7d6820020...53b83947a5a98c8d113130e565377fae1a50d02f)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-version: 5.0.4
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-dependencies
- dependency-name: actions/setup-node
  dependency-version: 6.3.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-08 06:45:17 +00:00
Jan Krivanek 070f8b4648 Add pats rotation to agentic workflows (#500)
* Copy PATs rotation from dotnet/core

* Recompile agentic workflows with gh-aw v0.67.1

Remove unsupported 'failure' safe-output (dropped in gh-aw v0.67.1).
Regenerate lock files with the new compiler version.
No functional changes.

* Add Copilot PAT pool rotation to agentic workflows

Use the select-copilot-pat action to randomly select from a pool of
COPILOT_GITHUB_TOKEN / COPILOT_GITHUB_TOKEN_2..8 secrets, distributing
Copilot API load across multiple PATs.

This is a stop-gap until organization-level billing is available for
Agentic Workflows. See .github/actions/select-copilot-pat/README.md.
2026-04-07 12:37:30 +02:00
Jan Krivanek d861bbf62e Add AGENTVIZ session replay integration (#494)
* Add AGENTVIZ session replay integration

- Add workflow_dispatch trigger to evaluation.yml
- Add --keep-sessions to skill-validator evaluate step
- Add publish-session-data job (mirrors publish-token-data)
- Add replay link to PR comments (comment-on-pr)
- Add AGENTVIZ SPA build/deploy to deploy-dashboard job
- Add setup-node step to deploy-dashboard
- Add per-plugin Sessions Visualisation links to dashboard.js
- Create build-replay-sessions.ps1 (manifest generation from sessions.db)
- Create purge-replay-sessions.ps1 (7-day retention management)

* Address code review feedback on PR #494

- Use [IO.Path]::PathSeparator instead of hardcoded ';' for cross-platform PATH
- Compute cutoff date in UTC for correct retention comparisons
- Precompute ID HashSet before merge loop to avoid O(n^2)
- Pin actions/setup-node to commit SHA (49933ea5...#v4)
- Pin AGENTVIZ clone to commit SHA with verification
- Skip npm ci + build when deployed commit SHA matches pinned SHA

* Remove hardcoded AGENTVIZ SHA; use cache + zero-clone checks

- Resolve AGENTVIZ target SHA via git ls-remote (no clone)
- Read deployed SHA via curl from raw.githubusercontent.com (no clone)
- Cache build output keyed by commit SHA (skip npm ci+build on hit)
- Only clone AGENTVIZ repo on cache miss when SPA needs rebuild

* Address round 2 code review feedback on PR #494

- Compare scheduled dir dates at day granularity (dirDate.Date vs cutoffDate.Date)
- Fix useTempDir null-safe path comparison via GetFullPath + null guard
- Use UTC for dateTag in build-replay-sessions.ps1 (consistent with purge)
- Fix cache/deploy path mismatch: deploy from /tmp/agentviz-dist (cache path)
- Deterministic clone: full clone + git checkout TARGET_SHA (fail on mismatch)
- URL-encode manifest param in PR comment link (jq @uri)
- Derive manifest generated timestamp from newest session mtime to avoid churn
2026-04-02 17:39:59 +02:00
dependabot[bot] 0417c8ec62 Bump actions/cache in the github-actions-dependencies group (#492)
Bumps the github-actions-dependencies group with 1 update: [actions/cache](https://github.com/actions/cache).


Updates `actions/cache` from 4.2.3 to 5.0.4
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/5a3ec84eff668545956fd18022155c47e93e2684...668228422ae6a00e4ad889ee87cd7109ec5666a7)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-version: 5.0.4
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-01 10:20:01 +00:00
Jan Krivanek 498755bc0d Make the infra eval gate faster (#489)
* Make the infra eval gate faster

* Address PR feedback: handle infra+skill overlap, detect evaluation.yml changes

- When both infra and skill changes exist in a PR, evaluate the changed
  skills normally instead of the random subset (fixes Copilot review).
- Expand infra detection to include .github/workflows/evaluation.yml
  changes in all three detection sites (pr-status, fork-pr-status, discover).
2026-03-31 19:50:49 +02:00
Viktor Hofer 167b7c3772 Fix evaluation cache key: compute explicitly instead of cache-primary-key (#488)
The pinned actions/cache SHA (5a3ec84, v4) does not expose the
cache-primary-key output. This caused the evaluate job to fail with
'Input required and not supplied: key' because the build-validator
job output was always empty.

Fix: compute the cache key in a dedicated step and pass it as a job
output, instead of relying on an output that doesn't exist at this
action version.

Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-31 17:31:52 +02:00
Viktor Hofer 3fff1f2573 Skill-validator: perf, cancellation, and reliability improvements (#482)
* Skill-validator: perf, cancellation, and reliability improvements

Performance:
- Cache skill-validator-dist.tar.gz in CI (evaluation.yml and skill-check.yml)
  to skip .NET SDK setup, build, and tar on cache hit. Evaluate and comment-on-pr
  jobs restore from cache with artifact download fallback.
- Replace SemaphoreSlim with Lock in SessionDatabase for lighter-weight
  synchronous locking.

Cancellation:
- Thread CancellationToken from System.CommandLine entry point through the
  entire evaluate call chain: Run -> EvaluateTarget -> EvaluateSkill/EvaluateAgent
  -> ExecuteScenario/ExecuteAgentScenario -> ExecuteRun/ExecuteAgentRun, plus
  EvaluateSkillNoise -> ExecuteNoiseTest.
- Add CancellationToken parameter to AgentRunner.RunAgent, Judge.JudgeRun,
  PairwiseJudge.Judge, and OverfittingJudge.Analyze, forwarding to
  RetryHelper and LlmSession.SendAsync respectively.
- Pass CancellationToken to all ConcurrencyLimiter.RunAsync call sites so
  semaphore waits are cancellable.
- Re-throw OperationCanceledException at the top-level skill evaluation loop
  so Ctrl+C / CI cancellation propagates instead of being swallowed as a
  skill failure.

Reliability:
- Smart timeline truncation in Judge and PairwiseJudge: error events
  (session.error, runner.error) are always preserved during truncation
  regardless of position, with a summary marker showing event type breakdown.
- Add jitter (±25%) to RetryHelper exponential backoff to prevent thundering
  herd on rate-limit recovery.
- Add CancellationToken parameter to ConcurrencyLimiter.RunAsync<T>.
- Add ConcurrencyLimiter tests (6 tests) and RetryHelper jitter tests (2 tests).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review feedback

- RetryHelper: re-clamp delay after jitter so it never exceeds the
  remaining budget or MaxSingleDelayMs.
- Judge/PairwiseJudge: handle edge case where error count alone exceeds
  MaxTimelineEvents by trimming errors with a summary instead of
  unconditionally appending all of them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add high-variance detection and refactor FailureKind to enum

- Add Statistics.CoefficientOfVariation() for measuring score consistency
- Flag scenarios with CV > 0.5 as high-variance in ScenarioComparison
- Show console warning and markdown footnote for high-variance scenarios
- Include high-variance scenario names in Comparator verdict reason
- Refactor FailureKind from magic strings to strongly-typed enum with
  JsonStringEnumMemberName attributes for backwards-compatible JSON output
- Add 6 new CV unit tests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-31 13:58:28 +02:00
Viktor Hofer 7747fcb427 Move skill-validator docs under src/ and include in packages (#476)
* Move skill-validator docs under src/ and include in packages

Move README.md to src/ and InvestigatingResults.md + OverfittingDetection.md
to src/docs/ so they ship in the NuGet package and .tar.gz archives.

- Include docs/*.md in NuGet pack and publish output via CopyToPublishDirectory
- Add LICENSE to publish output (tar.gz only, not NuGet — uses PackageLicenseExpression)
- Replace enumerated docOnlyFiles lists with a regex pattern in CI workflows
- Deduplicate eval.yaml docs between CONTRIBUTING.md and the shipping README
- Add test fixture files walkthrough to README (moved from CONTRIBUTING.md)
- Add AGENTS.md guidance: src/ is the packaging boundary

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update link format in failure diagnosis message

* Update AGENTS.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-30 14:00:12 +00:00
Jan Krivanek 75f72efb0e Fix token usage collection (#471)
* Fix token usage collection

* Ensure evaluate is run on schedule

* Fix PR title issue

* Update eng/dashboard/token-usage.js

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-30 13:03:58 +00:00