mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
eval-verify/nuget-comment
592 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Copilot
|
36cdf059aa |
Clarify CPM Directory.Packages.props location in convert-to-cpm skill
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Abhitej John
|
78054c1161 |
Migrate LLM evals to the Vally harness (#877)
* Make Vally the sole LLM eval engine, retiring skill-validator evaluate Collapses the parallel skill-validator + Vally eval pipeline into a single Vally-only path while preserving every capability skill-validator provided: PR gate, PR comment, dotnet/skills-data historical push, and the dashboard. The skill-validator `check` linter is retained (skill-check.yml) pending microsoft/vally #463. - evaluation.yml: delete build-validator and the skill-validator `evaluate` job; single `evaluate:` job now uses vally-evaluation.yml. Forward the COPILOT_PAT_0..9 pool onto it (from #911) so the reusable workflow's token gate is satisfied. Rewire comment-on-pr / report-status / publish-* / deploy-dashboard onto `evaluate` and vally-results-* artifacts. - vally-evaluation.yml: pin @microsoft/vally-cli@0.9.0; keep the #911 prepare guard; write the generated per-plugin experiment to $GITHUB_WORKSPACE so relative eval globs resolve. - adapt.mjs: reconcile #887's runCompareWithRetry + conclusive/unmatched hardening with the consolidation helpers (nonActivation, compareByStim, roleToDashboard, etc.). Add consolidate.mjs and gen-experiment.mjs. - build-replay-sessions.ps1: read Vally executor-session-logs/events.jsonl instead of skill-validator session output. - Rename all tests/*/*/eval.vally.yaml to eval.yaml (Vally format is now the only eval format); author Vally evals for the four skills that lacked one. - CONTRIBUTING.md + InvestigatingResults: point contributors at the Vally harness; drop non-public links. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix setup-local-sdk eval for Vally schema The hand-authored setup-local-sdk eval used skill-validator grader/environment schema that Vally rejects at validation (no stimuli executed): - file-contains/file-not-contains graders require 'value:' not 'substring:' (output-contains correctly keeps 'substring:'). - environment.files entries require 'src:'/'dest:' (copy a fixture) rather than inline 'path:'/'content:'. Moved the global.json body into fixtures/global.json. Verified with 'vally lint --eval-spec' (0 errors) across all 97 eval.yaml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix markdownlint violations in migration docs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback on eval migration - Consolidate run-vally-evals.sh into a single neutral entrypoint eng/run-skill-evals.sh (deletes the thin wrapper; adapt.mjs stays in eng/vally-adapter/ since CI invokes it directly) - Rename reusable workflow vally-evaluation.yml -> evaluation-run.yml and update all references/self-trigger regexes - Rename local output dir vally-results -> eval-results - Bake prerequisite preflight (Node 20+, GITHUB_TOKEN/gh) into the runner - Simplify CONTRIBUTING "Running tests locally" prerequisites - Scrub harness name from consumer-facing surfaces Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reinstate overfitting detection under the Vally harness Overfitting detection previously ran inside skill-validator's `evaluate` command, which the Vally migration replaced, so it silently stopped running. Reinstate it as a standalone step wired into the Vally pipeline: - Add `skill-validator overfitting` command that discovers skills, parses each eval.yaml, runs the existing OverfittingJudge, and emits [{plugin, skill, overfittingResult}] JSON (per-skill failures non-fatal, bounded parallelism). - Add a Vally-format parser bridge (ParseEvalConfigFlexible) so the judge handles the current stimuli/graders eval.yaml format; the legacy scenarios-only parser rejected all 98 evals, so the judge never ran. - adapt.mjs: merge overfitting results onto each verdict via --overfitting <file> (keyed by plugin/skill); output is byte-identical when the flag is absent. - evaluation-run.yml: build skill-validator (shared cache with check), run the judge per leg reading GITHUB_TOKEN, and pass --overfitting to adapt.mjs so the dashboard/skills-data consume it unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reconcile #919 Vally eval configs into the sole eval.yaml convention PR #919 landed on main adding fresh Vally specs for three coverage-gap skills (setup-local-sdk, convert-blazor-server-to-webapp, dotnet-webapi) under the interim eval.vally.yaml filename. This branch collapses to a single Vally spec named eval.yaml, so the merge left each skill with both my migrated eval.yaml and #919's newer eval.vally.yaml (a git-clean but semantic duplicate the pipeline would not discover). Adopt #919's reviewed Vally content as the canonical eval.yaml for all three skills and drop the redundant eval.vally.yaml files, preserving the 'sole eval.yaml, zero eval.vally.yaml' invariant. Content is byte-identical to #919's blobs ( |
||
|
Abhitej John
|
90e21cddd9 |
test: add cross-family Vally eval configs for 3 coverage-gap skills (#919)
Close the coverage gap where these skills ship a base eval.yaml but had no eval.vally.yaml, so the cross-family evaluation harness silently never scored them (root cause behind #903): - dotnet/setup-local-sdk (deterministic, network-free stimuli: host-version gating, team install-script generation, dry-run install-plan composition; live SDK/workload install stays covered by the interactive base eval.yaml) - dotnet-aspnetcore/dotnet-webapi - dotnet-aspnetcore/convert-blazor-server-to-webapp Verified in cross-family run 29785538175 (executors opus/sonnet46/gpt, n=3, judge != executor family, 0 errored trials): setup-local-sdk and dotnet-webapi pass on all three executors; convert-blazor is under-powered (single stimulus) and its per-executor pass flags should not be over-read. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
dependabot[bot]
|
1e2fc4f10e |
Bump the github-actions-dependencies group across 1 directory with 5 updates (#918)
Bumps the github-actions-dependencies group with 5 updates in the / directory: | Package | From | To | | --- | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | `6.0.2` | `7.0.0` | | [actions/cache/restore](https://github.com/actions/cache) | `5.0.5` | `6.1.0` | | [actions/cache](https://github.com/actions/cache) | `5.0.5` | `6.1.0` | | [actions/setup-dotnet](https://github.com/actions/setup-dotnet) | `5.2.0` | `6.0.0` | | [actions/setup-node](https://github.com/actions/setup-node) | `6.4.0` | `7.0.0` | Updates `actions/checkout` from 6.0.2 to 7.0.0 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v6.0.2...9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0) Updates `actions/cache/restore` from 5.0.5 to 6.1.0 - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/v5.0.5...55cc8345863c7cc4c66a329aec7e433d2d1c52a9) Updates `actions/cache` from 5.0.5 to 6.1.0 - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/27d5ce7f107fe9357f9df03efb73ab90386fccae...55cc8345863c7cc4c66a329aec7e433d2d1c52a9) Updates `actions/setup-dotnet` from 5.2.0 to 6.0.0 - [Release notes](https://github.com/actions/setup-dotnet/releases) - [Commits](https://github.com/actions/setup-dotnet/compare/c2fa09f4bde5ebb9d1777cf28262a3eb3db3ced7...a98b56852c35b8e3190ac28c8c2271da59106c68) Updates `actions/setup-node` from 6.4.0 to 7.0.0 - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](https://github.com/actions/setup-node/compare/48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e...820762786026740c76f36085b0efc47a31fe5020) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies - dependency-name: actions/cache/restore dependency-version: 6.1.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies - dependency-name: actions/cache dependency-version: 6.1.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies - dependency-name: actions/setup-dotnet dependency-version: 6.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies - dependency-name: actions/setup-node dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Abhitej John
|
4734904b77 |
Map dotnet11 CODEOWNERS to @dotnet/skills-csharp-language-reviewers (#914)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3710475e-1fd6-4570-87ea-2c2b5d2cbde0 |
||
|
YuliiaKovalova
|
b39a041ed0 |
Strengthen dotnet-template-engine skills per cross-family eval (#898) (#904)
* Strengthen dotnet-template-engine skills per cross-family eval (#898) Cross-family skill eval (issue #898) flagged 5 of 6 skills as STRENGTHEN ("mostly ties - too generic/non-prescriptive") and template-instantiation as KEEP-POLISH. Invocation is already ~100% and token overhead is a penalty, so these edits add decisive, opinionated OUTPUT requirements (not volume). All skills remain in the recommended "detailed" tier. - template-validation: require a one-line verdict header + single findings table, and a concrete fix for every ERROR/WARNING. - template-smart-defaults: require a "Defaults applied" log + the exact single dotnet new command line (only flags actually passed). - template-comparison: require a decisive Recommendation verdict and add opinionated decision shortcuts for canonical template pairs. - template-discovery: lead the answer with a ready-to-run command line. - template-authoring: require a complete template.json + a conventions-preserved confirmation table. - template-instantiation: add a lean "match the workspace, then stop" directive to cut the excess turns/tokens flagged for the Opus gap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: remove template-specific AOT claims, narrow authoring validation routing - template-smart-defaults: reword the AOT note, rules row, and pitfall so they no longer claim --aot is unavailable on webapi/webapp (dotnet new webapi --aot exists since .NET 8 and template-instantiation references it). Now say --aot is present only where dotnet new <template> --help lists it, without naming specific templates as unsupported. - template-authoring: narrow the frontmatter DO NOT USE to standalone validation (use template-validation) and align the When-to-Use bullet, so it no longer contradicts the body, which validates as a step during authoring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: keep Defaults-applied Source strictly user|rule The sample row used 'rule (AOT)', which made the Source column look free-form. Set it to 'rule' and moved the AOT context into the Why column, matching the guidance that Source is strictly user or rule. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: tighten discovery description, make comparison AOT row conditional - template-discovery: simplify the redundant 'two or more templates against each other' phrasing to 'comparing templates side by side' and rewrap so each '(use <skill>)' stays with its clause. - template-comparison: reword the example AOT row so it is conditional on what 'dotnet new <template> --help' exposes instead of asserting webapi has no --aot (which contradicted template-instantiation). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: align find/compare routing between frontmatter and body - template-discovery: narrow the frontmatter exclusion to a *detailed* side-by-side comparison (use template-comparison), matching the body which supports light comparison but routes detailed comparisons away. - template-instantiation: split the frontmatter 'finding or comparing' exclusion into finding (template-discovery) vs detailed comparison (template-comparison), and align the When-Not-to-Use bullet accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: mention validation in template-authoring description The description said only 'Guides creation' while the skill validates the template it authors (intro + Step 2). Restored 'creation and validation' and added a USE FOR entry for validating the template.json you are authoring, while keeping standalone validation routed to template-validation in DO NOT USE. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: mention high-level compare in discovery; clarify defaults log scope - template-discovery: description now says 'find, inspect, and compare (at a high level)' so it isn't under-scoped vs the body (detailed side-by-side still routes to template-comparison via DO NOT USE). - template-smart-defaults: clarify the 'Defaults applied' log has one row per parameter covering both preserved user values (Source=user) and rule-applied defaults (Source=rule). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: symmetric AOT cell + clarify clean verdict - template-comparison: make the webapp AOT cell reference 'dotnet new webapp --help' so it is symmetric with the webapi cell and unambiguous. - template-validation: the 'Ready to publish' verdict now says '0 errors, 0 warnings' and notes optional suggestions may still apply, instead of 'clean'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: allow line:col location for JSON parse errors Widened the findings-table Location column to 'JSON path or line:col' and added an example parse-error row, so the required table can represent the JSON parse errors that Step 2 asks to report with a line number. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Improve template-discovery resilience: answer before any dotnet new call Eval investigation (PR #904 results, run 29503063336) showed template-discovery scoring 1.0-2.0 on Opus because sessions died with '(no output)' after a 'persistence'/'mutex' error from concurrent dotnet new calls. When the session survived, the same content scored 4.0-4.3. The failing runs invoked the skill then ran 'dotnet new --help' as the first/last action and the mutex error left the turn empty. Make answer-first an imperative rule: write the recommendation and ready-to-run command from the intent mapping BEFORE running any dotnet new command, and never end the turn on a dotnet new call. This makes the answer survive the mutex error (and gives users a faster answer). Skill stays in the 'detailed' token tier. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Iterate on eval quality: shadow-verb shortName warning + discovery resilience From PR #904 eval run 29509140219: - template-validation & template-authoring: add a WARNING tier for shortNames that *shadow* a well-known top-level dotnet verb (build/run/test/publish/pack/clean/ restore). These do NOT technically conflict with dotnet new (correctness kept), but 'dotnet new build' is easily confused with 'dotnet build', so it warrants a warning + a distinctive-name suggestion. This is a genuine authoring footgun and also resolves the 'Validate a template.json file' scenario where the agent was penalized for correctly saying shortName 'build' is fine. - template-discovery: require the pre-CLI answer to be complete and standalone (never end on a 'let me confirm from the CLI…' teaser, which the mutex error was truncating), and add a dry-run-failure fallback (describe expected files from template knowledge) so preview requests never return empty. Skill stays 'detailed'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Rubber-duck follow-up: revert eval-gaming, fix real defects Independent review + empirical checks (dotnet new build/restore/list --help; EvaluateCommand.cs Task.WhenAll) confirmed: - The mutex '(no output)' deaths come from the validator running Baseline+Isolated +Plugin concurrently, not cross-skill parallelism — eval.yaml can't fix it. - build/restore/test do NOT conflict with dotnet new (only real subcommands like 'list' do), so the shortName-shadow WARNING added last commit was appeasing a factually-wrong rubric (and the 'restore' entry contradicted the validation eval). Changes: - Revert the shortName-shadow WARNING in template-validation and template-authoring (the skills were already correct; do not game an incorrect rubric). - template-discovery: fix a real mapping bug — 'windows auth' mapped to SingleOrg; it is --auth Windows. Split azure ad/entra (SingleOrg) from windows auth (Windows). - template-discovery: tighten the dry-run-failure fallback to give only a representative structure and forbid inventing specific parameter values/choices/ file paths (avoids confident hallucination), keeping the 'answer-first, never end on a teaser' resilience. Skill stays in the detailed token tier. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: fix validation findings-table example (real subcommand + line:col) - Use a real dotnet new subcommand ('list') in the ERROR example instead of 'restore' (which does not conflict), matching the corrected shortName rules. - Make the parse-error location example '12:5' match the 'line:col' column header. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: consistency fixes across three skills - template-authoring: add ExcludeAssets to the conventions-preserved table (the workflow above lists it, so include it to avoid implying partial carry-over). - template-discovery: add SingleOrg to the example --auth choices so options aren't under-reported. - template-validation: broaden the concrete-fix requirement to allow a specific edit instruction (e.g. 'remove the trailing comma'), matching the parse-error example. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Fix incorrect eval rubrics: shortName 'build'/'restore' are not dotnet new conflicts The 'Validate a template.json' (authoring) and 'Validate template with multiple errors' (validation) scenarios used shortName 'build'/'restore' and asserted they conflict with dotnet new. Verified empirically they do NOT (dotnet new build/ restore --help treat them as template-name lookups; only real subcommands like 'list'/'search' intercept). The skills correctly report this, so the wrong rubrics were penalizing correct behavior and causing regressions. Change both fixtures to shortName 'list' (a real dotnet new subcommand that genuinely conflicts) and update the rubric/assertion wording accordingly, so the eval tests correct behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * template-discovery: faithful dry-run output + built-in package guidance Fresh eval (run 29581297999) confirmed the earlier rubric fix worked (authoring Validate 4.3->5.0). Two repeatable clean-output defects remained in template-discovery (seen in surviving Plugin runs, not just mutex deaths): - Preview: the agent summarized/regrouped the dry-run output and invented files (e.g. ReconnectModal), and didn't explain key files. Now: present the actual dry-run file list faithfully (no summarizing/inventing) and give a one-line purpose for key entry points. - Search: for a built-in template the agent never clearly answered 'which package to install'. Now: name the exact package, or say 'no install needed - ships with the SDK' for built-in templates. Compressed the resilience note to keep the skill in the detailed token tier. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Reduce eval overfitting: outcome-focused rubrics + de-cued prompts The overfitting judge flagged moderate overfit on smart-defaults (0.45), comparison (0.34), and instantiation (0.28). Per its per-element analysis (assertions were all 'broad'/fine), make the rubrics/prompts outcome-focused: - Drop every 'The agent used the template-* skill' rubric item (tests activation, not outcome — expect_tools already enforces skill usage as a hard gate). - Reword skill-vocabulary items (e.g. 'native AOT needs a recent AOT-capable TFM', 'smart defaults only fill gaps') and skill-method items ('verify against dotnet new --help', 'inspected via --help', 'logged each applied default', 'verified the project was created') into result-focused criteria that any valid approach satisfies. - De-cue the three smart-defaults prompts that mirrored the skill's 'Required Output B' verbatim ('output the exact single dotnet new command line ... only the flags you will actually pass'); they still elicit a concrete command so the negative assertions remain meaningful. - Drop the blazor prompt's 'Inspect both with dotnet new --help' methodology cue. Assertions and expect_tools are unchanged, so activation and correctness are still verified — the eval now measures outcomes instead of parroting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * template-smart-defaults: defer --help to creation time (trim advice-only cost) Eval run 29586558625 showed the advice-only AOT scenario tie at 4.0 but the skilled run cost 45.7k tokens / 3 turns vs baseline 12.7k / 1 turn — driven entirely by running 'dotnet new worker --help' (feeding ~30k tokens of help output into context) just to advise, which flipped the weighted score negative. Make --help confirmation a creation-time step: for advice-only requests, answer from the rules and note names are confirmed at creation, rather than spending a --help call to advise on well-known parameters. The quality wins come from the decisive output (defaults log + exact command), not from --help, so quality holds while cost drops. Addresses the #898 TRIM-COST concern. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Address PR review: workspace-framework precedence + sync eval.vally.yaml - Precedence conflict (instantiation 'mirror workspace framework' vs smart-defaults 'latest AOT-capable when --framework unset'): treat the workspace framework discovered in Step 2 as an explicit --framework value that smart-defaults must not override; deviate only when incompatible with a requested feature (then flag it). Added a matching note to the smart-defaults AOT rule (a framework pinned by the workspace/global.json counts as set). - Synced eval.vally.yaml (cross-family harness) with the eval.yaml changes that had drifted: outcome-focused rubrics + de-cued prompts for smart-defaults/comparison/ instantiation, and the shortName build/restore -> list fixture+rubric fix for authoring/validation (including the referenced fixture template.json files). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 * Assert required skill outputs with broad checks (address review) Reviewer flagged that the outcome-focused rubric rewrite left several scenarios without any assertion of the skill's required outputs. Add broad (non-overfitting) assertions in eval.yaml and eval.vally.yaml: - smart-defaults (all 4 scenarios): assert a 'dotnet new' command line is present (output B) and a markdown table is present (the 'Defaults applied' log, output A). - comparison (webapi/webapp, blazor): assert a recommendation is present. - validation (multiple errors): assert a verdict summary (Not ready / N errors) and a findings table are present. - authoring (validate): assert the agent flagged that shortName 'list' conflicts. These are broad presence checks (any valid phrasing passes), so they verify the required output without re-introducing the overfitting the rubric rewrite removed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0dadc452-fb69-49aa-9631-1b86647d83b9 --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
4df4da469a |
Upgrade agentic workflows and fix stale PR cleanup (#916)
* chore: upgrade gh-aw runtime * fix: paginate stale pull request cleanup Upgrade the generated agentic workflow assets and ensure stale PR discovery includes every result page and draft pull requests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 968a22c2-327f-4d26-8f86-1c59bcd323ea * Improve dotnet-test eval coverage and efficiency Address remaining high-confidence items from #899 by bounding the code-testing pipeline and adding eval coverage for grade-tests and find-untested-sources. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e430fee9-d3df-4ef5-85a4-745ae4b17046 * fix: address agentic workflow review Pin the Copilot setup checkout action and include the cutoff date in stale PR search results. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 968a22c2-327f-4d26-8f86-1c59bcd323ea * Improve test migration skill guidance Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5e19d263-02a6-45b1-9cfb-424fa4d10863 * fix: add fixture namespace imports Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a464e6e4-3e45-41fe-b17d-887c8cb8a448 * test: assert GetOrderById grade Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a464e6e4-3e45-41fe-b17d-887c8cb8a448 --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Art Leonard
|
ab72985132 |
Harden vally comparison reliability (#887)
* Harden vally comparison reliability Retry transient comparison errors once, surface unmatched trajectories, and mark degraded reports as inconclusive. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 33efeadb-dcc9-4005-a63f-c5d832b06f45 * Address vally review feedback Harden test diagnostics, pass workflow paths as arguments, and clarify the shadow-result schema. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 33efeadb-dcc9-4005-a63f-c5d832b06f45 * Move Vally adapter guidance Keep shadow evaluation reliability documentation with its implementation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 33efeadb-dcc9-4005-a63f-c5d832b06f45 --------- Co-authored-by: Art Leonard <artl@Arts-MacBook-Pro-M5.local> |
||
|
Abhitej John
|
efe972568b |
Fix two cross-family vally eval silent drops (#878)
Vally's `experiment run` harness (0.7) treats a non-zero setup command as a fatal trial error and drops all recorded events, unlike the .NET skill-validator which never inspects setup exit codes. Two evals hit this (or a related fixture bug) and produced zero verdicts in the cross-family matrix: - dotnet-msbuild/binlog-failure-analysis: the intentionally-broken solution makes `dotnet build -bl` exit non-zero (the binlog is still written). Guard the command with `|| exit 0`, which is portable across cmd.exe and POSIX sh (`true` is not a cmd builtin, and vally shells out via `cmd /c` on Windows). Also preserve any staged skill directory in the source-strip step so the skilled variant keeps its SKILL.md instead of aborting with "contains no SKILL.md". - dotnet-test/run-tests: the multi-TFM SDK 9 scenario pointed at a removed fixture (fixtures/multi-tfm/); repoint to the existing multi-tfm-mtp-sdk9 fixture, matching the skill-validator eval.yaml which already uses it. Scope is limited to the .vally.yaml variants; the sibling skill-validator eval.yaml files are already correct and need no change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
dependabot[bot]
|
584669624c |
Bump the microsoft-and-system group with 4 updates (#882)
Bumps Microsoft.Bcl.Memory from 10.0.9 to 10.0.10 Bumps Microsoft.Data.Sqlite from 10.0.9 to 10.0.10 Bumps Microsoft.Extensions.FileSystemGlobbing from 10.0.9 to 10.0.10 Bumps System.CommandLine from 2.0.9 to 2.0.10 --- updated-dependencies: - dependency-name: Microsoft.Bcl.Memory dependency-version: 10.0.10 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Data.Sqlite dependency-version: 10.0.10 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Extensions.FileSystemGlobbing dependency-version: 10.0.10 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: System.CommandLine dependency-version: 2.0.10 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Bcl.Memory dependency-version: 10.0.10 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Data.Sqlite dependency-version: 10.0.10 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Extensions.FileSystemGlobbing dependency-version: 10.0.10 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: System.CommandLine dependency-version: 2.0.10 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Abhitej John
|
6f53aaf841 |
fix(ci): unblock vally eval on the reusable PR path (#911)
* Fix vally eval on the reusable PR path: skip spurious prepare, forward PAT pool The PR-evaluation path (evaluation.yml -> reusable vally-evaluation.yml) could not start vally. Two independent problems, both fixed here in code: 1. Spurious `prepare` run. The `prepare` job guarded only on `github.event_name == 'workflow_dispatch'`, but that event name propagates into workflow_call, so prepare ran during the reusable call, read empty dispatch inputs, and threw "Invalid plugin ''". Now additionally guarded on `inputs.entries == ''` so prepare runs only for a direct manual dispatch and is skipped whenever the caller supplies `entries`. 2. Empty COPILOT_PAT_0..9 pool. A reusable workflow does not inherit the caller's org/repo secrets, and the `copilot-pat-pool` environment holds no environment secrets, so `secrets.COPILOT_PAT_*` read empty across the workflow_call boundary and the vally jobs failed with "No copilot-pat-pool entries are configured". The caller (a top-level job) CAN read the existing org-level pool -- the same pool the scheduled validate-pat-pool workflow reads -- so evaluation.yml now forwards it via an explicit named `secrets:` mapping of the 10 tokens. This mirrors how skill-validator sources its own token set in-workflow and needs no admin configuration. Deliberately NOT `secrets: inherit`, which would forward every repo/org secret into the reusable workflow and reverse #868's blast-radius reduction; only these 10 named tokens cross the boundary. Validated with actionlint 1.7.7 (exit 0). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
1b99afac59 |
Fix dotnet-msbuild skill eval regressions and coverage gaps (#896) (#910)
* Fix dotnet-msbuild skill eval regressions and coverage gaps (#896) Addresses the cross-family evaluation findings in #896. P0: - eval-performance: add diagnostic-first framing + explicit stop-conditions and gate prescriptive fixes on measurement, to stop over-application regressions. - msbuild-antipatterns: rewrite description with an action-verb lead and concrete trigger phrases to fix low invocation (FIX-DISCOVERY). P1: - Add cross-family eval.vally.yaml for extension-points, item-management, property-patterns, and target-authoring (previously excluded from the matrix). - check-bin-obj-clash: trim 465 -> 315 lines, removing the redundant text-log fallback steps, duplicate example workflow, and verbose tips (TRIM-COST). P2: - build-parallelism: add an opinionated 'start here' critical-path checklist. - binlog-generation: add one-build-one-binlog rule and a verify step. Keeps the plugin under the 15,000-char rendered skill-menu budget (14,981 -> 14,873). * Address PR review: portable grep whitespace + per-project eval count; shell-labeled binlog verify commands (#896) - check-bin-obj-clash: use [[:space:]] instead of non-portable \s in ERE; replace 'grep -c' note with an accurate per-project uniq -c count. - binlog-generation: label verify commands by shell and add the Windows cmd (dir /b) equivalent instead of mislabeling ls as cmd. * Address PR review round 2: full PowerShell -m command; match quoted project path in per-project eval count (#896) - build-parallelism: spell out 'dotnet build -m -bl:{{}}' for PowerShell so the -m flag isn't dropped. - check-bin-obj-clash: match the full quoted project path (not just filename) so same-named projects in different dirs stay distinct and paths with spaces work. |
||
|
Amaury Levé
|
a9fc55a765 |
test(dotnet-test): pin fixture SDK via global.json for writing-mstest-tests and detect-static-dependencies (#907)
P0 reliability hardening for the cross-family eval (#899). Adds a pinned global.json (rollForward: latestMajor) to every buildable fixture in both skills and wires it into the flat-copy scenarios so the executor arm no longer floats on the runner's ambient SDK. Note: the 3 errored trials in run 29228914412 were judge-side infra failures (CAPIError 400 'organization disabled' + 120s judge session.idle timeout), not fixture nondeterminism. These pins are defense-in-depth; the real fix (judge-error retry / token-pool pre-flight) is a harness follow-up. Copilot-Session: 5474f870-6be7-4b24-a5e1-a6e1e6de4ace Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
0ca7f8921d |
Fix migrate-static-to-wrapper DateTimeKind regression (#899) (#906)
The replacement mapping instructed models to convert TimeProvider's DateTimeOffset back to DateTime via .DateTime, which yields Kind=Unspecified and silently changes behavior vs the original DateTime.UtcNow (Utc) / DateTime.Now (Local). The eval rubric penalizes exactly this behavioral change, a likely cause of the frontier-model regression flagged in the dotnet-test scorecard. Use .UtcDateTime / .LocalDateTime to preserve Kind, and add a guardrail plus validation/pitfall notes about matching the target member type and not forcing DateTime through DateTimeOffset. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 69261256-256e-4e4c-9bdc-932d5ca8e5df |
||
|
Viktor Hofer
|
b83443ac1e |
Document SetTargetFramework bin/obj clash on single-targeting ProjectReferences (#853)
* Document SetTargetFramework bin/obj clash on single-targeting ProjectReferences Setting SetTargetFramework metadata on a ProjectReference to a non-multi-targeting project injects a path-neutral TargetFramework global property, forking a redundant project instance that shares the same OutputPath/IntermediateOutputPath as the solution-rooted build. The project is built twice, causing a bin/obj clash under parallel builds. Adds AP-23 to the msbuild-antipatterns catalog and a matching Common Causes and Fixes section plus global-properties note to the check-bin-obj-clash skill. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify SetTargetFramework: same-TFM clash vs legitimate overrides Refine AP-23 and the check-bin-obj-clash guidance to reflect two additional cases: (1) the clash only occurs when the injected TFM equals the single-targeting project's own TFM (path-neutral); injecting a DIFFERENT TFM changes the output path and is a valid override, and (2) for framework-incompatible references (e.g. .NETFramework test project -> single-targeting .NETCoreApp), SkipGetTargetFrameworkProperties=true and ReferenceOutputAssembly=false are also required. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Generalize incompatible-TFM ProjectReference guidance SkipGetTargetFrameworkProperties=true and ReferenceOutputAssembly=false are required whenever the referencing and referenced projects target incompatible frameworks, independent of single- vs multi-targeting and independent of SetTargetFramework. Decouple that rule from the SetTargetFramework override case in both AP-23 and the check-bin-obj-clash skill. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Note TargetFramework leak when SkipGetTargetFrameworkProperties is set When SkipGetTargetFrameworkProperties=true bypasses P2P negotiation, the referencing project's TargetFramework global property can flow into a single-targeting referenced project and build it under the wrong TFM/output path. Guard with either SetTargetFramework (pin) or UndefineProperties=TargetFramework (strip), not both. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add eval coverage for SetTargetFramework bin/obj clash (AP-23) check-bin-obj-clash: extend ClashTest with a single-targeting ToolLib and a ConsumerApp that references it with redundant same-TFM SetTargetFramework, forking a second ToolLib instance that builds to the same bin/obj (verified: ToolLib builds twice). Adds rubric items for identifying and fixing it. msbuild-antipatterns: LibB now references LibA with redundant same-TFM SetTargetFramework; adds a rubric item for detecting AP-23 in the static review scenario. Both eval.yaml and eval.vally.yaml updated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
9c7436b949 |
Add copy-to-output-directory MSBuild skill (IfDifferent / SkipUnchangedFilesOnCopyAlways) (#883)
* Add copy-to-output-directory MSBuild skill Adds a focused dotnet-msbuild skill documenting how to choose a CopyToOutputDirectory / CopyToPublishDirectory mode, including the `IfDifferent` mode and the `$(SkipUnchangedFilesOnCopyAlways)` property introduced in MSBuild 17.13 / .NET SDK 9.0.2xx (dotnet/msbuild#11052). Covers mode semantics (Never/PreserveNewest/Always/IfDifferent), why `Always` is a per-build perf hit, the SkipUnchangedFiles timestamp+size comparison, transitive copy through ProjectReference, and version requirements. Registers the skill in the msbuild, msbuild-code-review, and build-perf agent routing lists. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 881d26ad-87b7-48e3-8ec6-043163944437 * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Address review feedback on copy-to-output-directory skill - Soften the IfDifferent description to avoid overclaiming: it skips when "unchanged per MSBuild's heuristic" (timestamp + size, not a content hash), rather than "identical"/full Always equivalence. - Correct the target-flow wording: the three copy targets run as dependencies of _CopySourceItemsToOutputDirectory (invoked by CopyFilesToOutputDirectory). - Restore the USE FOR / DO NOT USE FOR routing pattern for the skill's frontmatter description, matching sibling skills. - Reclaim skill-menu budget headroom (per maintainer note) by tightening the verbose property-patterns and target-authoring descriptions; dotnet-msbuild rendered menu is now ~14,897/15,000 chars. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 881d26ad-87b7-48e3-8ec6-043163944437 * Fix article: 'an SQLite' per review Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 881d26ad-87b7-48e3-8ec6-043163944437 --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
79a2ada302 |
Fix dotnet-test plugin timeout + skill activation issues (#875)
* Fix dotnet-test plugin timeout + skill activation issues Activation: - filter-syntax eval: both scenarios set expect_activation: false. The skill is a reference skill (disable-model-invocation/user-invocable false) loaded by run-tests/mtp-hot-reload; it can never self-activate — this was an eval mis-spec, not a skill bug. - writing-mstest-tests 'Fix swapped Assert.AreEqual arguments': expect_activation false. The rubric requires diagnosing the root cause unaided, so no invocation is expected (notActivated 11/11 both arms). - test-smell-detection: relax the over-restrictive 'INVOKE ONLY when explicitly asked for the testsmells.org catalog' guidance so nuanced named-smell reviews with justified severity also activate (both-arm notActivated gap). Funded within the 15k skill-menu budget by compressing redundant framework parentheticals to the compact style already used by sibling skills. Timeout: - run-tests 'Filter xUnit v3 tests by class pattern and trait using query filter language': raise 240 -> 360 (observed peak ~178s, closest to its cap). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address /evaluate: revert test-smell regression, remove untestable filter-syntax eval Log-driven iteration on the eval results (run 29019511076): Priority 1 - regression fix: revert the test-smell-detection description broadening. results.json shows it reliably regressed 'Decline request to write new tests from scratch' (perRunScores [-0.19,-0.10,-0.22]; plugin overallJudgment -0.28, quality 5.0 baseline -> 4.3 skilled) while NOT fixing its target 'Distinguish reasoned skips' (isolated 5.0 vs 5.0 tie failing on judge position-bias, CV=114% - not description-fixable). Net-negative, so restore the pristine description. Priority 3 - remove the structurally-untestable filter-syntax eval. filter-syntax is disable-model-invocation/user-invocable:false, so it can never activate and can never show a skilled-vs-baseline delta - both scenarios fail by construction regardless of expect_activation. Peer reference skills (platform-detection, code-testing-extensions) have NO eval.yaml; filter-syntax was the lone exception. Its filter functionality is already covered by run-tests scenarios (which load filter-syntax as an additional_required_skill). Following precedent, remove the eval rather than leave two permanently-red cases. Kept from the prior commit: run-tests query-filter timeout 240->360 (harmless hard-cap headroom) and writing-mstest 'Fix swapped Assert.AreEqual' expect_activation:false (documents the unaided-by-design scenario and prevents a spurious hard activation failure). Not done - plugin-arm sibling deflections: results.json shows every failing plugin-arm scenario has detectedSkills=[] (NO sibling captured the turn), the rendered menu is under the 15k budget (descriptions are shown), and the verdict failures are dominated by high-variance efficiency noise (CV 60-3000%, several with quality improved). Per docs/InvestigatingResults.md there is no captor to deflect and description edits would not help; with zero budget headroom, adding speculative deflections is unjustified. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * run-tests evals: flip expect_tools:[bash] to reject_tools to fix activation Root cause of run-tests plugin+isolated non-activation (skill_not_activated gate): every run-tests scenario is a 'show me the exact command' knowledge question, yet each carried expect_tools: [bash] — which is an ASSERTION that bash was used. That actively forced the model into empirical exploration (dotnet test --help, trial runs, web_fetch) instead of consulting the run-tests skill, and a clean knowledge-only answer would FAIL the assertion. Evidence from run 29253857976: run-tests failed to activate even in the ISOLATED arm (where it is the only skill loaded) on scenarios as easy as base=1.0 — proof the model was bypassing the skill via bash rather than not needing it. The forced bash use also drove the token/tool/ time overhead that produced the negative efficiency scores, and one scenario hit the 360s timeout doing web exploration. Fix: convert all 15 run-tests scenarios from expect_tools: [bash] to reject_tools: [bash, edit, create]. These scenarios only assert on the recommended command text (output_matches) — none run tests (no run_command_and_assert, no file assertions) — and the model can still inspect fixtures via view/glob. Forbidding bash makes them pure-knowledge comparisons, so the model must reason from knowledge and reaches for the run-tests skill (improving activation in both arms), while removing the tool-induced overhead and timeout risk. This is stricter, not weaker, gating and matches docs/InvestigatingResults.md sec.8. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Revert run-tests reject_tools change — did not affect activation gate The expect_tools->reject_tools flip was based on a wrong assumption: reject_tools is a POST-HOC scoring assertion only (AssertionEvaluator/BaselineStore), it does NOT remove bash from the agent toolset, so the agent kept using bash and skill activation was unchanged (12 activation-fail arms before AND after, runs 29253857976 vs 29256920057). Reverting to keep the PR minimal and avoid adding failing assertions that don't serve the activation goal. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * writing-mstest 'Fix swapped AreEqual': validate activation instead of waiving it Addresses review feedback (r3551729171): setting expect_activation:false on an in-scope prompt (swapped Assert.AreEqual is explicitly in writing-mstest-tests' USE FOR) masks real routing/activation regressions. The waiver existed only because the task was trivially solvable unaided (baseline 5.0/5, zero delta), so the skill never needed to activate. Fix: make the scenario genuinely skill-dependent so activation is expected and validated. The prompt now also asks which analyzer catches this and how to enable it — skill-specific MSTest knowledge (SKILL.md Step 8: MSTEST0017 + MSTest.Analyzers packaging) that a vanilla model does not reliably produce. Added an output_matches 'MSTEST0017' assertion and two rubric items requiring the analyzer id and its enablement, and removed expect_activation:false. This both restores activation validation and gives the skill a real quality advantage over baseline. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * run-tests: strengthen activation triggers to fix systemic non-activation Radical fix for run-tests' systemic activation-gate failures. Root cause from artifact analysis: these are 'show me the exact dotnet test command' prompts, so the model answers directly from its own knowledge without invoking run-tests — even in the ISOLATED arm where run-tests is the only skill loaded (proving it is prompt/description-trigger driven, not just plugin-menu competition). It often answers WRONG on the subtle SDK 8/9 '--' separator vs SDK 10+ direct-args rule (baseline 1.0-1.7) yet still does not reach for the skill. At ~70% per-run activation the strict '.Any(3 runs) in BOTH arms' gate flakes red. Two-pronged fix to raise per-run activation probability: 1. run-tests description: lead with an imperative trigger ('ALWAYS use when the user asks ... wants the precise command, flags, or argument order') and name the model's blind spot ('the right syntax depends on the platform and SDK version and is easy to get wrong from memory'). This signals the model should consult rather than guess, and is more trigger-matched for the plugin arm. Kept under the 1024-char description limit and the 15000-char plugin skill-menu budget (verified: skill-validator check passes). 2. The four currently-flaking scenario prompts now state the user needs the command to be correct for THIS project's specific test platform and SDK version (a natural need that raises the stakes) without naming the skill or leaking the answer — the model must still detect the platform and know the version-specific syntax. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * run-tests Negative test: fix brittle assertion penalizing the better skilled answer After the activation fix, run-tests activates on the Negative VSTest scenario in both arms (activation solved), but it still failed with completion_regression. Root cause: the assertion output_not_contains '--report-trx' is too naive. The skill-informed answer is strictly better — it recommends 'dotnet test --logger trx' AND educates 'use --logger trx, not --report-trx (which is MTP-only)'. That correct, more helpful contrast trips the substring ban, marking the better answer as a task-completion regression versus a bare baseline command. Fix: forbid --report-trx only when it appears as an actual dotnet test command (output_not_matches 'dotnet test[^\r\n]*--report-trx'), not when the answer names it to warn against it. Verified against the real failing outputs: both skilled runs' --report-trx mentions are in explanatory prose and now pass, while an actual 'dotnet test ... --report-trx' recommendation is still rejected. Stricter-but-correct. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * run-tests Negative test: drop expect_tools:[bash] to remove efficiency-only failure After the activation + assertion fixes, the Negative VSTest scenario activates in both arms and passes all assertions, but still failed on a -2.8% weighted score: the baseline is already perfect (5.0/5, correctly uses --logger trx), so the skill can add no quality — and expect_tools:[bash] forced the skilled arm to actually run the tests, inflating tokens (40k->88k) and tool calls (5->8) versus a leaner baseline. That efficiency delta was the entire failure (docs sec.8 baseline-already-good). This is a 'recommend the exact command' negative test: exit_success only checks the agent produced output, and the other assertions validate the recommended command text, so requiring bash execution over-specified the scenario. Dropping expect_tools lets both arms answer from knowledge, removing the forced-execution token/tool overhead while still fully validating that the answer uses --logger trx and never a 'dotnet test ... --report-trx' command. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Abhitej John
|
ca1b47371f |
Add automated per-plugin versioning (NBGV) with /version-bump + weekly backstop (#813)
* Add automated per-plugin versioning (NBGV) with /version-bump + weekly backstop WHY Tools that surface skills (Copilot CLI, Claude Code, Codex, Cursor) read a plugin's version directly from its checked-in manifest. With no versioning discipline, a plugin's behavior can change while its advertised version stays flat, so clients never learn to re-pull, and there is no human-readable signal of what changed. We want correct, current versions in the repo with minimal manual work and without bloating the marketplace clone. WHAT - Per-plugin semantic versioning via Nerdbank.GitVersioning (NBGV). Each plugin owns a version.json whose pathFilters exclude the generated manifests and the version.json itself, so version height tracks real content changes only. - The computed version is materialized into the checked-in manifests (plugin.json and .codex-plugin/plugin.json) so every consumer reads a current value with no build step on their side. - eng/version/Sync-PluginVersions.ps1 is the single workhorse. It resolves the set of changed plugins from a git diff, computes each version with nbgv (predicting the squash-merge height for PRs), and either reports or stamps. AUTOMATIONS (two, low-touch by design) - /version-bump: an admin/maintainer comments the command on a PR and the affected plugins are stamped on the PR branch. Gated on collaborator permission (admin/write/maintain); forks are rejected before any privileged step. No other PRs are auto-modified. - weekly-version-sync: a Monday backstop (and workflow_dispatch) that stamps any drift on main, opens/updates a single bot PR, and explains the per-plugin reason. This self-heals anything that merged without a bump. We deliberately did NOT auto-edit contributor PRs or add a noisy advisory comment bot; maintainers stay in control and the signal stays clean. SECURITY (multi-model adversarial review: GPT-5.5 + Gemini 3.1 Pro) - Supply chain (High, both models): dotnet tool restore would have honored a nuget.config authored in the PR tree, letting an attacker remap the nbgv package source to a malicious feed and run code in the privileged contents:write context. Mitigated with a trusted eng/version/nuget.config (clear + nuget.org-only + packageSourceMapping), overlaid from main and used via --configfile so PR-supplied configs are ignored. No nuget.config is tracked in the repo today, so this path was genuinely exploitable. - TOCTOU (Medium): /version-bump now checks out the authorized head SHA rather than the mutable branch name; a racing push fails non-fast-forward, which is the safe outcome. - Injection: Set-ManifestVersion uses a MatchEvaluator (not a replacement string) so a "$"-bearing version cannot re-expand, plus a strict major.minor.patch guard that throws on a malformed base, leaving manifests untouched. - A base-only version.json bump (0.1 -> 0.2) is correctly detected and stamped. VERIFIED End-to-end against a real NBGV git harness: content-scoped predict, base-only bump -> x.y.0, docs-only -> [], weekly drift stamping, malformed-base guard, and --configfile restore (exit 0). actionlint passes on both workflows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review feedback - Add missing plugins/dotnet-test-migration/version.json so it participates in versioning (it was the only plugin without one; manifests are at 0.1.0). - CONTRIBUTING: the two manifests are not byte-identical; say the version is duplicated across two manifest files instead. - weekly-version-sync: include version.json in commit attribution so a base-only bump is explained rather than showing 'no attributable commits'. - Get-NbgvInfo: capture nbgv stderr and include it in the thrown error so CI failures are diagnosable, while keeping stdout clean for JSON parsing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Aditya Mandaleeka
|
52ba152ed6 | Update to Vally 0.7.0 (#854) | ||
|
Abhitej John
|
80baddd6c4 |
Harden vally evaluation and source tokens from copilot-pat-pool (#868)
* Harden vally evaluation against command injection; source tokens from copilot-pat-pool Close a command-injection vector where fork-PR-controlled path segments flowed into a bash run: body via matrix.entry.plugin interpolation in the token-bearing job. - Allowlist-validate matrix entries (plugin/name/skills_path) before they enter the matrix in evaluation.yml, and re-validate defense-in-depth as the first step of the vally-evaluate job. - Pass all PR-controlled values through env: vars and reference them quoted; no raw expression interpolation remains in any run: body. - Build workflow_dispatch entries with jq --arg instead of string interpolation. - Replace secrets: inherit with explicit secret passing to reduce blast radius. - Source Copilot tokens from the copilot-pat-pool environment (COPILOT_PAT_0..9) via the job environment, with COPILOT_GITHUB_TOKEN as an empty-pool fallback. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address Copilot review: reject empty skills_path and quote dirname - Fail the matrix-entry validation when skills_path is empty (never a legitimate state for vally entries) instead of silently skipping the loop. - Quote the inner $(dirname ...) in EVAL_NAME to avoid word-splitting. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address jeffhandley review: drop PAT fallback, explain regex anchors - Remove the COPILOT_GITHUB_TOKEN fallback secret from vally-evaluation.yml (workflow_call.secrets, the token-selection step, and evaluation.yml's caller-side secrets passing). An empty copilot-pat-pool is now a hard configuration failure instead of falling back to a separate secret. - Add short comments explaining why bash's ^/$ anchors are safe here without the \A/\z equivalents used in the PowerShell discover job. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Harden regex validation against locale-dependent bracket ranges Add LC_ALL=C before the allowlist regex checks in both the prepare job and the Validate matrix entry step. POSIX bracket ranges like [A-Za-z] can behave unexpectedly under non-C locale collation; forcing the C locale makes the anchor/character-class behavior deterministic regardless of the runner's default locale. Found via cross-model (GPT-5.5) review of the prior fixup commit. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address review: constrain vally by skills_path, reject empty skills_path - vally: select eval specs per matrix leg's skills_path so sharded plugins run only their own skills (no N-times re-run / PAT-pool token waste) - evaluation.yml: hard-fail on empty skills_path in the matrix allowlist - vally: quote inner dirname substitution in the Write summary step Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Reject whitespace-only skills_path in matrix validation A value like a single space passed the bash `-z` empty check and then word-split to zero loop iterations, letting an effectively-empty skills_path slip through. Count word-split segments instead so empty and whitespace-only both hard-fail. Mirror the intent on the PowerShell discover side with a trim-based emptiness filter. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Bind skills_path segments to the matrix entry's plugin The skills_path allowlist previously validated only the generic shape (plugins/<any>/skills[/<skill>]); it did not require each segment to belong to the entry's own plugin. A mismatched entry (plugin=foo + skills_path=plugins/bar/skills) would pass, and downstream steps assume the plugins/$PLUGIN/skills prefix. Add a literal (non-regex) prefix binding in both validators: - vally-evaluation.yml: [[ seg != prefix && seg != prefix/* ]] with skills_prefix="plugins/$ENTRY_PLUGIN/skills". - evaluation.yml: case-sensitive -cne plus Ordinal StartsWith against "plugins/$($e.plugin)/skills". Literal comparison avoids regex-metachar pitfalls from '.'/'-' in plugin names; -cne/Ordinal keeps it case-sensitive to match Linux path semantics. Legitimate entries are always same-plugin, so this rejects no valid input. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
042972d411 |
Reduce evaluation fan-out to avoid Copilot rate limits (#872)
* Reduce evaluation fan-out to avoid Copilot rate limits Group PR evaluation entries by plugin and preserve existing executionShard bucketing so large PRs stop launching one skill-validator job per changed skill. Cap PR matrix concurrency and keep PR grouped legs at parallel-skills=1 so each assigned Copilot token stays below the rate-limit threshold. Keep vally on the proven reusable-workflow matrix pattern, cap its concurrency, and only synthesize manual entries when inputs.entries is empty so workflow_call runs use the caller-provided grouped entries. Also skip agent.* and eval-less skills when constructing vally --eval-filter arguments. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Make vally full runs work before the experiment file lands PR 872 is intentionally based on main only, but main does not have the new repo-root dotnet-skills.experiment.yaml yet. The vally workflow was hard-coded to that file, so full branch-level runs from PR 872 failed before starting any trials. When the repo file is absent, synthesize an equivalent temporary experiment file in the runner and use that for read_override() and vally experiment run. That keeps PR 872 self-contained while preserving the same per-skill skilled-vs-baseline experiment shape. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Wendy Breiding
|
f0700c74e9 |
Add Code of Conduct (#867)
* Add Code of Conduct document based on Contributor Covenant * Rename CODE-OF-CONDUCT.md to CODE_OF_CONDUCT.md for GitHub auto-detection --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Abhitej John
|
134917c42c |
fix: reinforce no-script-execution constraint in devops-health-check (#845)
* fix: reinforce no-script-execution constraint in devops-health-check The agent was hallucinating Python script creation (run_analysis2.py) and attempting to execute it, which was correctly blocked by the bash allowlist security policy. Strengthen the efficiency guideline to explicitly state that python/python3/node are NOT in the bash allowlist and will be blocked, directing the agent to use jq instead. Fixes #841 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: narrow wording to 'general-purpose language runtimes' per review Addresses review feedback: 'any script interpreter' was too broad since jq/sed are technically interpreters but ARE in the allowlist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
1270722c95 |
Fix migrate-static-to-wrapper skill activation on evals (#864)
Sharpen the description boundary between migrate-static-to-wrapper and generate-testability-wrappers so prompts that replace call sites when the abstraction is already registered in DI activate the migration skill. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
0451fe8d91 |
dotnet-test: raise timeout for assertion-quality decline-write-tests scenario 300->360 (#862)
The 'Decline request to write new tests from scratch' non-activation scenario writes a full MSTest suite from scratch (code generation) and was hitting its 300s wall-clock timeout. Per InvestigatingResults.md, code-generation scenarios often need 360s. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
e558acb8e8 |
dotnet-test: raise timeout for boundary comparison assertions eval scenario (#863)
The 'Use comparison assertions for boundary testing' scenario chronically hit its 240s wall-clock timeout. Raise it to 360s to match the other code-generation scenarios in this eval file, per InvestigatingResults.md timeout guidance. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
8bbf2bfd61 |
dotnet-test: reduce assertion-quality over-structuring on small suites (#865)
Scale report depth to suite size so a tiny test file gets a focused, direct answer instead of a padded multi-section dashboard. Preserves full-template behavior for substantial suites and keeps all rubric-relevant substance (assertion-free/trivial identification, quality verdict, concrete recommendations). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
a60d989504 |
Fix mtp-hot-reload SDK 9 eval timeout by removing expect_tools bash (#861)
The 'Suggest hot reload for failing test in MTP project (SDK 9)' vally stimulus forced bash usage on a purely advisory question, triggering a slow SDK 9.0.200 NuGet restore that exceeded the timeout. Align with the source-of-truth eval.yaml which does not constrain this scenario. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
77154137e8 |
dotnet-test: make code-testing agent tools Claude Code-compatible + add cross-host portability check (#856)
* dotnet-test: make code-testing agent tools declarations Claude Code-compatible PR #847 added `tools: ["agent", "skill", "read", "search", "edit", "execute"]` to the code-testing-* agents to enable VS Code / Copilot CLI subagent fan-out. Those lowercase aliases map to real tools in VS Code and the Copilot CLI, but Claude Code matches `tools:` against its own vocabulary (Task, Skill, Read, Glob, Grep, Edit, Write, Bash). None of the aliases matched, so when these agents are loaded into Claude Code via --plugin-dir and selected with `claude --agent`, the agent was granted ZERO tools. A tool-less model asked to generate tests emits a textual <tool_call> block and exits after one turn, producing no file changes. Append the Claude Code tool names to each agent's `tools:` list so the same declaration works across all three runtimes (each honors the names it knows and ignores the foreign ones): - Orchestrators (generator, implementer): add Task, Skill, Read, Glob, Grep, Edit, Write, Bash (Task is the Claude Code equivalent of the `agent` fan-out tool). - Workers (researcher, planner, builder, tester, fixer, linter): add Skill, Read, Glob, Grep, Edit, Write, Bash. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * skill-validator: complete built-in tools + add cross-host tool portability check Two related follow-ups to the agent tools fix: 1. Address the skill-check review feedback. The validator's BuiltInTools set was missing three legitimate host tool spellings that are not case-insensitive matches of existing entries, so they were flagged as non-built-in: - "write" — Claude Code file-creation tool (Copilot CLI / VS Code: "create") - "agent" — Copilot CLI / VS Code subagent fan-out tool (Claude Code: "task") - "execute" — Copilot CLI / VS Code run-command tool (Claude Code: "bash") "agent" and "execute" were already flagged before this branch (introduced by the fan-out PR); adding them to BuiltInTools clears the pre-existing warnings. 2. Add a cross-host tool portability check (CheckAgentToolPortability) so an agent that declares a capability for only one host is flagged. Tool names are matched case-sensitively (hosts resolve tools by exact spelling), so an agent that lists e.g. only "edit" (Copilot / VS Code) without "Edit"/"Write" (Claude Code) is reported as working on one host and silently tool-less on the other. Findings are advisory (do not fail CI) and allowlistable via "agent-tool-portability:AGENT:capability". Wired into the agents loop in CheckCommand and covered by unit tests. Also make the one existing single-host agent (optimizing-dotnet-performance) portable by adding its Claude Code tool spellings, so the new check reports a clean tree. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
misdolicc
|
17bd25bec4 |
Add pull request template (#686)
Co-authored-by: Wendy Breiding <55603905+webreidi@users.noreply.github.com> |
||
|
Amaury Levé
|
72af853455 |
Add slnf and slnx solution formats to run-tests skill (#857)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
ce75c35569 |
dotnet-test: raise timeouts for chronic-timeout eval scenarios (#850)
Several dotnet-test eval scenarios repeatedly hit their wall-clock timeout in the evaluation dashboard (skilled + plugin arms), producing empty output and failing all assertions. Increase per-scenario timeouts for the ones that timed out most often across recent runs: - writing-mstest-tests: 'Write unit tests for a service class', 'Write data-driven tests for a calculator', 'Use string assertions for format validation' 180->360; 'Use comparison assertions for boundary testing' 120->240 - coverage-analysis: 'Coverage plateau diagnosis', 'Project-wide coverage analysis with existing Cobertura data' 300->480 - test-gap-analysis: 'Decline request to write new tests from scratch' 120->300; 'Acknowledge well-tested code with few surviving mutations' 300->420 - run-tests: 'Filter xUnit v3 tests by trait on MTP', 'Filter xUnit v3 tests by class pattern and trait using query filter language' 120->240 Per eng/skill-validator/src/docs/InvestigatingResults.md, timeouts are the highest-priority failure mode; code-generation scenarios often need 360s. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
a677ec43d2 |
Consolidate dotnet-test-frameworks into test-analysis-extensions (#851)
* Consolidate dotnet-test-frameworks into test-analysis-extensions Remove the orphaned dotnet-test-frameworks reference skill, whose content was a duplicate subset of test-analysis-extensions/extensions/dotnet.md. Nothing actually loaded it. Redirect the remaining references and drop its eval tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Make exp-test-maintainability Step 1 self-contained and fix activation Address PR review: exp-test-maintainability (dotnet-experimental plugin) must not point at test-analysis-extensions (dotnet-test plugin) — inline the .NET framework markers instead. Also strengthen the description/When-to-Use triggers so the 'each new case needs a whole new method / suggest a better structure' scenario reliably activates the skill instead of routing to a test-writing skill. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Revert exp-test-maintainability description broadening; keep self-contained markers The description broadening regressed activation (3/4 -> 0/4 across a variance-dominated single eval run, CV up to 924%). Description is the activation lever, so restore the original wording and keep only the review-comment fix (inlined .NET framework markers in Step 1). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix scenario 1 eval: use project fixture so the skill activates Scenario 1 pasted a self-contained snippet inline and asked to 'suggest a better structure', which the base model answers directly without loading the skill -> NOT ACTIVATED with no headroom. Convert it to the project-fixture pattern used by the activating scenarios (3 & 4): move the repetitive InputValidatorTests into a Validation.Tests fixture and ask the model to analyze the project. Mirror the change in eval.vally.yaml. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix scenario 2 eval: use project fixture so the skill activates Scenario 2 also pasted a self-contained snippet inline (well-maintained Auth.Tests) and asked to 'review these tests', which the base model handles without loading the skill. Convert to the project-fixture pattern used by the activating scenarios; add fixtures/well-maintained/Auth.Tests. Mirror in eval.vally.yaml. All four scenarios now reference a project on disk. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add reject_tools to exp-test-maintainability evals to create headroom All four scenarios scored NOT ACTIVATED / no-headroom (#8 in InvestigatingResults.md): baselines are already 4.0-5.0/5, so loading the skill only adds token/tool overhead with no offsetting quality gain. Match the convention of the sibling analysis skill test-anti-patterns: reject bash/edit/create on every scenario. This skill is analysis-only ('do not modify any files'), so the constraint is a no-op for correct behavior but levels the playing field between baseline and skilled runs — scoring focuses on answer quality (weight 0.40+0.30) instead of tool-induced overhead. Applied to both eval.yaml and eval.vally.yaml. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address review: make well-maintained fixture self-contained and deterministic - Import System.Security.Authentication for AuthenticationException - Replace wall-clock assertion (token.ExpiresAt > DateTime.UtcNow) with a deterministic one anchored on a fixed clock (FixedNow), and drive the injected FakeClock from that fixed time. This removes the flakiness and makes the FakeClock injection meaningful — a better exemplar for the 'well-maintained' scenario. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Petr Pokorny
|
325890d119 |
Bump dotnet-test to 0.2.0 (#852)
* Bump dotnet-test plugin to 0.2.0 Signals the VS Code subagent fan-out fix (#847) to existing installs. Minor bump; no breaking changes. * Sync .codex-plugin manifest to 0.2.0 Keeps plugins/dotnet-test/.codex-plugin/plugin.json in sync with the main plugin.json, matching the convention other plugins follow (e.g. dotnet-ai). Addresses PR review feedback. |
||
|
Abhitej John
|
733cc264e3 |
fix: add .claude-plugin manifest for dotnet-msbuild binlog MCP server (#844)
Claude Code discovers MCP servers via .claude-plugin/plugin.json inside each plugin directory. The existing .codex-plugin/ manifest only works for Codex CLI. Add the Claude Code-specific manifest with inline mcpServers config, and also declare mcpServers in the root marketplace manifest entry for belt-and-suspenders coverage. Fixes #840 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Petr Pokorny
|
06dd46e5f1 |
Enable VS Code subagent fan-out for the dotnet-test code-testing agents (#847)
* Add VS Code subagent metadata to dotnet-test code-testing agents
Enable the code-testing-* Research-Plan-Implement pipeline to fan out as
subagents in VS Code while keeping the GitHub Copilot CLI working.
Frontmatter (VS Code "coordinator/worker" pattern; portable tool aliases that
map in both VS Code and the CLI):
- code-testing-generator: tools: [agent, read, search, edit, execute] + an
agents: list (researcher, planner, implementer, builder, tester, fixer,
linter); softened the three runSubagent({ agent, prompt }) blocks to
tool-agnostic delegation prose.
- code-testing-implementer: tools: [agent, read, search, edit, execute] +
agents: (builder, tester, fixer, linter).
- Leaf agents (researcher/planner/builder/tester/fixer/linter):
tools: [read, search, edit, execute].
Why explicit tools (not ["*"]): VS Code has no all-tools wildcard for ools:
and a subagent's ools: overrides its inherited set, so ["*"] matched nothing
and stripped subagents of file tools (they ran without read/edit/search). The
CLI treats ["*"] as all-tools, so this was VS-Code-specific. The portable
aliases agent/read/search/edit/execute map to real tools in both environments;
agents: is ignored by the CLI.
README: document the VS Code chat.subagents.allowInvocationsFromSubagents
setting (off by default) needed for the nested implementer->builder/tester/
fixer/linter layer to fan out on large scopes; the CLI has no such gate.
Validated end to end: VS Code shows researcher->planner->implementer fanning out
with real file I/O (no "subagents lack file tools" warning); CLI fan-out intact
with skills still loading and tests passing. An all-tools baseline used only
tools within this enumerated set, confirming no CLI capability is restricted.
* Include skill tool in code-testing agent tool allowlists
Address PR review: the explicit `tools:` allowlists omitted the `skill`
tool, but every code-testing-* agent's prompt instructs calling skills
(e.g. `code-testing-extensions` for per-language guidance, `test-gap-analysis`,
`assertion-quality`). Because `tools:` is an override, omitting `skill` can
prevent the agents from loading those skills in environments that gate skill
invocation by the allowlist.
Add `skill` to all eight agents:
- orchestrators (generator, implementer): [agent, skill, read, search, edit, execute]
- workers (researcher/planner/builder/tester/fixer/linter): [skill, read, search, edit, execute]
Re-verified in the Copilot CLI: full fan-out (researcher -> planner ->
implementer/tester), the `code-testing-extensions` skill is invoked, and the
generated tests pass.
|
||
|
Wendy Breiding
|
4975793af0 |
Moving niche skills to dotnet-advanced (#807)
* Add csharp-development skill and update CODEOWNERS and README.md - Introduced a new skill for C# development guidance. - Updated CODEOWNERS to include reviewers for the new skill. - Modified README.md to list the new csharp-development skill. - Added evaluation scenarios for the csharp-development skill. * Refactor C# evaluation scenarios to enhance production code review and validation guidance * Refactor production code review scenarios to improve clarity and focus on async error handling * Update CODEOWNERS to reflect current ownership and streamline reviewer assignments * Remove csharp-development skill documentation and evaluation tests * Remove csharp-development skill from README * Fix formatting in Constants.g.cs by adding missing semicolon * Revert Constants.g.cs to main * Remove csharp-development skill from CODEOWNERS * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Update CODEOWNERS to include additional reviewers for dotnet-ai skills * Move dotnet skills to dotnet-util to keep dotnet plugin focused and targeted. * Add CODEOWNERS, SKILL.md, package-types.md, and publish-workflow.md for NuGet trusted publishing; implement eval.yaml for dotnet-pinvoke and dotnet-util skills * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Update README files to enhance plugin descriptions and remove outdated LSP section * Rename repository title from 'dotnet' to 'dotnet-util' in README.md for clarity. * Refine README.md to clarify included .NET skills and remove redundant descriptions * Add 'dotnet-util' plugin for utility skills in .NET development * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Add 'dotnet-util' plugin for utility skills in .NET development * Add dotnet-util to Codex marketplace manifest * Align dotnet-util marketplace descriptions * Refactor CODEOWNERS and update README for nuget-trusted-publishing; add evaluation scenarios and workflow template for trusted publishing * Add P/Invoke diagnostics and type mapping references; implement NuGet trusted publishing setup - Created diagnostics.md for common pitfalls and failure modes in P/Invoke. - Added type-mapping.md for native-to-.NET type mappings. - Introduced nuget-trusted-publishing skill for setting up OIDC-based NuGet publishing. - Added package-types.md detailing structural requirements for various NuGet package types. - Created publish-workflow.md as a template for GitHub Actions workflows for NuGet publishing. * Add evaluation YAML files for csharp-scripts, dotnet-pinvoke, and nuget-trusted-publishing skills * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Complete nuget-trusted-publishing move: remove from dotnet, add to dotnet-specialized README * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Remove eval.vally.yaml and eval.yaml files for nuget-trusted-publishing skill * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Add NuGet trusted publishing skill and related references - Introduced `nuget-trusted-publishing` skill for setting up OIDC-based trusted publishing on GitHub Actions. - Added detailed documentation in SKILL.md covering prerequisites, process, safety rules, and troubleshooting. - Created `package-types.md` to outline structural requirements for various NuGet package types. - Developed `publish-workflow.md` as a template for GitHub Actions workflows for NuGet publishing. - Removed obsolete `dotnet-specialized` plugin and updated `dotnet` plugin description. - Added evaluation tests for `nuget-trusted-publishing` skill to ensure proper guidance for users. - Implemented evaluation tests for `dotnet-pinvoke` and `csharp-scripts` skills to validate functionality. * Update dotnet plugin description to include high-level .NET development skills * Add NuGet trusted publishing and P/Invoke documentation - Introduced type mapping reference for native-to-.NET types in dotnet-pinvoke. - Added a comprehensive guide for setting up NuGet trusted publishing using OIDC in GitHub Actions. - Created a reference document detailing structural requirements for various NuGet package types. - Provided a template for GitHub Actions workflow for publishing NuGet packages with trusted publishing. - Implemented evaluation scenarios for testing the dotnet-pinvoke and nuget-trusted-publishing skills, including prompts and assertions for expected outputs. * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Add CODEOWNERS entries for setup-local-sdk * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Update dotnet README to list setup-local-sdk skill * Add setup-local-sdk to skills section in README * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
8a52700bce |
Add TFM-aware buildTransitive->build forwarding guidance to msbuild skills (#836)
* Add TFM-aware buildTransitive->build forwarding guidance to msbuild skills buildTransitive/*.props should forward through the corresponding build/*.props (ownership chain buildTransitive -> build -> shared) rather than importing buildMultiTargeting/ directly. When build/ is packed per-TFM (build/<tfm>/), the forwarder must include the TFM segment and derive it from the file own folder, not $(TargetFramework) (NuGet nearest-match can serve a different asset folder), otherwise transitive consumers hit MSB4019. Updates extension-points (new Forwarding chain section), msbuild-antipatterns AP-13, and the msbuild-code-review agent. Lesson learned from microsoft/testfx#9431. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify ambiguous props/targets glob in forwarder review check Address review feedback on dotnet/skills#836: spell out ".props/.targets forwarders" instead of the ambiguous `buildTransitive/*.props|targets`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tighten forwarding-chain guidance to reduce skill token overhead Condense the new extension-points Forwarding chain section (+26 -> +11 lines) and the AP-13 note by dropping the redundant non-TFM example and self-evident derivation explanation, keeping the chain rule, MSB4019 cause, and the TFM derivation expression. Lower token footprint addresses the skill-validator weighted-score token penalty without losing substance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review nits: AP range and forward-slash path Update the additional anti-patterns range to AP-16 through AP-22 (the reference doc now includes AP-22), and use a forward-slash build/MyPackage.props in the forwarding-chain prose to match the build/<tfm>/ convention used in the section. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Condense AP-13 forwarding note to a cross-reference to reduce token overhead The full TFM-forwarding guidance lives in extension-points; AP-13 only needs a concise pointer. Removes duplication and trims the msbuild-antipatterns skill footprint flagged by the skill-validator token penalty. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Address remaining review nits: targets forwarders, single-source wording, quoted MSBuildThisFileDirectory --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Huo Yaoyuan
|
4560ea4d4d | Update P/Invoke skill (#798) | ||
|
Javier Calvarro Nelson
|
9ed853bffc |
Port dotnet-blazor evals to Vally format (#843)
Adds eval.vally.yaml for all 9 dotnet-blazor skills, converted from the legacy skill-validator eval.yaml specs, and registers a dotnet-blazor suite in .vally.yaml so the shadow Vally pipeline discovers them. Conversion preserves prompts, assertions, and rubric verbatim. Legacy setup blocks map to Vally environment config: 'dotnet new' scaffolding commands become environment.commands. Verified end-to-end by running the author-component eval (baseline + skilled + adapt), which passed the shadow threshold. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
691cd472a1 |
Fix writing-mstest-tests activation by deflecting sibling skills (#835)
* Add MSTest deflection to assertion-quality and test-anti-patterns skills The writing-mstest-tests skill failed to activate in the plugin arm for six MSTest-specific scenarios (fix swapped Assert.AreEqual, modernize legacy patterns, type/string assertions, DynamicData with ValueTuples) because sibling skills captured the routing. assertion-quality and test-anti-patterns matched these prompts but lacked explicit deflection to writing-mstest-tests. Add DO NOT USE entries pointing MSTest test writing/fixing/modernizing to writing-mstest-tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: restore general assertion-fix deflection, align test-anti-patterns - assertion-quality: restore the general 'fixing or rewriting assertions' deflection (dropped in the prior edit) alongside the MSTest deflection, and fix punctuation to use comma-separated DO NOT USE items. - test-anti-patterns: align the 'writing new tests' deflection to also point MSTest test-writing to writing-mstest-tests, matching the body guidance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jeff Handley
|
11c5dfc738 |
Require environment input to pat_pool shared workflow. (#832)
* Require environment input to pat_pool shared workflow. - Updates the PAT pool shared workflow to latest version - Updates the validate-pat-pool workflow to match other repos - Updates all agentic workflows to use the updated PAT pool import with the standard secret names used across other repos - The new version of the import does not require the 'needs' workaround but the issue-triage workflow's 'roles: all' configuration requires a different workaround to ensure the pre_activation job exists for the pat_pool job to be able to depend on it (necessary for ordering). * Update markdown bullet style * Apply Copilot PR feedback Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Bump versions to match gh-aw --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
592a53008e |
skill-validator: restore 15K aggregate cap as the real Copilot CLI skill-menu budget (#803)
* skill-validator: restore 15K aggregate cap, document it as the real Copilot CLI skill-menu budget The per-plugin aggregate description cap had been raised 15,000 -> 20,000 -> 22,000 under the belief that 15K was 'a local repo policy, NOT a documented Copilot constraint'. That belief was wrong: the GitHub Copilot CLI renders the model-facing <available_skills> menu under a hard 15,000- char budget (the agent SDK's SKILL_CHAR_BUDGET, default 15e3, confirmed in CLI 1.0.36 and 1.0.61). Skills are listed alphabetically and emitted with their full <description> only until the budget is exhausted; every skill past the cut-off collapses to a bare name with no description and can no longer be reliably model-activated. Raising the validator cap merely masked this silent menu truncation — e.g. dotnet-test's run-tests and test-* skills stopped activating in plugin eval runs because they fell into the name-only overflow. Changes: - SkillProfiler.MaxAggregateDescriptionLength: 22,000 -> 15,000, with the comment rewritten to document the real Copilot CLI budget (and correct the prior 'not a documented constraint' claim). - CheckCommand aggregate now excludes skills marked 'disable-model-invocation: true' — the CLI drops those from the menu, so they do not consume the budget. This makes the cap satisfiable by hiding reference / agent-orchestrated primitives rather than only by trimming. - InvestigatingResults.md: document plugin-arm-only non-activation caused by skill-menu budget overflow, and how to fix it. Note: dotnet-test currently exceeds 15K and must be slimmed below it (via disable-model-invocation on reference/primitive skills plus description trims) before this cap can go green repo-wide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * skill-validator: use source-generated regex for disable-model-invocation check Address review: replace Regex.IsMatch(pattern-string) with a [GeneratedRegex] partial method (AOT-friendly, no per-call cache lookup), matching FrontmatterParser's style. Runs once per skill during checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * skill-validator: parse disable-model-invocation via YAML to avoid block-scalar false positives The regex-based check matched any line in the frontmatter, so a block-scalar description that merely mentioned 'disable-model-invocation: true' on its own line was wrongly treated as disabling model invocation. Parse the frontmatter with the existing YAML deserializer (which correctly handles block scalars) by adding a DisableModelInvocation field to SkillFrontmatter, and drop the regex entirely. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
26f01b5d5c |
Add eval coverage for dotnet-test/writing-mstest-tests (#828)
* Add eval coverage for dotnet-test/writing-mstest-tests Enrich the ServiceRegistry collection/null/reference scenario with deterministic file_contains assertions for the modern MSTest assertion APIs Assert.IsNull, Assert.AreSame, Assert.Contains, Assert.DoesNotContain, Assert.IsEmpty, and Assert.IsNotEmpty. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reject legacy CollectionAssert/StringAssert helpers in collection scenario Address PR review: file_contains is a case-sensitive substring check, so the Assert.Contains and Assert.DoesNotContain value checks would also pass for CollectionAssert.* and StringAssert.* helpers. Add file_not_contains guards to keep the scenario a deterministic check for the modern Assert.* API. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
dc9261f483 |
Add eval coverage for dotnet-test/run-tests (#822)
* Add eval coverage for dotnet-test/run-tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Consolidate redundant rubric bullets in run-tests eval Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
988f5d24ee |
Add eval coverage for dotnet-test/generate-testability-wrappers (#821)
* Add eval coverage for dotnet-test/generate-testability-wrappers Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reword ambient-context prompt to avoid DO-NOT-USE activation trigger Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
7b02af3fcf |
Add eval coverage for dotnet-test/filter-syntax (#824)
* Add eval coverage for dotnet-test/filter-syntax Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: align xUnit v3 fixtures and tighten filter-syntax assertions Match known-good MTP xUnit v3 fixtures, enforce the real Trait annotation and SDK 9 separator, and tighten the query-filter scenario with a realistic fixture. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
32ec973eeb |
Add eval coverage for dotnet-test/test-smell-detection (#826)
* Add eval coverage for dotnet-test/test-smell-detection Adds a scenario plus fixture covering concrete fix examples, contextually-obvious-number calibration, justified severity levels, and reasoned vs bare skip distinction. Skill coverage now 100%. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: tighten Sleepy regex and block bash for read-only scenario Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
29f09cc6e4 |
Add eval coverage for dotnet-test/grade-tests (#825)
* Add eval coverage for dotnet-test/grade-tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add missing per-test grade assertions to grade-tests evals Address PR review: verify TestParse_NoError=C in scenario 4, and Charge_NegativeAmount=A / Refund_ExistingCharge=C in scenario 5 so the scenarios cannot pass on misgraded output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
91d68cbd5e |
Add eval coverage for dotnet-test/test-gap-analysis (#829)
* Add eval coverage for dotnet-test/test-gap-analysis Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Mark InvoiceProcessor as partial to match generated part Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |