mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
eval-exec-opus48
589 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Amaury Levé
|
4ca17dc05b |
Add eval coverage for dotnet-test/migrate-static-to-wrapper (#827)
* Add eval coverage for dotnet-test/migrate-static-to-wrapper Extend eval.yaml with rubric items and two scenarios covering: required using directives, build-succeeds-after-migration, test-double updates, no-behavioral-changes, not breaking static classes, and incremental migration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review on migrate-static-to-wrapper eval Restore expect_activation: false on the decline scenario and add missing using BillingApp; to the test fixture so it starts buildable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
7d4f9fd639 |
Fix hashtable .Keys/.Values member-access collision in skill coverage (#831)
Get-SignificantTerms returned $terms.Keys; when a teaching point contains the word 'keys', PowerShell hashtable member access resolves to that entry's value instead of the key collection, collapsing the keyword set and making points like test-tagging Step 3 uncoverable. Use get_Keys()/get_Values() to bypass key-name shadowing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
f0f42a7918 |
Add eval coverage for dotnet-test/assertion-quality (#819)
Adds outcome-focused rubric items so two previously-uncovered Validation teaching points become covered: - Metrics are computed correctly (counts add up) - If the suite has good diversity, the report acknowledges this Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
cbb6121e76 |
dotnet-test: trim skill descriptions to fit skill-menu under 15K (#816)
The plugin's rendered skill-menu was 15,428 chars, over the 15,000 limit. Compressed the verbose per-framework parentheticals in the longest polyglot descriptions and dropped a few redundant clauses, preserving all routing keywords. Aggregate name+description length now ~14.5K. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Abhitej John
|
9b50865f24 |
build: recompile gh-aw workflows to v0.79.8 (follow-up to #658) (#814)
PR #658 bumped .github/aw/actions-lock.json to v0.79.8 but its automation could not recompile the .lock.yml files, because the GITHUB_TOKEN it runs under lacks the 'workflow' scope. PR #815 then shipped prompt fixes but likewise did not re-pin the runtime. As a result main is in a split state: actions-lock.json declares v0.79.8 while all 8 compiled workflows still pin and execute the v0.77.5 runtime (setup@v0.77.5). Run 'gh aw compile' (CLI pinned to v0.79.8) so the deployed workflows actually run the version main already declares, eliminating the declared-vs-deployed drift. This only re-pins the runtime and regenerates compiled output (the 8 .lock.yml files, the generated agentics-maintenance.yml, and the compile-managed dependabot entry); it makes no prompt/behaviour changes, so #815's groom and health-check fixes are preserved verbatim (groom body_hash unchanged at a705abb0). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Gerald Versluis
|
72c0d2c7b7 |
Add setup-local-sdk skill for global.json paths feature (#508)
* Rebase setup-local-sdk skill Replay PR #508 onto current main and address the remaining setup-local-sdk review feedback. The install examples now use fail-fast temporary downloads, the team scripts preserve existing global.json content, and eval assertions cover the critical paths/version/workload behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address setup-local-sdk review comments Add the missing skill license metadata and fix the PowerShell .gitignore regex examples so they correctly detect an existing .dotnet/ entry. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Stabilize setup-local-sdk evals Reduce overfit-prone assertions, make incompatible host handling explicit, configure base SDK resolution before slow workload installs, and tighten exact-version/team-script validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address setup-local-sdk follow-up review Include the shared dotnet skill reviewer team in CODEOWNERS, make .gitignore appends newline-safe, and tighten the incompatible-host eval to reward stopping before local install commands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tighten exact local SDK eval Measure project-level SDK resolution and exact roll-forward behavior using global.json artifact assertions so the scenario distinguishes PATH-based local installs from SDK paths setup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address install script review feedback Use the documented lowercase dotnet-install quality value and call out both bash and PowerShell official install script URLs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Increase team script eval timeout Give the team install script scenario the same realistic long-running timeout as other scenarios that install preview SDK bits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Windows local SDK check Document the PowerShell equivalent for detecting an existing local SDK and give exact SDK installs enough time to complete in eval. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Stabilize team script eval Focus the team script scenario on script/config generation, create reproducible setup guidance before long downloads, and avoid brittle transcript assertions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make mktemp usage portable Use BSD/macOS-compatible mktemp templates in setup-local-sdk bash snippets and generated team install script. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
d656edc4f2 |
fix(devops-health): stop recurring groom node_id and health-check tool/token failures (#815)
* fix(devops-health): stop recurring groom node_id and health-check tool/token failures Groom (fixes 780/792/797): MCP issue_read(get_comments) sometimes omits node_id, which hide-comment requires. The agent was reporting missing_data and failing the run. Make node_id best-effort: skip hiding comments without a node_id and continue; the weekly cleanup workflow removes them by age. Health check (fixes 810/791/788): the agent attempted sed and git commands (incl. manual git plumbing on the cache-memory dir) that the bash allowlist denied, looping on the denials and exhausting the 25M effective-token budget. Add sed and git to the allowlist, and add guidelines to not git-manage cache-memory and not retry denied commands. Recompiled lock files with pinned gh-aw v0.77.5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(groom): clarify skipped-comment count goes in noop summary only Address PR review: Step 6 only emits a summary via noop when no other safe-output calls were made, so 'note in the Step 6 summary' was misleading. Track the skipped count internally and include it in the noop message only when that summary is emitted. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Andrew Barnes
|
96bd58f72c |
docs: clarify required .NET SDK version (#682)
Co-authored-by: Abhitej John <aajohn@microsoft.com> |
||
|
github-actions[bot]
|
8652212cb8 |
chore: upgrade gh-aw runtime (#658)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
YuliiaKovalova
|
67e2a47e87 |
Trim dotnet-msbuild skill-menu under 15K char limit (#809)
The rendered skill-menu for the dotnet-msbuild plugin was 16,679 characters, exceeding the 15,000 limit. Trim two categories of content from the 18 skill descriptions that do not help agents decide when to invoke a skill: - Remove INVOKES: clauses (which MCP tools / fallback shell commands a skill runs internally) — implementation mechanics, not routing signal. - Remove the boilerplate 'Only activate in MSBuild/.NET build context.' repeated verbatim in all 18 skills — redundant, since every description is already MSBuild-specific and each DO NOT USE FOR still excludes non-MSBuild build systems. USE FOR triggers and DO NOT USE FOR cross-references are preserved. Aggregate skill description size drops from 14,551 to 12,536 chars (rendered menu 16,679 -> 14,664). Co-authored-by: Yuliia Kovalova <ykovalova@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
35ace775b3 |
Add dotnet-test-migration plugin and move migration skills there (#808)
Move the .NET test framework/platform migration skills (migrate-mstest-v1v2-to-v3, migrate-mstest-v3-to-v4, migrate-vstest-to-mtp, migrate-xunit-to-mstest, migrate-xunit-to-xunit-v3) and the test-migration orchestrator agent out of dotnet-test into a new dedicated dotnet-test-migration plugin, along with their evals. Update all marketplace manifests, READMEs, CODEOWNERS, and .vally.yaml accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
33110eeba8 |
code-testing-agent: fix workspace-integrity activation + stabilize Contoso rubric (#806)
* code-testing-agent: fix workspace-integrity activation + stabilize Contoso rubric
Workspace-integrity scenario was NOT ACTIVATED in plugin mode: the prompt
("Generate unit tests for its core module") was terse and small-scoped, so
the runtime did not route to the code-testing-agent skill. Reframe it with
high-level test-generation language ("comprehensive pytest test suite",
"scaffold", "thorough unit tests") that matches the skill description,
while preserving the guardrail anchors: it still points at the on-disk module
without naming it and never implies restoring the gutted tree.
ContosoUniversity rubric item #5 (find-untested-sources) was conditional on
that skill being loaded — only true in plugin mode. In isolated runs the
agent cannot satisfy it, so the judge penalized it asymmetrically, injecting
isolated-vs-plugin variance. Make the conditional deterministic: explicitly
N/A when the skill is not loaded, without lowering the bar when it is.
Applied to both eval.yaml and eval.vally.yaml.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* code-testing-agent: make find-untested-sources rubric decidable from session timeline
Address review feedback: the 'treat as satisfied (N/A) when the skill is
not loaded' clause is not verifiable from the judge's inputs (which show
which tools were called, not which were available). Rewrite the criterion
to be decidable from the session timeline by requiring a source-to-test
pairing map recorded in .testagent/research.md that either cites
find-untested-sources output or documents the equivalent manual approach.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
dependabot[bot]
|
9d698df6da |
Bump Microsoft.Bcl.Memory and 3 others (#742)
Bumps Microsoft.Bcl.Memory from 10.0.7 to 10.0.9 Bumps Microsoft.Data.Sqlite from 10.0.7 to 10.0.9 Bumps Microsoft.Extensions.FileSystemGlobbing from 10.0.7 to 10.0.9 Bumps System.CommandLine from 2.0.7 to 2.0.9 --- updated-dependencies: - dependency-name: Microsoft.Bcl.Memory dependency-version: 10.0.9 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Data.Sqlite dependency-version: 10.0.9 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Extensions.FileSystemGlobbing dependency-version: 10.0.9 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: System.CommandLine dependency-version: 2.0.9 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Amaury Levé
|
29bfae5f5b |
run-tests: fix evals (query-filter regex, sibling skills, command observability) (#800)
* run-tests: fix evals (query-filter regex, sibling skills, command observability)
Investigated run-tests eval failures by running the validator locally.
- SKILL.md Step 3: document xUnit v3 --filter-query so the agent stops
answering that complex xUnit v3 filters 'cannot be combined'.
- eval.yaml: fix a broken assertion regex. The query-filter pattern is a
single-quoted YAML scalar using '\\s'/'\\[', which (unlike a double-quoted
scalar) is NOT unescaped, so the regex searched for a literal '\s' and
could never match. Corrected to single backslashes.
- eval.yaml: add additional_required_skills (filter-syntax / platform-detection)
to the filter and detection scenarios, so the isolated arm loads the sibling
reference skills that run-tests explicitly defers to.
- eval.yaml: ask the agent to show the exact command in execute-style prompts.
output_matches only sees the final assistant message; 'run my tests' prompts
make the agent execute and summarize ('tests passed'), so the recommended
command never appears. The assertions still catch wrong commands.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* run-tests: rewrite description for reliable plugin-arm activation
The run-tests skill activated reliably in the isolated eval arm but
unreliably in the plugin arm (all dotnet-test sibling descriptions
loaded), so every scenario's pluginImprovementScore went negative and
dragged min(isolated, plugin) below zero.
Lead the description with natural-language intent triggers that mirror
how the eval prompts phrase requests (run all tests, run a subset via
filters, produce TRX reports, collect crash/hang dumps, run a single
TFM) instead of opening with platform-detection mechanism, and add
explicit DO NOT USE redirects to code-testing-agent / mtp-hot-reload.
Stays under the 1024-char description cap.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* dotnet-test: fix run-tests plugin-arm activation via skill-menu budget
Root cause (verified against the Copilot CLI SDK skill renderer): the
model-facing skill menu has a 15000-char budget. Skills are listed
alphabetically and emitted with full descriptions only until the budget
is exhausted; the rest collapse to bare names with no description and
effectively cannot be model-activated. With 27 dotnet-test skills,
run-tests (alphabetical position ~20) fell into the name-only overflow,
so it never activated in the plugin eval arm even though it activated
reliably in isolation. This is a real user-facing discoverability bug,
not just an eval artifact.
Fix: hide reference/primitive skills that are never meant to be
model-invoked from the menu via 'disable-model-invocation: true', which
the SDK filters out of the budget entirely:
- filter-syntax, platform-detection, dotnet-test-frameworks,
code-testing-extensions, test-analysis-extensions — already
user-invocable:false reference data ('DO NOT USE directly').
- find-untested-sources, find-untested-sources-polyglot — researcher
primitives invoked by-name from the code-testing-researcher agent
(which has a manual fallback); no standalone evals.
These remain invocable by explicit name (agents/users), only auto-
suggestion is suppressed.
This frees enough budget that run-tests (plus migrate-xunit-to-xunit-v3
and mtp-hot-reload) now receive full descriptions; no previously-visible
skill regresses. Also trimmed the run-tests description so its menu block
fits with margin while keeping all activation triggers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
Amaury Levé
|
102663d337 |
Fix code-testing-agent activation for the Flask pytest scenario (#802)
* Fix code-testing-agent activation for the Flask pytest scenario The 'Generate pytest tests for the Flask tasks API' scenario failed to activate code-testing-agent in BOTH isolated and plugin mode: its prompt enumerated, file by file, exactly what to mock/inject/test (TaskService with repo mocked + clock injected, queries.apply_query over fixed lists, both repositories, the blueprint via test_client), acting as an answer key that let the base agent generate tests directly with edit tools instead of routing to the skill's research-plan-implement pipeline. Rewrite the prompt to a realistic, high-level ask (mirroring the ContosoUniversity scenario that does activate): describe the app at a layer level, keep the 'no tests yet', project-wide multi-file framing and the 80% coverage floor, and drop the per-module test checklist. Assertions, rubric and timeout are unchanged. Verified locally that the skill now activates in both isolated and plugin mode. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use python3 in Flask prompt to match the grader Review feedback: the prompt told the agent to run \python -m ...\ but the grader (and the vally command) invoke \python3\. On Linux runners that may lack a \python\ shim the agent could hit command-not-found. Align the prompt to \python3\ in both eval.yaml and eval.vally.yaml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
5ad4286d06 |
skill-validator: pin SQLitePCLRaw.lib.e_sqlite3 to 3.50.3 (CVE-2025-6965) (#804)
The build/check pipeline was failing repo-wide with NU1903 (warnings as errors): Microsoft.Data.Sqlite 10.0.7 transitively pulls SQLitePCLRaw.lib.e_sqlite3 2.1.11, whose bundled SQLite is older than 3.50.2 and carries a known high-severity vulnerability (GHSA-2m69-gcr7-jv3q / CVE-2025-6965). A NuGet advisory update started flagging it, breaking every build. Pin the native lib to 3.50.3 (patched SQLite), following the existing transitive-pin pattern in this project. The 3.50.x native package is ABI-compatible with the 2.1.x managed provider; verified: restore is clean (no NU1903/NU1605), build succeeds with 0 warnings, and all 586 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
e3f0becd4c |
Fix test-gap-analysis skill activation in plugin eval runs (#801)
* Fix test-gap-analysis skill activation in plugin eval runs In plugin-mode eval runs (all dotnet-test skills loaded), test-gap-analysis lost activation: 'Acknowledge well-tested code' was stolen by assertion-quality and 'logic/null-check gaps' loaded no skill at all. Front-load the concrete 'would my tests catch a bug if the code changed' trigger phrasing the prompts use, and add a DO NOT USE -> test-gap-analysis redirect in assertion-quality for mutation-style reasoning. Both descriptions stay within the 1024-char cap; aggregate unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore Jest/TS activation signal for assertion-quality polyglot scenario The earlier trim of assertion-quality's description dropped the Jest/Vitest and toBeTruthy() signals, breaking plugin-mode activation for the 'Polyglot: Jest/TypeScript' scenario (agent answered directly, loading no skill). Restore those polyglot signals, and make the scenario prompt realistic — the prior prompt enumerated the full check catalog and Jest matcher list inline, acting as an answer key that let the agent self-serve without loading the skill. Verified locally: the scenario now activates assertion-quality in plugin mode across runs; all other assertion-quality scenarios still activate; description stays within the 1024-char cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
03f06321a4 |
Cover MSTESTxxxx analyzer diagnostics in writing-mstest-tests skill (#794)
* Cover MSTESTxxxx analyzer diagnostics in writing-mstest-tests skill Add a 'Fix MSTest analyzer diagnostics' workflow step mapping the common MSTESTxxxx rules to their idiomatic fixes, plus MSTestAnalysisMode guidance, instead of creating one skill per rule. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix MD012 markdown lint (trailing blank line) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: reword analyzer availability and fix table grammar - Don't tie MSTest.Analyzers availability to TestFramework 3.7; note metapackage/SDK/explicit reference. - Fix grammatically broken fix text for the MSTEST0002-0014 layout row. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify the MSTESTxxxx table is non-exhaustive; defer to full reference Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
0c98ce6bd9 |
Direct strategy must still run the Step 7 pre-completion gate (#793)
* Direct strategy must still run the Step 7 pre-completion gate The Direct strategy correctly skips the research/plan/implement sub-agents for small single-file tasks, but the wording let agents also skip the Step 7 pre-completion gate (test-gap-analysis + assertion-quality + scenario coverage) — treating a single-file task that enumerates specific behaviors as 'trivially small'. This is the dominant failure mode observed on behavior-enumerating tasks: the agent writes one test file directly and finishes with no assertion-strength or scenario-coverage check, producing weak assertions (mutation survivors) and missing required edge/negative cases. Clarify in both the generator Step 2 strategy table and the code-testing-agent SKILL.md that Direct trades away only the sub-agents, never the gate, and that a request naming a specific symbol or enumerating scenarios is not 'trivially small' and must run the gate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: align Direct gate trigger with Step 7 threshold; clarify gate in SKILL.md - Step 2 Direct cell no longer introduces a separate 'names a specific symbol' gate trigger that contradicted Step 7. It now defers to Step 7's own threshold (>=5 tests, or any enumerated behaviors/scenarios). - SKILL.md now names what/where the gate is: the generator's Step 7 (test-gap-analysis + assertion-quality). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Amaury Levé <evangelink@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Wendy Breiding
|
a7a744ce18 |
Make owner adjustments (#790)
* Add csharp-development skill and update CODEOWNERS and README.md - Introduced a new skill for C# development guidance. - Updated CODEOWNERS to include reviewers for the new skill. - Modified README.md to list the new csharp-development skill. - Added evaluation scenarios for the csharp-development skill. * Refactor C# evaluation scenarios to enhance production code review and validation guidance * Refactor production code review scenarios to improve clarity and focus on async error handling * Update CODEOWNERS to reflect current ownership and streamline reviewer assignments * Remove csharp-development skill documentation and evaluation tests * Remove csharp-development skill from README * Fix formatting in Constants.g.cs by adding missing semicolon * Revert Constants.g.cs to main * Remove csharp-development skill from CODEOWNERS * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Update CODEOWNERS to include additional reviewers for dotnet-ai skills --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
5d717dbdd1 |
Add prompt-scenario coverage check to code-testing-generator gate (#789)
* Add prompt-scenario coverage check to code-testing-generator gate The pre-completion gate already verifies assertion strength (pseudo-mutation and assertion-depth checks), but two recurring failure modes still slip through when the prompt enumerates specific behaviors: - Testing an *adjacent* function/helper instead of the exact feature named in the objective, leaving the requested behavior uncovered. - Covering only a single representative case when the scenario wording implies multiple variations or pins a condition to a specific position or structure. Add a third gate item that maps each enumerated scenario to a dedicated test, requires targeting the exact named function (preferring the canonical existing test file), and requires honoring range/positional qualifiers literally. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: genericize example, fix gate-count consistency - Remove benchmark-specific symbol names from the target-the-named-function bullet to avoid overfitting; phrase it generically. - Fix the gate intro that said 'The two skills below' now that there are three numbered items (the third is a prompt self-review, not a skill). - Update Step 8 and Rule 11 so re-running the gate includes the new prompt-scenario coverage check, not just test-gap-analysis + assertion-quality. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Amaury Levé <evangelink@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
14d727fee6 |
Fix test-anti-patterns skill activation for 5 evals (#786)
* Fix test-anti-patterns skill activation for 5 evals Sibling skills with overlapping descriptions were stealing activation from test-anti-patterns in plugin eval runs (coverage-analysis, assertion-quality, test-smell-detection). Reword descriptions so test-anti-patterns owns the umbrella 'audit my tests for anti-patterns' severity-ranked report, and add DO NOT USE redirects in the metric-focused siblings. Kept all descriptions within the 1024-char cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use toBeTruthy() call form in assertion-quality example Addresses review feedback: the Jest matcher example read like a property without parentheses. Description stays within the 1024-char cap (1023). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval validation error and improve activation triggers - assertion-quality eval.yaml/eval.vally.yaml: replace hyphenated 'assertion-quality' (the target skill name) with spaced 'assertion quality' in two scenario prompts, fixing the 'prompt mentions target name' validation error that biased baseline runs. - test-anti-patterns description: add 'what's wrong with my tests' / 'are these tests any good' / 'flaky tests' trigger phrasing to improve organic activation for the flakiness, well-written and polyglot scenarios (which intermittently failed to activate in plugin runs). Stays within the 1024-char description cap (1007). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use Jest matcher call form in assertion-quality polyglot prompt Addresses review feedback: write toBeDefined()/toBeTruthy()/not.toBeNull()/ toBe()/toThrow() in call form in the prompt so they read as matcher calls, consistent with the skill description examples. Regex assertions and rubric left untouched (they match agent output, which may use either form). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Sharpen test-anti-patterns description for flakiness/polyglot activation The flakiness and Python-pytest scenarios failed to activate even in isolated runs (where it's the only candidate skill), because their prompts enumerate the methodology and the description's keywords were too generic. Front-load the concrete trigger keywords those prompts use: Thread.Sleep, DateTime.Now, time.sleep, order-dependent, reflection coupling, and Python/pytest. Stays within the 1024-char cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make mixed/flakiness eval prompts realistic to fix plugin activation The mixed-severity and flakiness scenarios consistently failed to activate test-anti-patterns in plugin runs (detected=[] — the agent loaded no skill at all and answered directly). Both prompts enumerated the full anti-pattern catalog inline, acting as an answer key that made the agent self-sufficient. Replace the embedded checklists with realistic user asks while keeping the 'for .NET test anti-patterns' trigger, file references, severity-ranked output format, and read-only constraint. Rubric and output_matches assertions are unchanged — they validate the produced report. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
0c2b9710ee |
Fix activation for OutputType=Exe Directory.Build.props scenario (#785)
The 'Set OutputType=Exe only for test projects in Directory.Build.props' eval scenario was failing skill activation: the model answered from its own knowledge and proposed the IsTestProject condition the skill explicitly warns against. Strengthen the migrate-vstest-to-mtp SKILL.md description so the skill router matches this scenario: add the literal trigger phrase and surface the MSBuildProjectName (correct) and IsTestProject (anti-pattern) keywords in the USE FOR clause. Trimmed lower-value text to stay within the 1024-char description limit (1011). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
cd1a42dfa5 |
Fix skill activation for 5 MSTest v3-to-v4 eval scenarios (#784)
Five migrate-mstest-v3-to-v4 eval scenarios were answered from general model knowledge without loading the skill (NOT ACTIVATED): Migrate custom TestMethodAttribute from Execute to ExecuteAsync, Replace ExpectedExceptionAttribute with Assert.ThrowsExactly, Fix multiple v4 breaking changes, Fix TestMethodAttribute display name constructor, and Fix Assert.IsInstanceOfType out parameter removal. Reworded each prompt in both eval.yaml and eval.vally.yaml into an explicit 'I upgraded my test project to MSTest v4' fix request embedding the exact SKILL.md trigger phrase 'Help me fix this MSTest v4 breaking change' plus the matching USE FOR entry (Execute to ExecuteAsync, ExpectedExceptionAttribute removal, CallerInfo constructor on TestMethodAttribute, Assert.IsInstanceOfType out parameter removal). Graders and rubrics are unchanged; both YAML files parse cleanly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
9bc2eafec3 |
Fix skill activation for MSTest.Sdk/MTP v4 migration eval scenario (#782)
Reword the 'Handle MSTest.Sdk and MTP changes in v4' prompt in both eval.yaml and eval.vally.yaml from a passive 'What changed?' question into an explicit 'fix this MSTest v4 breaking change' request embedding exact SKILL.md trigger phrases, while still requiring the skill's knowledge to answer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
d54fc84225 |
Make vanilla series distinguishable on overlapping dashboard charts (#783)
* Make vanilla series distinguishable on overlapping dashboard charts Vanilla lines were hard to tell apart from the isolated/plugin lines when their values coincided. Give every vanilla series hollow diamond markers (larger than the round markers of the other series, transparent fill so the underlying line shows through) and a more prominent dash, across the quality and efficiency charts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
231592396c |
Fix skill activation for dropped-TFM v3 migration eval scenario (#781)
Reword the 'Handle dropped target framework during v3 migration' prompt in both eval.yaml and eval.vally.yaml from a passive yes/no question into an explicit migration request. The prior phrasing was answerable from general knowledge without loading the skill (NOT ACTIVATED pattern). The new prompt embeds exact SKILL.md trigger phrases and names the fixture TFMs without revealing which one is dropped. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
f389af86c7 |
code-testing-generator: mandatory pre-completion self-review gate (#768)
* code-testing-generator: mandatory pre-completion self-review gate Replaces the prose `Verify tests are implementation-specific'' bullet in Step 7 of the code-testing-generator agent with a mandatory pre-completion self-review gate that invokes two existing plugin skills: 1. `test-gap-analysis'' (pseudo-mutation check) against the source files tested and the produced test files 2. `assertion-quality'' (trivial/tautological assertion check) against the produced test files Both skills already ship in plugins/dotnet-test; this PR only wires them into the generator's workflow as a mandatory gate before declaring a run complete. The two skills' `When to Use'' sections are extended to list `called by code-testing-generator as a pre-completion self-review step'' as a recognised use case so the model does not refuse the invocation. The skill descriptions are unchanged (frontmatter is already at the 1024-char limit). Rule 11 of the generator agent is updated to list the gate alongside final build, final test, and coverage-gap review as mandatory for ALL strategies including Direct. A matching rubric item is added to the ContosoUniversity scenario in the code-testing-agent eval (yaml and vally) so the LLM judge can verify the gate was actually invoked on the trajectory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
11fd40e04c |
Add worked example files for C++, PowerShell, Ruby, Rust, Kotlin (#775)
* Add missing language worked examples Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: fix runnable example bugs (PS type/scope, Rust impl Trait arg, Catch2 macros) - powershell: accept [pscustomobject] in Get-InvoiceTotal; assert \ - rust: use concrete fn pointer type instead of impl Trait as generic arg - cpp: use REQUIRE_THROWS_WITH for message-substring assertions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
2bdec9b741 |
Expand C++ testing extension: coverage instrumentation, frameworks, CMake/ctest setup (#777)
* Expand C++ testing extension guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: group find -o predicates, include AppleClang, note libgtest-dev caveat Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
fc1e54f1a4 |
Harden Python testing guidance: reuse repo test runner/layout, handle heavy deps, never ship failing tests (#776)
* Harden Python testing guidance: reuse repo runner/layout, handle heavy deps, never ship failing tests Python is the weakest benchmark language (grade ~0.31): ~60% of generated tests fail and coverage stays ~2%. Trajectory analysis showed heterogeneous causes — custom runners not discovered (Django), custom layouts (ansible test/units), native-import failures (PyTorch), and over-reach that ships failing tests (Flask/FastAPI), which zeroes the grade via the pass-all gate. - Discover and reuse the repo's own test runner/layout before writing tests. - Add Django runner / DJANGO_SETTINGS_MODULE / subdir-runner guidance. - Verify heavy/native modules import before testing; otherwise scope down or skip. - Add a green-suite-or-remove finalization rule (delete tests that can't pass). - Start with high-certainty pure-function tests to bank coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: reword nonexistent Skip section reference Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: clarify env-wrapper usage for module vs script/-c, import probes, and green-suite check Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
5aa08ce41b |
Resolve Microsoft.AITools.BinlogMcp from nuget.org (#774)
The Microsoft.AITools.BinlogMcp package is now published on nuget.org as stable 1.0.0, so the binlog MCP server no longer needs the dnceng AzDO dotnet-tools feed or the --prerelease flag. Drop --prerelease and --add-source <azdo> from the binlog server args in both plugin.json and the .codex-plugin/.mcp.json mirror. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
754011b5ee |
Add workspace-integrity guardrail to code-testing agents (#773)
The code-testing-generator/implementer agents could treat an unusual or
scaffolded workspace (e.g. a gutted repo with an injected synthetic module)
as corruption and "repair" it with git checkout/restore/reset/clean or rm,
restoring deleted tracked files and testing the wrong code.
- Replace generator Rule 5 ("Clean git first - stash changes") with an
explicit "Treat the workspace as delivered" rule, and add a "Never mutate
version control" rule. Output must be purely additive test files.
- Add a no-revert/no-clean invariant to the implementer's edit boundaries.
- Add a 'workspace integrity' eval to the code-testing-agent suite
(eval.yaml + eval.vally.yaml). The fixture looks gutted: a metricsd project
whose real core/io modules are committed at HEAD but deleted from the
working tree, leaving only a synthetic 'synthstr' decoy. A git restore would
resurrect the deleted sentinel files; graders fail if they reappear and
require passing pytest tests for the module as delivered.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
YuliiaKovalova
|
76a5ec8cb3 |
Decouple agent runs from judging in skill-validator (--no-judge + cross-dir rejudge) (#769)
* Decouple agent runs from judging in skill-validator Allow baseline and treatment agent runs to execute in one parallel pool, with judging deferred to a final step, removing the serial precompute-baseline -> evaluate barrier. evaluate --no-judge: runs the requested agent arms and persists sessions.db + metrics, but skips all judging and requires no baseline file (exits 0 on successful runs). Forces session persistence; mutually exclusive with --baseline-out/--baseline-from; requires --results-dir. Only the agent model is validated (judge model recorded as metadata); warns when zero runs persist. rejudge --baseline-dir: cross-directory judging that pairs each treatment run with its baseline run by the BaselineStore key (prompt SHA + target SHA, persisted in a new sessions.db baseline_key column), runs the same pairwise/independent judges and pass/fail gates evaluate uses, and writes verdicts/reports. Baseline and treatment must share --model (rejected if either side mixes models); effective judge model resolves to --judge-model, else the treatment DB's, else the baseline DB's, rejecting a persisted mismatch. The inline per-run judging is extracted into a shared JudgeRunGroup helper reused by both the same-dir and cross-dir paths (no duplicated judging logic). Adds tests for baseline_key round-trip + legacy migration, ComputeScenarioKey, cross-dir pairing, and model/judge-model compatibility. Updates README and InvestigatingResults docs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Hoist ComputeScenarioKey to scenario level to avoid per-run fixture hashing Compute the baseline key once per scenario instead of once per run, so multi-run evaluations (--runs N) don't re-hash fixture files N times. The key is threaded into ExecuteRun/ExecuteAgentRun as a parameter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Memoize scenario-key fixture hashing for the duration of an evaluation Route the persisted baseline_key through an evaluation-scoped BaselineStore cache (ComputeScenarioKeyCached) instead of the static ComputeScenarioKey, so a scenario's setup fixtures are hashed at most once across every run, arm, and scenario of an evaluation. When a baseline store already exists (--baseline-from /--baseline-out) its warm input-hash cache is reused; otherwise a dedicated, side-effect-free cache (BaselineStore.ForKeyCache) is threaded through. Addresses the PR review comment about ComputeScenarioKey re-hashing large fixture directories. Adds a test asserting the cached variant equals the canonical static computation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bump session schema version to 3 for the baseline_key column The baseline_key column was added without bumping schema_info.version, so external tools keying off the version couldn't detect the new column. Introduce a single SchemaVersion constant (= 3), stamp it after migrations run, and drop the drift-prone version literal from the raw CREATE SQL so SetSchemaInfo is the sole source of truth. Update the schema-info and legacy-upgrade tests to expect version 3. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fail fast on judging-dependent options under --no-judge; fix README prefix Under --no-judge the noise test and overfitting fix cannot run (they require judging), so reject --noise-skills-dir and --overfitting-fix up front instead of silently ignoring them, mirroring the existing --baseline-out/--baseline-from guard. Add EvaluateCommandTests covering these early rejections (they short- circuit before any model/network call). Also prefix the decoupled-runs README example with 'skill-validator' for consistency with the other CLI invocations in the doc. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
a98fb44e52 |
code-testing-agent: pin down behavior in generated tests (#767)
Adds a new `Write Tests That Pin Down Behavior'' section to `unit-test-generation.prompt.md'' covering five universal test-craft principles: 1. Mutation thinking - each assertion would fail under a plausible bug 2. Property intersections - test combinations, not only coordinate axes 3. Behavior radius - assert on at least one secondary observable 4. Fixture realism - never set the parameter under test to a degenerate value 5. Quick self-review before declaring a test method done Mirrors the same depth requirements in `code-testing-implementer.agent.md'' Step 4 as a cross-language invariant block alongside the existing `Edit boundaries'' rules. Extends the existing `code-testing-agent'' eval rubrics (yaml and vally) with one or two depth-oriented bullets per scenario: * ContosoUniversity: minimal IsNotNull-only assertions + secondary observable check on controller actions * python-flask-tasks: minimal `is not None''-only assertions + at least one combined-property TaskService validation test * typescript-vitest-cart: minimal `toBeDefined''/`toBeTruthy''-only assertions + at least one intersection test (discount + tax + shipping together) Rationale and prior-art references in PR description. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
01b4875ac2 |
Add authoring-github-workflows skill + actionlint CI gate (prevent workflow-YAML breakage) (#760)
* Fix evaluation.yml: quote run-name so '#' is not parsed as a YAML comment The run-name added in #746 is an unquoted plain scalar containing ` #{0}`. In YAML, a space followed by '#' starts a comment, so everything from '#{0} ...' onward was stripped, leaving an unterminated ${{ }} expression. The file still parses as YAML (yaml.safe_load succeeds) but GitHub Actions rejects it ("This run likely failed because of a workflow file issue"), which broke every evaluation run on main after #746 merged. Wrapping the value in double quotes keeps the full expression intact; verified with actionlint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add authoring-github-workflows skill + actionlint CI gate Prevents the class of bug that broke evaluation on main (#746): an unquoted `${{ }}` workflow expression containing `#`, which YAML treats as a comment and silently truncates, producing a file that parses as YAML but that GitHub Actions refuses to run ("This run likely failed because of a workflow file issue", no jobs started). Two layers: - Knowledge: .agents/skills/authoring-github-workflows/SKILL.md teaches the `#`-as-comment trap and other quoting rules, and to validate with actionlint (plain YAML linters accept the truncated form). - Enforcement: .github/workflows/actionlint.yml runs a pinned actionlint on hand-authored workflows for any PR/push touching .github/workflows. Generated gh-aw files (DO NOT EDIT headers) are skipped; shellcheck/pyflakes are off so the gate focuses on workflow/expression errors. .github/actionlint.yaml declares the repo's custom runner labels (ubuntu-slim, windows-11-arm). Verified locally: actionlint exits 0 on all hand-authored workflows, and exits 1 on the original unquoted run-name, confirming the gate catches it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix markdownlint MD038: remove spaces inside code spans in SKILL.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments: actionlint config trigger, curl -f, syntactic-vs-semantic scope Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Harden actionlint workflow: pin checkout to SHA, disable persisted creds, verify actionlint checksum Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
bfad7293ad |
Raise per-plugin aggregate description cap 20,000 -> 22,000 (#772)
The dotnet-test plugin (largest and most active) reached ~20,397 aggregate description characters after the find-untested-sources-polyglot skill merged (#735), exceeding the 20,000 cap and failing the skill-validator 'check' job on every PR that merges with main (e.g. #767, #768) as well as main itself. This is legitimate plugin growth — a useful cross-language skill — not metadata bloat. The cap's own comment notes it is an unvalidated local heuristic meant to 'leave enough headroom for reasonable plugin growth', with precedent for raising it as plugins grow (15,000 -> 20,000). Bump to 22,000 to restore ~1.6k headroom rather than degrade the routing keyword lists of existing dotnet-test skills (owned by @dotnet/dotnet-testing) to squeeze back under the old limit. Tests reference SkillProfiler.MaxAggregateDescriptionLength dynamically, so no test changes are required. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Daniel Plaisted
|
8a437b1d4f |
Add anti-pattern for forking a project instance via <MSBuild> with path-neutral global properties (#770)
Adds msbuild-antipatterns AP-22 (in references/additional-antipatterns.md): using
the <MSBuild> task to build/publish a project with an extra global property that
does not change the output path forks a distinct instance sharing the same
OutputPath/IntermediateOutputPath as the instance the graph already builds,
causing duplicate builds and parallel-write races on PDBs, *.sourcelink, and
project.assets.json. Covers both shapes:
(a) a project re-invoking itself (publish-on-build), and
(b) project A invoking Build/Publish on another project B it consumes
(e.g. a test/layout project publishing a tool) — the more common trigger.
GOOD fixes: (a) static _IsPublishing + DependsOnTargets with a cycle guard;
(b) let the producer publish in its own build and have the consumer sequence it
with a non-output ProjectReference instead of re-publishing it. Also updates the
SKILL.md pointer (AP-16 through AP-22) and the quick-reference checklist.
Extends check-bin-obj-clash with the matching cause (self or cross-project
<MSBuild> Build/Publish with a path-neutral global property), binlog-based
detection via the evaluations/global-properties/double_writes tools, the two
fixes, and an _IsPublishing row in the global-properties table.
Adds eval coverage: a publish-on-build smell in the msbuild-antipatterns review
fixture (LibA.csproj) plus a matching rubric line in eval.yaml and eval.vally.yaml.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
Amaury Levé
|
76b4c73a39 |
Add find-untested-sources-polyglot skill (tree-sitter, 9 languages) (#735)
* Add find-untested-sources-polyglot skill (tree-sitter, 9 languages) Sibling of the C# find-untested-sources skill that uses tree-sitter (via tree-sitter-language-pack) to extract declarations + imports across Python, TS/JS, Go, Java, Rust, C#, and Ruby with no build step. Output schema mirrors the C# skill's so prompts can consume either tool. Pairing strategies per language: - Import resolution (Python module path, TS/JS relative ./../index.*, Go pkg/<name>.go, Java FQCN, Rust use::, Ruby require). - Identifier overlap (every >=4-char token in the test source is cross-referenced with the declared-name index). Smoke-tested on: - python-flask-tasks fixture: 8 source / 0 test, 8 untested. - typescript-vitest-cart fixture: 8 source / 0 test, 8 untested (node_modules pruned). - AITestAgent C# repo: 3138 source / 761 test, 1419 tested, 1719 untested, 15 orphan. - gh-skills Go repo: 14 source / 7 test, 8 tested, 6 untested, 0 orphan. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Polyglot: union structure+symbols for declarations (fixes Go types, Python methods) tree-sitter-language-pack's ProcessResult exposes two complementary views: - structure: top-level items (classes, functions, structs, traits) - symbols: flat declared-name list Neither alone is complete: - Go: structure has methods/functions but NOT 'type' declarations; symbols has the types. Without union, a test that references a struct name finds no matching source and the source is incorrectly marked untested. - Python: structure stops at top-level, symbols also lists nested methods. - Rust: structure has structs and impls; symbols has fn names inside impls. Fix: union both lists, then drop kind='module'/'namespace' (these are packaging items that would cause false positives, e.g. Java's tree-sitter output emits the package name 'com' as a Module). Verified across all 9 supported languages with synthetic source+test+orphan fixtures (python, javascript, typescript, tsx, go, java, rust, csharp, ruby): each fixture now reports tested=1, untested=1, orphan=0. Real-repo regression check (AITestAgent C#): tested 1419 -> 1429, untested 1719 -> 1709 (10 sources now correctly paired via types previously missing from the declared-name index). Orphan count unchanged at 15 (no false-positive pairings introduced). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback on find-untested-sources-polyglot Code fixes (find_untested_sources.py): - Java test-file detection: drop the `name.startswith("test")` clause so data files like `testdata.java` are no longer misclassified as tests. Keep the documented `*Test.java` / `*Tests.java` rule and the path segment heuristic. - C# test-file detection: tokenize the (PascalCase-preserving) stem and treat the file as a test only when the final word is `Test` or `Tests`. This correctly classifies `UserServiceTests.cs` while no longer misclassifying `Contest.cs`, `Latest.cs`, `Manifest.cs`. - Identifier-overlap pairing: iterate the test's referenced identifiers and do O(1) lookups in `by_decl` instead of scanning every declaration in the index. Pairing is now O(#test_identifiers) per test instead of O(#decls × #tests) across the run, which matters on large repos. - JS/TS relative-import normalization: collapse `<seg>/../` segments in a fixed-point loop instead of a single regex pass so chained imports like `../../foo` and `a/b/../../c` resolve fully. Doc fix (SKILL.md): - Clarify that `symbols` is unioned with `structure` rather than being used only as a fallback when `structure` is empty, matching the actual parse_file implementation. - Tighten description to stay within the per-skill (1,024 chars) and plugin aggregate (20,000 chars) limits enforced by skill-validator. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Allow tree-sitter.github.io in known-domains for find-untested-sources-polyglot The polyglot skill's SKILL.md links to https://tree-sitter.github.io/ for the tree-sitter project; add the domain to eng/known-domains.txt so the skill-validator reference scanner stops failing with EXTERNAL-DOMAIN. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #735 review feedback (round 2) - parse_file: log a stderr WARN when tree-sitter parsing throws instead of silently returning a 0-declaration FileInfo, so users can see when results are degraded by parser failures. - Python relative imports: _python_module_to_path now respects PEP 328 leading dots, resolving 'from .utils import x' against the test file's package rather than treating it as an absolute 'utils.py' at repo root. Returns an empty candidate set when no test context is available or when the imported name isn't captured ('from . import x'), to avoid false-positive pairings. - Go import resolution: _build_indexes now produces a by_filename index, and _resolve_test_imports looks up '<pkg>.go' in O(1) per import instead of scanning every source path for each import target. - tested_sources: emit sorted by path for deterministic JSON output across runs (matches the existing sort on untested_sources / orphan_tests). - SKILL.md: JS/TS test-detection row now lists the 'test' path segment to match the implementation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
d0924d4f17 |
Add find-untested-sources skill (C# parse-only test pairing) (#733)
* find-untested-sources: C# static test-pairing prototype Parse-only Roslyn analysis that maps C# source files to the test files referencing their declared types and lists production sources with no referring test. No Compilation, no MetadataReferences, no binding. - File discovery prunes bin/obj/node_modules/.git and generated *.g.cs. - Test classification by .csproj suffix or test-SDK reference. - Source index records (ShortName, Namespace, FilePath) per parsed file. - Test scan walks IdentifierTokens, disambiguates strictly against the test file's using directives + enclosing namespace. - Suggests a test-file path by mirroring the source under the test project that already <ProjectReference>s the source's project. Runs in ~9s on a ~3,900 .cs-file repo (AITestAgent + msbench fixtures). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback on find-untested-sources Code fixes (Find-UntestedSources.cs): - ProjectFor: sort .csproj matches and wrap enumeration in try/catch so selection is deterministic when a directory contains multiple project files. - GetNamespaceFor: walk all enclosing namespace declarations and join from outermost to innermost so nested namespace blocks (namespace A { namespace B { ... } }) return "A.B" rather than just "B". - Test scan: stop unconditionally skipping global-namespace declarations. Global types are visible without a using directive, so accept them as visible to any test file. Previously, types without a namespace were never attributed to any test and always reported as untested. - BuildProductionToTestProjectMap: sort the recursive .csproj enumeration and wrap in try/catch so the "first write wins" rule selects the same test project across runs/machines regardless of filesystem ordering. - IsSkippedFile: switch suffix comparisons to OrdinalIgnoreCase and collapse the five if statements into a single loop over an array. Skill metadata (SKILL.md): - Tighten the description so the skill stays under the 1,024-character per-skill limit and the dotnet-test plugin stays under the 20,000-char aggregate limit enforced by skill-validator. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
05597a329c |
migrate-xunit-to-mstest: switch file-state graders from output to file (no false negatives) (#761)
The eval was tripping false-negative graders: the skill activates and the agent EXPLAINS what it converted (e.g., "I removed using Xunit;", "Converting [Theory] to [TestMethod]"), so output-not-matches on patterns like `using Xunit;`, `[Theory]`, `Trait\(`, `IClassFixture`, `TestContext.Current.CancellationToken`, etc. fired on the agent's prose even when the migrated file was correct. Conversely output-matches on `Microsoft\.VisualStudio\.TestTools\.UnitTesting`, `[TestClass]`, `[DataRow(`, etc. could match the agent's explanation without verifying the file. Convert every grader that is really checking FILE state into file-contains / file-not-contains (with the relevant glob: **/*.cs or **/*.csproj). Keep the prose-checking graders where the rubric genuinely wants the agent to talk about something (unsupported-TFM explanation, scope-decision narrative, already-on-MSTest "no migration needed" message). The two CancellationToken positive graders stay on prose because they use regex alternation that file-contains (literal only) can't express; the file is still indirectly checked by the dotnet test run-command. Mirrored across both eval.vally.yaml (vally runner) and eval.yaml (skill-validator). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
d9c1e7a801 |
code-testing-agent: mention find-untested-sources for C# discovery (#734)
* code-testing-agent: mention find-untested-sources for C# discovery Adds a conditional pointer (gated on 'when available') to the find-untested-sources skill in two places: - SKILL.md Step 3 (Research Phase): high-level note for C# / .NET multi-file scopes — prefer the helper over manual find/grep/glob walks. - code-testing-researcher.agent.md Section 7 (Discover Preexisting Tests): directive instruction telling the researcher to invoke the helper before manually pairing source <-> test files, and to use its source_to_tests / untested output to fill the research document. Both callouts are phrased as 'when available', so installations without the find-untested-sources skill continue to work via manual discovery. Adds no behavior for non-C# repos. Context: in a 5x136-instance internal experiment on the msbench .NET test bench, adding equivalent pointers to the routed code-testing-agent yielded a 15.67% input-token reduction at neutral pass rate — the model trusted the documented pairing heuristics and skipped its own discovery walk. The helper itself was not invoked in those runs (the Copilot CLI router did not auto-load the sibling skill), so the measured win comes from the doc text causing the model to short-circuit its manual exploration, not from the helper executing. Depends on dotnet/skills#733 (which adds the find-untested-sources skill itself). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * code-testing-agent eval: grade find-untested-sources usage on C# scenario PR #734 adds a doc pointer steering the researcher toward the `find-untested-sources` skill on .NET / C# multi-file tasks. The existing ContosoUniversity scenario is the natural fit: it's the only C# multi-file scenario in this eval, and the new pointer is .NET- scoped. Add one rubric item to that scenario (in both eval.vally.yaml and the legacy eval.yaml) asking the grader to verify the researcher actually leveraged the helper — either by citing its `source_to_tests` / `untested` JSON output in `.testagent/research.md`, or by executing `scripts/Find-UntestedSources.cs` — instead of falling back to manual `find` / `grep` / `glob` walks. Gated on `when available in the workspace` to match the SKILL.md wording, so installs without find-untested-sources are not penalized. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
c8be21018f |
Fix evaluation.yml: quote run-name so '#' is not parsed as a YAML comment (#759)
The run-name added in #746 is an unquoted plain scalar containing ` #{0}`. In YAML, a space followed by '#' starts a comment, so everything from '#{0} ...' onward was stripped, leaving an unterminated ${{ }} expression. The file still parses as YAML (yaml.safe_load succeeds) but GitHub Actions rejects it ("This run likely failed because of a workflow file issue"), which broke every evaluation run on main after #746 merged. Wrapping the value in double quotes keeps the full expression intact; verified with actionlint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
bcaa91882c |
Add reusable shared baseline to skill-validator evaluate (--baseline-out / --baseline-from) (#754)
* Add shared/precomputed baseline reuse to skill-validator evaluate (#751) Add --baseline-out and --baseline-from options to the evaluate command so the no-skill/no-agent baseline arm can be computed once and reused as a shared control group across multiple skill/agent evaluations. This eliminates redundant baseline runs and removes baseline run-to-run variance from cross-config comparisons. - New BaselineStore + BaselineFile/BaselineScenarioEntry models, keyed per scenario on SHA-256(prompt) with a header recording version, model, validator version and runs. Load validates version + model and fails fast on mismatch or missing scenarios. - Register the new serializable types in SkillValidatorJsonContext (AOT source-gen). - Wire two mutually-exclusive CLI options into ValidatorConfig; thread an optional BaselineStore through both execution paths. - On reuse, skip the baseline agent run, its assertions/constraints/ task-completion/judging, and attribute no extra pairwise tokens to the baseline; report the scenario with the baseline-reused session phase and a reused status. In write mode, record each scenario's averaged baseline and persist it after the run. - Add unit tests for BaselineStore and document the feature in README and InvestigatingResults. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bind reused baseline to fixture identity (targetSha), not just prompt Align baseline reuse with the (prompt, model, targetSha) identity contract from the upstream eval-harness design: previously the per-scenario reuse key was the prompt SHA + model only, so two scenarios that share an identical prompt but feed the agent different input artifacts (e.g. a different build.binlog) would collide and silently reuse the wrong baseline. - Add BaselineScenarioEntry.TargetSha: a SHA-256 over the scenario's materialized inputs — files auto-copied via copy_test_files, explicit setup files (inline content or copied sources), and the setup command recipe. The reuse key is now (promptSha, targetSha); both must match. Bump the on-disk schema to version 2. - Memoize target hashing per process via a cheap, file-I/O-free setup signature to avoid re-hashing large fixtures across the N runs. - Thread the originating eval.yaml path into Record/TryGetBaseline/ FindMissingScenarios so inputs can be fingerprinted. - Tests: target SHA is stable and content-sensitive; same-prompt/different-fixture scenarios do not reuse each other's baseline and are surfaced by FindMissingScenarios. Update README and InvestigatingResults. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Harden baseline reuse identity after review Address rubber-duck review findings on the shared-baseline feature: - Fix Sha256Hex 32-byte bug: a 32-byte input was treated as an already-computed digest and not hashed. Split into Sha256Hex (always hash) + HexDigest (encode existing digest). - Broaden reuse identity: the cached baseline RunResult depends on the judge model and on per-scenario evaluation criteria (rubric, assertions, expect/reject tools, turn/token/timeout limits). Add JudgeModel to the baseline header (validated on load) and fold the criteria into the per-scenario targetSha so changing them invalidates reuse instead of silently serving a stale result. - Mirror AgentRunner.SetupWorkDir exactly when hashing copied fixtures: exclude only the top-level eval.yaml (nested eval.yaml files are copied, so they must be hashed). - Make the target-SHA cache instance-scoped (memoizing only the expensive fixture-input hashing) so it can't serve stale hashes or leak across evaluations/tests; hash inline file Content in the cache key instead of embedding it. - Deterministic Save ordering (Name, PromptSha, TargetSha); guard Load against null Scenarios; enrich FindMissingScenarios output with the eval path. - Document that setup commands are fingerprinted by recipe, so reuse assumes they are deterministic/hermetic. - Tests + docs updated; add judge-model-mismatch and criteria-identity tests (562 pass). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments on baseline reuse - Mirror AgentRunner.SetupWorkDir exactly when hashing copied fixtures: enumerate only the files actually copied (top-level siblings except eval.yaml, recursing into directories) and skip reparse points and out-of-root junctions, instead of blindly hashing every file under the eval directory. This keeps the fixture identity restricted to the intentionally-copied set so stray output/log files can't poison reuse. - Stream baseline JSON to/from disk (File.OpenRead/File.Create with JsonSerializer) so large baselines never materialize as one giant in-memory string. - Enrich the fail-fast 'missing scenario' output with the eval path and short prompt/target SHA prefixes so it is actionable when scenario names collide across eval files. - Add a test locking in recursive (nested-directory) fixture hashing. 563 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address second round of PR review comments - Skip top-level reparse points in EnumerateCopiedFixtures (not just nested ones) so a top-level symlink/junction can't cause hashing of data outside the eval directory; code now matches the docstring. - Record uses first-writer-wins (TryAdd) instead of overwriting, so a scenario identity recorded by multiple parallel targets yields a deterministic --baseline-out regardless of completion order. - Persist the baseline judge result to the session DB even when the baseline is reused, so the registered 'baseline-reused' session record is complete for downstream investigation tooling (pairwise was already saved; the judge result was incorrectly gated on a fresh run). - Add first-writer-wins test (564 pass). Note: BaselineStore stays internal — the test project already has InternalsVisibleTo, so it compiles. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Pin MessagePack to 2.5.301 to fix NU1903 vulnerability The transitive MessagePack 2.5.198 (via GitHub.Copilot.SDK -> StreamJsonRpc) has a high-severity vulnerability (GHSA-hv8m-jj95-wg3x) that fails the build under TreatWarningsAsErrors. Pin a direct reference to the patched 2.5.301. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: bare-filename fixture hashing, clone reused baseline, consistent token attribution - ComputeInputsSha: normalize evalPath via Path.GetFullPath so a bare filename still hashes sibling fixtures (avoids TargetSha collisions / unsafe reuse). - RunMetrics.Clone(): per-run copy with fresh collections; reuse paths now clone the cached baseline so concurrent evaluations never share a mutable instance. - Pairwise judge tokens attributed to both compared runs in every mode (the baseline clone makes this safe), keeping token deltas comparable across --baseline-from modes. - Reword Record first-writer-wins doc to describe the within-run stabilization guarantee rather than order-independence. - Add tests for bare-filename fixture hashing and clone isolation (566 pass). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
8854249222 |
migrate-xunit-to-mstest: forbid CancellationToken.None replacement explicitly (#758)
* migrate-xunit-to-mstest: forbid CancellationToken.None replacement explicitly The TestContext.Current.CancellationToken guidance already forbids replacing it with a fresh CancellationTokenSource, but real-world agent migrations have been observed substituting CancellationToken.None instead, which silently drops the test host's cancellation linkage. Call CancellationToken.None out explicitly so both common shortcuts are covered. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: mention property injection for MSTest < 3.6 Per Copilot review on PR #758: the section above notes that projects pinned to MSTest < 3.6 must use property injection. Reword the CancellationToken callout to cover both constructor and property injection so the guidance stays correct in that supported scenario. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
0c0f6f0263 |
Fix PR-triage eval trigger: dispatch evaluation.yml instead of bot label (#746)
The triage worker added the 'evaluate-now' label via GITHUB_TOKEN, but label events emitted by GITHUB_TOKEN do not start workflows (GitHub's recursion guard), so evaluation.yml's pull_request_target:[labeled] entry point never fired for the bot (repro: PR #745). workflow_dispatch and repository_dispatch are the only token-initiated events exempt from that guard. The worker now dispatches evaluation.yml directly via 'gh workflow run' with a pr_number input, routed through the existing gate job so the path is identical to /evaluate. A dispatched run's head_sha is the default branch (not the PR head), so idempotency now matches the deterministic run name 'Evaluate PR #<n> @ <sha7>'. The 'evaluate-now' label remains a valid human entry point. Worker granted actions:write for 'gh workflow run'. |
||
|
YuliiaKovalova
|
74c58505e5 |
Improve dotnet-template-engine plugin: accuracy, dedup, and two new skills (#745)
* Improve dotnet-template-engine plugin: accuracy, dedup, and two new skills Fix inaccurate reserved-shortName guidance, consolidate validation rules into a single skill, expand discovery mappings, add explicit CPM/version steps, and introduce template-comparison and template-smart-defaults skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix .codex-plugin manifest indentation; add evals for new skills Make .codex-plugin/plugin.json byte-consistent with plugin.json (2-space indent on the agents line). Add eval.yaml + eval.vally.yaml capability evals for the new template-comparison and template-smart-defaults skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: source reserved list from dotnet new --help, note workload/package availability, tighten version-refresh - Clarify the reserved shortName set is the current dotnet new subcommands (authoritative source: dotnet new --help); create is verified as a real subcommand (alias behind dotnet new <template>). - template-discovery: note that some mapped short names (maui, winui3, aspire, func, orleans) need workloads/template packages, with fallback to dotnet new list/search. - template-instantiation: keep template versions by default; if refreshing, use dotnet list package --outdated + user confirmation and constrain to same major/minor rather than always latest stable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address 2nd review round: enforce eval negatives, split combined assertion, reframe reserved list - smart-defaults evals: enforce --no-https absence (auth scenario), absence of minimal-API flag (controllers scenario), and no newer --framework TFM when net8.0 is explicitly required, using output_not_contains/output_not_matches. - comparison eval: split the combined (auth|aot|docker|controllers) check into four separate output_matches assertions so partial comparisons fail. - template-validation/authoring: reframe the reserved shortName list as non-exhaustive examples and source the authoritative set from dotnet new --help; drop the specific create-alias assertion in favor of parsing-ambiguity wording. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make template-comparison evals robust to CI SDK currency Switch the Blazor comparison scenario from blazorserver (absent in the CI SDK) to blazor (Blazor Web App) vs blazorwasm, both reliably present in .NET 8+, and instruct the agent to inspect each via --help. Update the SKILL.md example reference for currency. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add CLI-failure resilience guidance to discovery/comparison skills The isolated eval runs failed because the agent ran 'dotnet new <t> --help', hit the template engine's global-mutex/persistence error (common when the command runs concurrently in a sandbox), and then returned no answer at all. Instruct both skills to run 'dotnet new' calls sequentially, retry once on a transient mutex/persistence error, and fall back to the intent/parameter mapping so a concrete answer is always produced instead of empty output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: accurate flags + robust negative-assertion prompts - template-smart-defaults SKILL.md: drop the non-existent --publish-aot flag. Clarify --aot is a dotnet new flag only on templates that expose it (console/worker/grpc, not webapi) and that publish-time AOT is the MSBuild PublishAot=true property, not a dotnet new flag. - template-discovery SKILL.md: replace the hardcoded --enable-docker mapping (not a real flag on common templates) with generic 'confirm with --help'. - smart-defaults evals: tighten the negative-assertion prompts to output only the command line and not mention unused flags, so a negated explanation can't trip output_not_contains/output_not_matches. Switch the AOT scenario from webapi to worker (which actually supports --aot). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review round 3: scope negative checks, de-emphasize stale lists - smart-defaults evals: anchor the negative assertions to the 'dotnet new' command line (same-line regex) instead of whole-output substring/regex, so a flag mentioned only in prose can't fail the test. - template-validation / template-authoring: mark the dotnet new subcommand examples as illustrative/version-dependent and tell readers not to hardcode them; the live 'dotnet new --help' output is canonical. - template-comparison: fix the example table's AOT row — webapi/webapp do not expose a --aot template flag; native AOT is publish-time via PublishAot. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
0a387fcdfe |
code-testing-implementer: add mandatory harness-discovery check (#757)
* code-testing-implementer: add mandatory harness-discovery check
Generic CI/benchmark verifiers run the framework's default discovery from the repo root. Tests that pass via a scoped command (e.g. 'dotnet test MyProject.Tests.csproj', 'bundle exec rspec subgem/spec', 'Invoke-Pester -Path ./tools') but are invisible to the harness count as 0 generated tests.
Repeatedly observed in msbench cta-nightly runs:
- new C# test project never 'dotnet sln add'ed (ocelot)
- Pester tests placed under custom directories invisible to default Invoke-Pester (homebrew, ruby/ruby)
- RSpec specs placed in a sub-gem's spec/ dir invisible from repo root (fastlane)
Changes:
- code-testing-implementer: capture baseline test count in Step 2, new Step 7 'Verify Harness Discovery (MANDATORY)' with concrete failure examples, HARNESS_DISCOVERY line in report template, reminder after Step 3 to revisit registration if Step 4 creates a new project.
- code-testing-researcher: instruct to record BOTH scoped test command and harness-equivalent discovery command in .testagent/research.md.
- dotnet.md: strengthen 'Registering a new test project' heading to MANDATORY when dotnet new was used; add 'Harness Discovery Check' section with 'dotnet test <solution> --list-tests' from repo root.
- powershell.md: add 'Harness Discovery Check' section with default-config Invoke-Pester from repo root.
- ruby.md: add gem-monorepo trap guidance (fastlane, ruby/ruby) in Test Placement Contract; add 'Harness Discovery Check' section with 'bundle exec rspec --dry-run' from repo root.
* Address review feedback on harness-discovery check
- dotnet.md: replace stale `Step 8 cleanup` reference with the actual Step 3 (`Register Test Project with Build System`).
- dotnet.md: demote `Harness Discovery Check` to `###` so it nests under the parent `## .csproj / .sln Handling` section alongside `### Registering...`.
- dotnet.md: replace `\s\{4\}` (non-POSIX) in the grep regex with 4 literal spaces so the count works under BRE.
- ruby.md: group the rake/rails fallback with `{ ...; } | wc -l` so the pipe applies to both branches of `||`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
Amaury Levé
|
9be44ad6e5 |
migrate-xunit-to-mstest: add learnings from dotnet/sdk migration (#756)
* migrate-xunit-to-mstest: add learnings from dotnet/sdk migration Refines the skill based on a real migration of 5 test projects in dotnet/sdk: - MSTest.Sdk implicit global using: do not add `<Using Include='Microsoft.VisualStudio.TestTools.UnitTesting' />` or per-file `using` in MSTest.Sdk projects (it's already in scope). Note Option A (`MSTest` metapackage) still needs the per-file using. - `Assert.IsExactInstanceOfType<T>` (MSTest 4.1+) is the proper single-call equivalent of xUnit's exact-type `Assert.IsType<T>`; previous guidance silently degraded it to assignable semantics (closes #755). - `Assert.AreSequenceEqual` (MSTest 4.3+) is the modern element-wise equivalent of xUnit's `Assert.Equal` on `IEnumerable<T>`, avoiding the `CollectionAssert` + `.ToList()` dance and the MSTEST0065 trap on plain `AreEqual`. - Clarify `AwesomeAssertions` ships in the `FluentAssertions` namespace, so it's a no-source-change swap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * SKILL.md: split IsType/IsAssignableFrom row to avoid implying drop-in equivalence Address dotnet/sdk#54727 review comment: the Step 6 summary table conflated xUnit's exact-type Assert.IsType<T> and assignable Assert.IsAssignableFrom<T> into a single row, which implies both map interchangeably even though their semantics differ. Split into three rows -- IsType<T> -> IsExactInstanceOfType<T>, IsNotType<T> -> IsNotExactInstanceOfType<T>, IsAssignableFrom<T> -> IsInstanceOfType<T> -- and explicitly call out the silent-weakening trap. The cheatsheet already maps these correctly; this aligns the SKILL.md summary with the cheatsheet so the at-a-glance table doesn't suggest a wrong mapping. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * SKILL.md: resolve Step 3/Step 4 conflict on the MSTest using for Option B Address review feedback on dotnet/skills#756: Step 3 said to skip the per-file 'using Microsoft.VisualStudio.TestTools.UnitTesting;' on Option B (MSTest.Sdk provides it as an implicit global using), but Step 4 unconditionally instructed replacing xUnit usings with that MSTest using -- so an agent following Step 4 literally would re-add the redundant using on Option B projects. Also drop the dangling 'Step 4 rewriter' phrasing in Step 3 (there is no rewriter in Step 4, just a list of mechanical rewrites the agent applies). Step 4 now explicitly says to skip the MSTest using on Option B and only remove the 'using Xunit;' lines, matching Step 2's note and Step 3's branch instructions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Evangelink <amaury@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Abhitej John
|
8b7867331c |
Pin github/gh-aw-actions to commit SHA and add pinning guard (#749)
* Pin github/gh-aw-actions to commit SHA and add pinning guard The agentic workflows referenced github/gh-aw-actions/setup and setup-cli by mutable tag (@v0.77.5) because .github/aw/actions-lock.json still carried a stale pre-migration entry (github/gh-aw/actions/setup@v0.71.5) with no entry for the action the compiled workflows actually use. With no matching lock entry, gh aw compile fell back to emitting the bare tag. Refresh the lock with SHA-pinned entries for github/gh-aw-actions/setup@v0.77.5 and setup-cli@v0.77.5 (commit 3ea13c02...), and regenerate the workflows via gh aw compile so every uses: ref, decorative comment, and manifest sha is pinned. This mirrors how dotnet/msbuild pins the same action and unblocks enabling org-level "require actions pinned to a full-length commit SHA". Also add enforce-action-pinning.yml, a self-contained PR/push check that fails any workflow referencing an action by tag/branch. The agentic workflows are schedule/issue-triggered and never run on PRs, so neither /evaluate nor the runtime org policy gates them pre-merge; this check does. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Remove enforce-action-pinning guard workflow Drop the CI guard in favor of relying on the org/repo 'Require actions pinned to a full-length commit SHA' setting, per review feedback. The lock-file fix and SHA pins remain the root-cause fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |