skills/dotnet__skills
1
0
Fork 0
mirror of https://github.com/dotnet/skills.git synced 2026-09-20 09:49:54 +08:00
Code Issues Packages Projects Releases Wiki Activity
589 Commits 131 Branches 2 Tags
eval-exec-opus48
Commit Graph

7 Commits

Author SHA1 Message Date
Amaury Levé a677ec43d2 Consolidate dotnet-test-frameworks into test-analysis-extensions (#851) 
* Consolidate dotnet-test-frameworks into test-analysis-extensions

Remove the orphaned dotnet-test-frameworks reference skill, whose content
was a duplicate subset of test-analysis-extensions/extensions/dotnet.md.
Nothing actually loaded it. Redirect the remaining references and drop its
eval tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Make exp-test-maintainability Step 1 self-contained and fix activation

Address PR review: exp-test-maintainability (dotnet-experimental plugin)
must not point at test-analysis-extensions (dotnet-test plugin) — inline the
.NET framework markers instead. Also strengthen the description/When-to-Use
triggers so the 'each new case needs a whole new method / suggest a better
structure' scenario reliably activates the skill instead of routing to a
test-writing skill.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Revert exp-test-maintainability description broadening; keep self-contained markers

The description broadening regressed activation (3/4 -> 0/4 across a
variance-dominated single eval run, CV up to 924%). Description is the
activation lever, so restore the original wording and keep only the
review-comment fix (inlined .NET framework markers in Step 1).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Fix scenario 1 eval: use project fixture so the skill activates

Scenario 1 pasted a self-contained snippet inline and asked to 'suggest a
better structure', which the base model answers directly without loading the
skill -> NOT ACTIVATED with no headroom. Convert it to the project-fixture
pattern used by the activating scenarios (3 & 4): move the repetitive
InputValidatorTests into a Validation.Tests fixture and ask the model to
analyze the project. Mirror the change in eval.vally.yaml.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Fix scenario 2 eval: use project fixture so the skill activates

Scenario 2 also pasted a self-contained snippet inline (well-maintained
Auth.Tests) and asked to 'review these tests', which the base model handles
without loading the skill. Convert to the project-fixture pattern used by the
activating scenarios; add fixtures/well-maintained/Auth.Tests. Mirror in
eval.vally.yaml. All four scenarios now reference a project on disk.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Add reject_tools to exp-test-maintainability evals to create headroom

All four scenarios scored NOT ACTIVATED / no-headroom (#8 in
InvestigatingResults.md): baselines are already 4.0-5.0/5, so loading the
skill only adds token/tool overhead with no offsetting quality gain.

Match the convention of the sibling analysis skill test-anti-patterns:
reject bash/edit/create on every scenario. This skill is analysis-only
('do not modify any files'), so the constraint is a no-op for correct
behavior but levels the playing field between baseline and skilled runs —
scoring focuses on answer quality (weight 0.40+0.30) instead of tool-induced
overhead. Applied to both eval.yaml and eval.vally.yaml.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Address review: make well-maintained fixture self-contained and deterministic

- Import System.Security.Authentication for AuthenticationException
- Replace wall-clock assertion (token.ExpiresAt > DateTime.UtcNow) with a
  deterministic one anchored on a fixed clock (FixedNow), and drive the
  injected FakeClock from that fixed time. This removes the flakiness and
  makes the FakeClock injection meaningful — a better exemplar for the
  'well-maintained' scenario.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
 2026-07-01 16:53:02 +02:00
Petr Pokorny 06dd46e5f1 Enable VS Code subagent fan-out for the dotnet-test code-testing agents (#847) 
* Add VS Code subagent metadata to dotnet-test code-testing agents

Enable the code-testing-* Research-Plan-Implement pipeline to fan out as
subagents in VS Code while keeping the GitHub Copilot CLI working.

Frontmatter (VS Code "coordinator/worker" pattern; portable tool aliases that
map in both VS Code and the CLI):
- code-testing-generator: tools: [agent, read, search, edit, execute] + an
  agents: list (researcher, planner, implementer, builder, tester, fixer,
  linter); softened the three runSubagent({ agent, prompt }) blocks to
  tool-agnostic delegation prose.
- code-testing-implementer: tools: [agent, read, search, edit, execute] +
  agents: (builder, tester, fixer, linter).
- Leaf agents (researcher/planner/builder/tester/fixer/linter):
  tools: [read, search, edit, execute].

Why explicit tools (not ["*"]): VS Code has no all-tools wildcard for 	ools:
and a subagent's 	ools: overrides its inherited set, so ["*"] matched nothing
and stripped subagents of file tools (they ran without read/edit/search). The
CLI treats ["*"] as all-tools, so this was VS-Code-specific. The portable
aliases agent/read/search/edit/execute map to real tools in both environments;
agents: is ignored by the CLI.

README: document the VS Code chat.subagents.allowInvocationsFromSubagents
setting (off by default) needed for the nested implementer->builder/tester/
fixer/linter layer to fan out on large scopes; the CLI has no such gate.

Validated end to end: VS Code shows researcher->planner->implementer fanning out
with real file I/O (no "subagents lack file tools" warning); CLI fan-out intact
with skills still loading and tests passing. An all-tools baseline used only
tools within this enumerated set, confirming no CLI capability is restricted.

* Include skill tool in code-testing agent tool allowlists

Address PR review: the explicit `tools:` allowlists omitted the `skill`
tool, but every code-testing-* agent's prompt instructs calling skills
(e.g. `code-testing-extensions` for per-language guidance, `test-gap-analysis`,
`assertion-quality`). Because `tools:` is an override, omitting `skill` can
prevent the agents from loading those skills in environments that gate skill
invocation by the allowlist.

Add `skill` to all eight agents:
- orchestrators (generator, implementer): [agent, skill, read, search, edit, execute]
- workers (researcher/planner/builder/tester/fixer/linter): [skill, read, search, edit, execute]

Re-verified in the Copilot CLI: full fan-out (researcher -> planner ->
implementer/tester), the `code-testing-extensions` skill is invoked, and the
generated tests pass.
 2026-06-30 19:42:39 +02:00
Amaury Levé 35ace775b3 Add dotnet-test-migration plugin and move migration skills there (#808) 
Move the .NET test framework/platform migration skills (migrate-mstest-v1v2-to-v3, migrate-mstest-v3-to-v4, migrate-vstest-to-mtp, migrate-xunit-to-mstest, migrate-xunit-to-xunit-v3) and the test-migration orchestrator agent out of dotnet-test into a new dedicated dotnet-test-migration plugin, along with their evals. Update all marketplace manifests, READMEs, CODEOWNERS, and .vally.yaml accordingly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-23 17:19:32 +02:00
Amaury Levé 803b91f167 Add grade-tests skill for per-test PR-comment quality grading (#727) 
* Add grade-tests skill for per-test quality grading

Adds a new polyglot skill that grades a provided list of test methods on a
letter scale (A-F) with a score band and 1-line notes, suitable for posting
in a PR comment table. Complements existing audit skills (assertion-quality,
test-anti-patterns, test-smell-detection) which produce suite-wide reports.

The skill:
- Takes an explicit list of tests (refuses to grade a whole workspace silently)
- Uses a 3-dimension rubric (Assertion strength, Structure & focus, Anti-pattern hygiene)
- Combines via 0.45/0.30/0.25 weighting with safety caps
- Starts at A and only deducts for observable issues
- Does not flag idiomatic patterns (table-driven sub-tests, bare assert, etc.)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback

- OrderProcessorTests.cs: fix 4 invocations that mixed named arg
  customerId: with subsequent positional args -- invalid C# syntax
  (Copilot review on PR #727)
- README.md: update 'five test-analysis skills' to 'six' in three
  places now that grade-tests is added to the polyglot list, and
  clarify that only the original 5 load from test-analysis-extensions
  while grade-tests has an inline rubric

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* eval: drop 'grade-tests skill' wording from scenario prompts

The skill-validator's evaluate pre-flight check ValidateEvalPrompts
rejects any prompt that names the target skill, because mentioning the
skill biases baseline (skill-not-loaded) runs and inflates measured
lift. The two scenarios that asked the agent to 'use the grade-tests
skill' now just ask it to grade test methods individually -- the agent
is still expected to pick the right skill organically.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix rubric consistency per review feedback

- README: drop incorrect 'inline rubric instead' claim (grade-tests also
  loads test-analysis-extensions for language-specific guidance)
- Assertion sub-grade: literal always-true assertions are now F (was D),
  matching the eval expectation that 'assert True' produces overall F
- Anti-pattern catalog: split always-true literals (now -> F) from
  self-referential assertions (still -> D) — they were conflated
- Combination rule: overall grade is now capped at the worst sub-grade
  (was capped at D), so an F in any one dimension yields overall F

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-05 15:42:36 +00:00
Amaury Levé 7fc6d472c3 Add migrate-xunit-to-mstest skill (#718) 
* Add migrate-xunit-to-mstest skill

Adds a skill to migrate .NET test projects from xUnit.net (v2 or v3) to

MSTest v4. Covers package replacement, attribute/assertion/fixture/lifecycle

translation, ITestOutputHelper -> TestContext, [Trait] -> [TestCategory]/

[TestProperty], and xUnit v3 TestContext.Current.CancellationToken.

Front-loads parallelization handling because xUnit parallelizes test classes

by default while MSTest serializes — the single largest source of post-

migration regressions. Step 11 enumerates three explicit choices and how to

translate CollectionBehavior/MaxParallelThreads/[Collection] settings.

Preserves the existing test platform (VSTest stays VSTest; MTP stays MTP)

by default — bundling a platform migration would violate the

test-migration agent's 'never mix migration steps' rule.

- plugins/dotnet-test/skills/migrate-xunit-to-mstest/SKILL.md (workflow)

- plugins/dotnet-test/skills/migrate-xunit-to-mstest/references/mapping-cheatsheet.md

- tests/dotnet-test/migrate-xunit-to-mstest/eval.yaml (12 scenarios)

- tests/dotnet-test/migrate-xunit-to-mstest/eval.vally.yaml (parity with sibling skill)

- plugins/dotnet-test/README.md (table entry)

- plugins/dotnet-test/agents/test-migration.agent.md (triage routing)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix step cross-references flagged in PR review

- Step 1.5 'Custom DataAttribute' bullet pointed to Step 6 (assertions);

  custom DataAttribute mapping lives in Step 5 (data-driven tests).

- Step 7 lifecycle table 'Constructor' row pointed to Step 8 for

  ITestOutputHelper; ITestOutputHelper conversion is in Step 9 (output).

- Commit-strategy callout said 'after Step 5 (asserts fixed)' and 'after

  Step 9 (fixtures/lifecycle rewritten)'; asserts are in Step 6 and

  fixtures/lifecycle complete after Step 8.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Map [assembly: Xunit.Trait] to [assembly: TestCategory]/[TestProperty]

Both TestCategoryAttribute and TestPropertyAttribute target Assembly,

Class, and Method, so an xUnit [assembly: Trait] does not need to be

demoted to per-class/per-method MSTest attributes — assembly scope is

preserved 1:1.

- references/mapping-cheatsheet.md §8 now maps [assembly: Trait] to

  [assembly: TestCategory] / [assembly: TestProperty] instead of 'Remove'.

- SKILL.md Step 12 promoted to 'Convert' (was 'Remove'); the same mapping

  is documented inline.

- Notes after the Step 4 and §1 trait tables now disclose the assembly

  target so users do not assume class/method-only scope.

Addresses PR feedback from @Evangelink.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address second round of PR review feedback

From @Evangelink:

- [Ignore]/[Timeout] do NOT discover a test on their own — they are

  modifiers. SKILL.md Step 4 + cheatsheet §1 now show [Fact(Skip)] ->

  [TestMethod] + [Ignore(...)] (and same for [Timeout]). Eval scenario 9

  rubric updated to require both attributes.

- TestProperty IS filterable (was incorrectly described as 'metadata that

  does not need filter integration'). Notes after the trait tables in

  both SKILL.md and cheatsheet §1 now show both filter syntaxes.

- For environmental gates (OS-/CI-/arch-conditional), point at MSTest

  3.10+'s condition attributes ([OSCondition], [CICondition],

  [ArchitectureCondition], [NonParallelizableCondition]) as the preferred

  alternative to overloading [TestCategory] or scattering Assert.Inconclusive

  through test bodies. Strengthened in cheatsheet §3.9, SKILL.md Step 6,

  Step 10 SkippableFact row, and the Common Pitfalls table.

- TestDataRow<T> is also a supported MSTest data source (strongly typed

  with per-row DisplayName/Ignore metadata). Added to both SKILL.md Step 5

  and cheatsheet §2 with link to the docs.

- MSTest.Sdk + <UseVSTest>true</UseVSTest> pulls in Microsoft.NET.Test.Sdk

  automatically — the manual PackageReference was incorrect. Removed it

  from both files (Step 2, Step 3 'common mistakes', and Common Pitfalls).

- Added the 'pin MSTest.Sdk in global.json msbuild-sdks' alternative as

  the recommended pattern for multi-project solutions.

- Xunit.Combinatorial -> Combinatorial.MSTest (Youssef1313 community port).

  Updated both SKILL.md Step 10 and cheatsheet §10.

From Copilot reviewer:

- MSTest.Sdk example in SKILL.md Step 2 Option B hardcoded

  <TargetFramework>net9.0</TargetFramework>, contradicting the

  preserve-TFM rule. Replaced with a placeholder + explicit comment.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Improve xUnit-to-MSTest skill effectiveness based on eval feedback

Eval results on PR #718 showed regressions on three rubric items where the

agent did the migration correctly but did not COMMUNICATE the judgement-call

decisions, so the judge could not verify the choice was deliberate:

- ICollectionFixture scope decision (rubric: 'Explicitly explains the scope

  decision instead of silently widening' scored 1/5 because the agent said

  'Let me convert everything' with zero scope discussion). Step 8 now requires

  a templated explanation of (source scope, target scope, what is shared, what

  is serialized, why the rejected alternative was rejected) before applying.

- Parallelization 'MSTest default is serial' messaging (rubric scored 1.7/5

  because the agent only implicitly conveyed it). Step 11 Choice A now demands

  a templated sentence explaining that the [assembly: Parallelize] is REQUIRED

  to match xUnit, otherwise the suite silently regresses to serial.

- Response Guidelines now lists the three judgement-call decisions that MUST

  be communicated up-front: fixture scope, parallelization model, and the

  Throws/ThrowsExactly semantic flip.

Additional fixes from this round:

- Step 2: explicit guidance to default to Option A (MSTest metapackage) when

  the user says 'preserve VSTest', so the safer PackageReference path is

  picked over MSTest.Sdk + UseVSTest.

- Step 4: pulled the '[Ignore]/[Timeout] are modifiers, need [TestMethod]'

  callout out of a table footer into a visible note above the table.

- known-domains.txt: allowlist github.com/Youssef1313/Combinatorial.MSTest

  (referenced from SKILL.md Step 10 and cheatsheet §10 after the recent

  Xunit.Combinatorial -> Combinatorial.MSTest switch).

Validator: green (24 skills, 11 agents).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add concrete TestContext.Current.CancellationToken migration example

The CancellationToken scenario was failing/low-scoring because the skill only
contained a one-line mapping with no concrete example. The agent often missed
the requirement to:

  - Add constructor TestContext injection (even when ITestOutputHelper is absent)
  - Map Assert.False -> Assert.IsFalse alongside the rename
  - NOT fabricate a CancellationTokenSource as a replacement

Add a full xUnit v3 -> MSTest worked example next to the TestContext.Current
mapping (Step 9), and reinforce the CancellationTokenSource warning in the
cheatsheet. The example covers all 3 rubric points for the scenario.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address third round of PR review feedback

- Remove fictional MSTest condition attributes [ArchitectureCondition] and
  [NonParallelizableCondition] from SKILL.md and cheatsheet (only [OSCondition]
  and [CICondition] exist in MSTest 3.10+; for non-parallel intent use
  [DoNotParallelize], for architecture gating fall back to runtime
  Assert.Inconclusive). (Copilot)
- TestPropertyAttribute targets only [Class|Method] (no AttributeTargets.Assembly),
  so [assembly: TestProperty(...)] does not compile. Update the trait notes in
  SKILL.md Step 4, cheatsheet section 1, Step 12, and cheatsheet section 8 to
  collapse a non-category xUnit [assembly: Trait] to [assembly: TestCategory] or
  push it down to per-class [TestProperty]. (@Evangelink)
- Step 1 platform detection: stop inlining a buggy VSTest-vs-MTP matrix;
  delegate to the platform-detection skill. Note explicitly that
  <UseMicrosoftTestingPlatformRunner> only affects 'dotnet run' and is not a
  reliable runner signal. (@Youssef1313)
- Step 2 Option A: add MTP code-coverage caveat -- Microsoft.NET.Test.Sdk
  pulls VSTest's Microsoft.CodeCoverage transitively, which can interfere with
  MTP's collector. Prefer Option B (MSTest.Sdk without UseVSTest) for MTP
  projects. (@Youssef1313)
- Step 3: remove fictional MSBuild properties (<CaptureConsoleOutput>,
  <UseRoslynCompilers>); restate the bullet about xunit.runner.json -> Step 11
  port instead. Restate the <UseMicrosoftTestingPlatformRunner> bullet to make
  clear it is not a runner switch. (@Youssef1313)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address Copilot review round 4

- mapping-cheatsheet.md (Assert.Contains predicate): the suggested fallback
  Assert.Contains(collection.First(x => predicate), collection) is buggy --
  First() throws InvalidOperationException on no-match (changing the failure
  mode), and feeding the result back into Assert.Contains is circular. Keep
  only the correct translation: Assert.IsTrue(collection.Any(x => predicate)).
- SKILL.md Step 4 / cheatsheet table: stop instructing 'mark every class
  sealed'. xUnit projects commonly use base/derived test classes (shared
  setup, generic base fixtures); mechanically sealing them would break
  compilation. Sealing is now framed as an optional follow-up handled by
  writing-mstest-tests, not part of the mechanical migration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-06-05 11:40:40 +02:00
Amaury Levé 680292e04c Make dotnet-test analysis skills and auditor agent polyglot (#707) 
* Make dotnet-test analysis skills and auditor agent polyglot

Five test analysis skills and the test-quality-auditor agent in
plugins/dotnet-test previously activated only on .NET MSTest/xUnit/NUnit/
TUnit cues, even though the underlying techniques (assertion diversity,
anti-pattern detection, pseudo-mutation gap analysis, academic smell
catalog, trait tagging) apply equally to other test frameworks.

This change makes them polyglot across .NET, Python (pytest/unittest),
TypeScript/JavaScript (Jest/Vitest/Mocha/node:test), Java (JUnit/TestNG),
Go, Ruby (RSpec/Minitest), Rust, Swift (XCTest/Swift Testing), Kotlin
(JUnit/Kotest), PowerShell (Pester), and C++ (GoogleTest/Catch2/doctest).

Changes
-------

* New reference skill `test-analysis-extensions` with per-language
  extension files (dotnet, python, typescript, java, go, ruby, rust,
  swift, kotlin, powershell, cpp). Each documents test markers,
  assertion APIs, sleep / time / random APIs, skip annotations,
  setup/teardown, mystery-guest indicators, integration markers, and
  tag-support capability for the corresponding framework.
* `assertion-quality`, `test-anti-patterns`, `test-gap-analysis`,
  `test-smell-detection`, `test-tagging` SKILL.md updated: frontmatter
  now lists all polyglot frameworks (keeping .NET trigger words for
  backward compat); bodies reference `test-analysis-extensions` and
  add per-language calibration (e.g., Go/Rust table-driven loops are
  idiomatic and not Conditional Test Logic; pytest bare `assert` is
  canonical; missing-await on async assertions is a new Critical smell).
* `test-quality-auditor` agent now runs language detection, gates
  .NET-only pipeline steps (coverage-analysis, CRAP, detect-static-
  dependencies, testability migration, experimental dotnet skills), and
  recommends native tooling for non-.NET projects via an explicit
  Capability Matrix.
* README updated to split sections by polyglot vs .NET-only.
* New polyglot eval scenarios added (Python/pytest, TS/Jest,
  Java/JUnit) alongside existing C# scenarios in
  `test-anti-patterns`, `assertion-quality`,
  `test-smell-detection`.

The .NET-only skills (writing-mstest-tests, migrate-*, coverage-analysis,
crap-score, detect-static-dependencies, run-tests, filter-syntax,
platform-detection, dotnet-test-frameworks, mtp-hot-reload, exp-*) are
unchanged and continue to activate only for .NET work.

Validated with skill-validator `check --plugin ./plugins/dotnet-test`.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR #707 review feedback

* typescript.md: replace xpect.fail() recommendation (Jest has no
  such API) with 	hrow new Error(...) / explicit failing assertion.
* test-tagging SKILL.md description: clarify that unittest has no
  canonical tag syntax (pytest markers are the auto-edit mechanism;
  unittest is report-only). Switch doctest reference to "decorators"
  rather than implying a Catch2-style [tags] mechanism exists.
* test-tagging SKILL.md auto-edit list and table: use
  `* doctest::test_suite("tag")` decorator chain syntax for doctest
  in both the bullet list and the per-framework table, matching the
  example already shown.
* cpp.md extension: spell out the Catch2 vs doctest tagging
  distinction in the capability matrix (Catch2 [tag], doctest
  * doctest::test_suite("tag") decorator).

Validated with `dotnet run --project eng/skill-validator/src -- check
--plugin ./plugins/dotnet-test` (23 skills + 11 agents).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
 2026-06-01 16:46:26 +02:00
Amaury Levé 5856011d8f Add README.md for dotnet-test plugin (#636) 2026-05-11 15:54:27 +00:00