* code-testing-agent: add polyglot pipeline examples for Python/TypeScript/Go/Java
The code-testing-agent skill family is polyglot in description but in
practice biased toward .NET because dotnet-examples.md was the only
filled-in pipeline walkthrough. The four sub-agents that participate in
the Research-Plan-Implement pipeline (researcher, planner, implementer,
generator) all pointed to dotnet-examples.md whenever they suggested a
concrete example, which made it harder for the agent to produce
idiomatic non-.NET tests (e.g. in msbench top5-* benchmarks for
Python/Flask, TypeScript/Express).
This change
* adds four new example files mirroring dotnet-examples.md format
(source → research → plan → generated test → fix cycle → final report):
- python-examples.md (pytest, unittest.mock, Mock(spec=...), parametrize)
- typescript-examples.md (Vitest with notes for Jest; it.each,
async tests, fake timers, ESM/CJS fix cycle)
- go-examples.md (standard testing package, table-driven subtests,
hand-written fake repository, injected clock)
- java-examples.md (JUnit 5 + Mockito on Maven, @ParameterizedTest +
@CsvSource, Clock.fixed, Surefire fix cycles)
* updates code-testing-extensions/SKILL.md TOC to list the new files
and clarifies usage instructions to read the matching <language>-
examples.md alongside the base extension
* makes the "Concrete example" pointers in code-testing-generator,
code-testing-implementer, code-testing-planner and
code-testing-researcher agents language-agnostic (list all available
example files instead of hard-coding dotnet-examples.md)
* expands code-testing-researcher project-structure detection list to
cover more Python (tox.ini, noxfile.py, requirements*.txt, uv.lock,
poetry.lock, pdm.lock), JS/TS (.mts/.cts/.jsx, vitest.config.*,
jest.config.*), C++ (CMakeLists.txt, BUILD.bazel, meson.build),
Java/Kotlin (pom.xml, build.gradle[.kts], wrappers), and other
ecosystem files; expands the Identify-Language section accordingly
* extends the "Language-Specific Examples" section in
code-testing-agent/SKILL.md to summarise each example file
Validated with: skill-validator check --plugin ./plugins/dotnet-test
(23 skills, 11 agents — all checks passed) and markdownlint-cli2.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #708
- go-examples.md: replace the hand-rolled `contains`/`stringIndex`
helpers with `strings.Contains` from the standard library. The hand-rolled
`contains` had subtly wrong semantics — `contains(""abc"", """")` returned
`false` while `strings.Contains` returns `true` — and the helpers are
unnecessary complexity for a code-generation example.
- go-examples.md: in the `go test -run` "wrong selection regex" sample fix
cycle, quote the test name and use `single_item` (matching the underscore
that the surrounding diagnosis text refers to) instead of the unquoted
`single item` which the shell would parse as two separate CLI arguments.
- java-examples.md: the source-tree file list described `Invoice.java` as a
`record` but the `InvoiceService.markAsPaid` example mutates the invoice
via `setStatus(...)` and `setPaidDate(...)` — records are immutable, so
the description was internally inconsistent. Re-describe it as a mutable
POJO with explicit mutators to match the service code.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Make dotnet-test analysis skills and auditor agent polyglot
Five test analysis skills and the test-quality-auditor agent in
plugins/dotnet-test previously activated only on .NET MSTest/xUnit/NUnit/
TUnit cues, even though the underlying techniques (assertion diversity,
anti-pattern detection, pseudo-mutation gap analysis, academic smell
catalog, trait tagging) apply equally to other test frameworks.
This change makes them polyglot across .NET, Python (pytest/unittest),
TypeScript/JavaScript (Jest/Vitest/Mocha/node:test), Java (JUnit/TestNG),
Go, Ruby (RSpec/Minitest), Rust, Swift (XCTest/Swift Testing), Kotlin
(JUnit/Kotest), PowerShell (Pester), and C++ (GoogleTest/Catch2/doctest).
Changes
-------
* New reference skill `test-analysis-extensions` with per-language
extension files (dotnet, python, typescript, java, go, ruby, rust,
swift, kotlin, powershell, cpp). Each documents test markers,
assertion APIs, sleep / time / random APIs, skip annotations,
setup/teardown, mystery-guest indicators, integration markers, and
tag-support capability for the corresponding framework.
* `assertion-quality`, `test-anti-patterns`, `test-gap-analysis`,
`test-smell-detection`, `test-tagging` SKILL.md updated: frontmatter
now lists all polyglot frameworks (keeping .NET trigger words for
backward compat); bodies reference `test-analysis-extensions` and
add per-language calibration (e.g., Go/Rust table-driven loops are
idiomatic and not Conditional Test Logic; pytest bare `assert` is
canonical; missing-await on async assertions is a new Critical smell).
* `test-quality-auditor` agent now runs language detection, gates
.NET-only pipeline steps (coverage-analysis, CRAP, detect-static-
dependencies, testability migration, experimental dotnet skills), and
recommends native tooling for non-.NET projects via an explicit
Capability Matrix.
* README updated to split sections by polyglot vs .NET-only.
* New polyglot eval scenarios added (Python/pytest, TS/Jest,
Java/JUnit) alongside existing C# scenarios in
`test-anti-patterns`, `assertion-quality`,
`test-smell-detection`.
The .NET-only skills (writing-mstest-tests, migrate-*, coverage-analysis,
crap-score, detect-static-dependencies, run-tests, filter-syntax,
platform-detection, dotnet-test-frameworks, mtp-hot-reload, exp-*) are
unchanged and continue to activate only for .NET work.
Validated with skill-validator `check --plugin ./plugins/dotnet-test`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR #707 review feedback
* typescript.md: replace xpect.fail() recommendation (Jest has no
such API) with hrow new Error(...) / explicit failing assertion.
* test-tagging SKILL.md description: clarify that unittest has no
canonical tag syntax (pytest markers are the auto-edit mechanism;
unittest is report-only). Switch doctest reference to "decorators"
rather than implying a Catch2-style [tags] mechanism exists.
* test-tagging SKILL.md auto-edit list and table: use
`* doctest::test_suite("tag")` decorator chain syntax for doctest
in both the bullet list and the per-framework table, matching the
example already shown.
* cpp.md extension: spell out the Catch2 vs doctest tagging
distinction in the capability matrix (Catch2 [tag], doctest
* doctest::test_suite("tag") decorator).
Validated with `dotnet run --project eng/skill-validator/src -- check
--plugin ./plugins/dotnet-test` (23 skills + 11 agents).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
The code-testing-* agent family generates tests for 12 languages (not just
.NET). Until now, no instruction told the implementer to keep its changes
additive: it would routinely modify existing test files non-additively
(deleting/reformatting lines) or edit non-test production code to make
something easier to test. Both behaviors cause the change to be rejected
by:
- the msbench 'simple' test verifier (any 'dels > 0' on a test file, or
any change to a non-test file, sets reward=0)
- most real-world test-quality gates, code-review policies, and CI
Add explicit cross-language rules in the implementer agent prompt:
1. Existing test files are append-only (no reformat/reorder/remove).
2. Do not modify non-test source files; surface untestable seams as
follow-ups for the testability-migration agent.
3. Prefer new test files over edits to existing ones when equivalent.
4. Build-system manifests may be edited only for project/dependency
registration.
Rule 6 in the Rules section now points at the Step 4 detail so the
implementer reads it on every phase.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The four agents (code-testing-generator, test-migration, testability-migration,
test-quality-auditor) were the only dotnet-test agents declaring a 'tools:'
list. Per Jan's suggestion on #660, drop the field so they inherit the runtime's
default tool surface (matching the convention used by every other agent in the
repo).
Follow-up to #660.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The skill-validator only recognizes 'bash' and 'powershell' as built-in shell tools, not 'terminal'. Update the four affected agents in the dotnet-test plugin to use both for cross-platform support.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix coverage-analysis activation for plateau diagnosis prompts
coverage-analysis SKILL.md:
- Trim verbose implementation details (provider detection,
ReportGenerator) that consumed description budget without
aiding skill activation
- Add explicit USE FOR keywords: coverage stuck, coverage plateau,
can't increase coverage, what's blocking coverage
code-testing-agent SKILL.md:
- Add 'diagnosing coverage plateaus or CRAP score computation
(use coverage-analysis)' to DO NOT USE FOR boundary to prevent
test-generation skill from intercepting diagnostic prompts
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Strengthen code-testing-agent activation; harden coverage-analysis isolated mode
Address eval regressions reported on PR #647 (run 25813728646):
1. code-testing-agent: `Generate tests for ContosoUniversity ASP.NET Core MVC app`
was NOT ACTIVATED in plugin mode (detectedSkills=[], skillEventCount=0,
invokedAgents=[]). The model bypassed the skill system entirely.
- SKILL.md description: restructure to use the proven `Use when user says ...`
pattern with quoted trigger phrases (matching the run-tests skill that
consistently activates), make the link to the code-testing-generator
sub-agent explicit, and tighten DO NOT USE FOR clauses.
- eval prompt (eval.yaml + eval.vally.yaml): make the request
pipeline-shaped (`project-wide, multi-file test generation task`,
`scaffold a new test project`) so the model recognizes it as multi-step
work that benefits from the orchestrated pipeline. Explicitly request
coverlet.collector + a Cobertura XML run so rubric criterion 1
(`high line coverage as reported by the Cobertura XML in TestResults/`)
becomes achievable without overfitting.
2. code-testing-tester agent + code-testing-extensions/dotnet.md: open a
scoped exception to the `skip coverage tools` rule. Default behavior
stays the same, but when the user/harness explicitly asks for a
Cobertura/XML coverage artifact, the agent may add coverlet.collector
to the generated test csproj so the harness's coverage command produces
output. The agent still does not run the coverage command itself.
3. coverage-analysis SKILL.md: add a `User-visible output is mandatory`
guard at the top of the Workflow section. The latest eval showed isolated
mode producing literally `(no output)` in 2 of 3 scenarios — the agent
ran Compute-CrapScores.ps1 / Extract-MethodCoverage.ps1 / ReportGenerator
in parallel, then the session ended without ever surfacing findings.
The guard tells the agent to always return a partial summary instead of
ending silent, and to deprioritize ReportGenerator HTML when budget is
tight. (Plugin-mode quality is already strong: 4.3 / 4.3 / 5.0 — no
regression risk there.)
Aggregate dotnet-test plugin description size: 14,925 chars (limit 15,000).
skill-validator check passes (22 skills, 11 agents, 1 plugin); markdownlint
passes for all 4 modified files.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Restore MSTest modernization exclusion in code-testing-agent description
* Fix isolated-mode coverage-analysis: emit summary before optional ReportGenerator
The previous workflow encouraged the agent to run `dotnet tool install` for
ReportGenerator in parallel with the CRAP scoring scripts (Phase 2 "Steps
3 and 4 in parallel" + Phase 3 "Steps 5 and 6 in parallel"). In isolated
mode that pattern reliably crashed the session with "Failed to persist
session events: timeout while waiting for mutex to become available"
right after the scripts returned valid data, so the agent never produced
the user-facing summary.
Restructure the workflow into 5 phases:
- Phase 1 (Setup) - unchanged
- Phase 2 (Test execution) - skip when Cobertura XML already exists
- Phase 3 (Analysis) - run only the two PowerShell scripts, no RG
- Phase 4 (User-facing summary) - MANDATORY, must be the next assistant
response after Phase 3, before any RG work; also save
coverage-analysis.md as a secondary follow-up
- Phase 5 (ReportGenerator HTML/CSV) - strictly optional, post-summary,
skipped by default for existing-Cobertura and plateau-diagnosis paths
Also update references/output-format.md so the Reports section marks RG
artifacts as "Not generated (optional - request HTML reports to enable)"
when Phase 5 has not run, and update references/guidelines.md so the
"show and open the markdown report" rule explicitly defers to the
user-facing assistant response.
Targets the isolated-mode regressions in PR #647 eval:
- Project-wide coverage with existing Cobertura: 1.0/5 -> expected 3+
- Coverage plateau diagnosis: 1.0/5 -> expected 3+
- Run coverage from scratch: 2.3/5 -> expected steady or up
Verified: skill-validator check --plugin ./plugins/dotnet-test passes;
markdownlint-cli2 clean on all 3 modified files.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR #647 review comments
1. Prerequisites: distinguish the from-scratch path (needs NuGet for the
coverage-provider package install + optional internet for ReportGenerator)
from the existing-Cobertura path (needs neither).
2. Add Step 2c "Discover or accept existing Cobertura XML" so the
existing-data path actually has $coberturaFiles populated before
Phase 3, instead of relying on Phase 2's discovery (which it
skips). Also clarify Step 2's destructive Remove-Item only manages
the skill-owned coverage-analysis/ subdirectory.
3. references/output-format.md: replace the unconditional
"Reports saved to: <coverageDir>/reports/" line with one that
always points at <coverageDir>/ (markdown summary + raw Cobertura)
and only mentions reports/ if Phase 5 ran.
4. Have Compute-CrapScores.ps1 emit OVERALL_LINE_COVERAGE and
OVERALL_BRANCH_COVERAGE from the Cobertura root attributes, and
update Phase 4 to read those values directly from the script's
output. The Phase 4 mandatory-summary rule no longer requires a
separate XML parse before composing the response.
Verified: skill-validator check --plugin ./plugins/dotnet-test passes;
markdownlint-cli2 clean on all modified files; Compute-CrapScores.ps1
smoke-tested on a synthetic Cobertura XML (emits OVERALL_LINE_COVERAGE:75,
OVERALL_BRANCH_COVERAGE:50 alongside HOTSPOTS).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Apply suggestion from @Evangelink
* Address unresolved coverage-analysis and eval review comments
* Refine follow-up review feedback from validation
* Tighten coverage aggregation fallback notes and counters
* Clarify pre-response save instruction wording
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* CTA: invocation-only baseline (dispatch mechanics, no content rules)
Experiment branch to isolate the impact of "invoke prompted subagents
more often" from the impact of "make those subagents do richer work."
Same baseline as dev/ykovalova/cta-prompt-tuning (main = 66628b6), but
strips out every content/quality rule and keeps only the dispatch
plumbing.
Comparison branch: dev/ykovalova/cta-prompt-tuning (HEAD: bd530be)
which contains both the dispatch mechanics AND content/quality rules.
Files modified (3 vs 5 in cta-prompt-tuning):
- code-testing-generator.agent.md +110 lines
- code-testing-implementer.agent.md +11 lines
- code-testing-fixer.agent.md +1 line
- code-testing-researcher.agent.md UNTOUCHED (baseline)
- code-testing-planner.agent.md UNTOUCHED (baseline)
KEPT (invocation / dispatch mechanics):
Generator:
- Rule 1: every task() call MUST use agent_type "dotnet-test:code-testing-..."
(without this, calls dispatch generic built-ins and never reach the
named CTA agents)
- Rule 2: routing table -- which named agent for which job
- Rule 3: prefer one named-agent dispatch over many tool calls
- Rule 4: orchestrator MUST NOT edit/create test files itself
(forces implementer dispatch)
- Rule 5: orchestrator MUST NOT run builds/tests via terminal
(forces builder/tester dispatch)
- Rule 6: every run MUST dispatch the planner (no exceptions for "small"
scope; Direct still goes through planner)
- Rule 7: every build/test failure MUST dispatch the fixer
- Step 1b: mandatory initial researcher dispatch (every strategy)
- Direct strategy rewritten: dispatches planner -> implementer -> builder
-> tester -> fixer -> linter (was "Skip Steps 3-5, write tests inline")
- All Step 3/4/5/6/7/8/9 dispatches converted from runSubagent({agent:...})
to task({ agent_type: "dotnet-test:code-testing-...", name:..., prompt:...})
- Step 9 validator dispatch (forces builder dispatch for cleanup)
- Steps 6/7 mandatory builder/tester dispatch wrapper
Implementer:
- Section 5: "you MUST dispatch fixer for build errors" + no-inline-edit
block (forces fixer dispatch on build failures)
- Section 6: "you MUST dispatch fixer for test failures" + no-inline-edit
block (forces fixer dispatch on test failures)
- Section 7: "Format Code (mandatory if a lint command exists)"
(was "Optional"; mandatory firing of linter)
- Rule 6: never declare SUCCESS while build/tests fail (gates SUCCESS on
fixer dispatch)
- Rule 7: no inline test-file edits between failed dispatch and fixer
Fixer:
- Frontmatter description widened to advertise handling of failing tests
(without this, the orchestrator's routing logic does not select the
fixer for test failures, so even Rule 7's mandate produces no firing
-- this is the change that took fixer firing from 0.00/inst to 0.39/inst
in earlier iterations)
DROPPED (content / quality rules -- in cta-prompt-tuning, NOT here):
Generator:
- Test-strength rules embedded in implementer dispatch prompt
- Test-design rules embedded in implementer dispatch prompt (OFAT,
mutation self-check, never mock subject under test)
- File-location rules embedded in implementer dispatch prompt
- TARGET ENTITIES / PHASE CHECKLIST / TEST TRACEABILITY blocks in
implementer dispatch prompt
- CHECKLIST format spec in planner dispatch prompt
- Step 9 validator's detailed cleanup classification
Implementer:
- Section 4b "Verify CHECKLIST coverage" pre-completion check
- Section 8 "CHECKLIST COVERAGE" report block
- "Honor the CHECKLIST" rule
Fixer:
- "Process -- Failing Tests" section (5-step diagnosis flow)
- All anti-weakening / anti-skipping rules
- "Re-derive expected from production source" guidance
Planner:
- CHECKLIST format ("one item per TARGET BEHAVIOR, Source/Variants/
Expected mandatory")
- "Test name from research.md conventions" rule
- "At least 2 phases" rule
Researcher:
- Section 8 "Extract Local Test Naming & Style Conventions"
- TARGET ENTITIES / TARGET BEHAVIORS / TEST INFRASTRUCTURE structure
in research.md
- Test naming pattern extraction
WHAT THE SUBAGENTS WILL ACTUALLY DO:
The researcher / planner / implementer / fixer all operate at baseline
behavior -- they receive the same prompts they receive in the upstream
"vanilla" runs. The only difference vs vanilla is that the orchestrator
ACTUALLY DISPATCHES THEM (where vanilla often inlines the work or skips
sub-agent dispatch entirely).
EXPECTED COMPARISON:
If quality on this branch is similar to or higher than dev/ykovalova/
cta-prompt-tuning (bd530be), then "more dispatches" is the dominant
quality lever and the content/quality rules in cta-prompt-tuning are
adding marginal or noise-level value.
If quality on this branch is materially lower than cta-prompt-tuning,
then the content/quality rules are doing the heavy lifting and the
dispatch mechanics alone are insufficient.
If quality on this branch matches or exceeds vanilla but trails
cta-prompt-tuning, then the dispatch mechanics provide a baseline lift
and the content rules add an incremental quality layer on top.
Rubber-duck check passed (validated dispatch-vs-content classification;
fixer frontmatter is routing metadata not a runtime gate; surviving
dispatch prompts contain no dangling references to removed CHECKLIST /
TARGET ENTITIES / TEST STRENGTH / naming-convention concepts).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* CTA: extract unit-under-test + behaviors quality cue (no caps)
* Address PR #646 review feedback
- Step 1b prompt: explicitly request unit-under-test (file:line) and behaviors
so the verification gate rarely needs re-dispatch.
- Step 3: rename to 'Deep Research Phase', mark skipped for Direct strategy,
and switch from overwriting research.md to extending it (no double-research).
- Step references: update '6-9' -> '6-10' and 'Step 9' -> 'Steps 9-10' in the
strategy table and the All-strategies-MUST line, since reporting is Step 10.
- Step 6 builder prompt: drop '*.sln' glob (could expand to multiple args);
use 'dotnet build --no-incremental' (auto-discovers .sln) per dotnet.md.
- Step 9: stop overloading the builder agent; perform diff/cleanup directly
in the orchestrator (Rule 5 forbids inline build/test, not git/fs hygiene).
- Fixer agent: update mission text to cover failing tests and assertion
correction (front-matter description already mentioned this; body now
matches), with explicit no-Ignore/no-Skip/no-production-rewrite guardrails.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Yuliia Kovalova <ykovalova@example.com>
Move assertion-quality, test-gap-analysis, test-smell-detection, and
test-tagging from dotnet-experimental to dotnet-test. These skills have
demonstrated sufficient eval coverage, unique value beyond baseline model
capabilities, and are already integrated into the test-quality-auditor
agent pipeline.
Changes:
- Move skills and tests, dropping the exp- prefix
- Update all cross-references in skill descriptions and routing tables
- Update test-quality-auditor agent and eval scenarios
- Update CODEOWNERS for the moved paths
- Delete duplicate exp-dotnet-test-frameworks (identical to dotnet-test-frameworks)
- Fix stale exp-dotnet-test-frameworks reference in exp-test-maintainability
Skills held back for further iteration:
- exp-mock-usage-analysis: baseline model scores 5/5 on key scenarios
- exp-test-maintainability: quality gains offset by token overhead
* Improve test quality for benchmark performance
Based on SWE Atlas benchmark analysis (48.8% vanilla vs 19-34% CTA):
1. Default to Direct strategy for single-task requests — reduces
multi-agent overhead that costs tokens without improving results
2. Run tests immediately in Direct strategy — catches assertion
errors early instead of accumulating failures
3. Read source thoroughly before writing tests — trace actual logic
and return values, not just function signatures
4. Quality over Quantity guidelines — cover stated requirements first,
fewer focused tests beat many shallow ones
5. Verify tests are implementation-specific — tests that pass with
an empty function body aren't testing anything useful
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Combine redundant implementation-specificity bullets into one
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add end-to-end input/output examples to anchor expected behavior for the
LLM, addressing the Example Quality (2/5) review feedback.
Changes:
- New extensions/dotnet-examples.md with .NET-specific examples: sample
source code, research output, plan output, generated test file, fix
cycle walkthroughs, and final report
- SKILL.md: replace one-liner examples with strategy selection table,
pipeline walkthrough, and pointer to language-specific extensions
- Generator agent: add strategy decision examples and sample final report
- Researcher/Planner/Implementer agents: add references to extension
examples for concrete output shapes
Architecture keeps core agents language-agnostic — all language-specific
examples live in the extensions/ folder.
Harden the test generator agent to ensure final build validation, test
validation, and git commit always run — even when using the Direct
strategy for small requests.
Changes:
- Step 1: Mandate reading language-specific extensions (e.g., dotnet.md)
before writing any code. The extension contains critical project
registration (dotnet sln add) and build validation guidance.
- Step 2: Direct strategy now skips only Steps 3-5 (research, plan,
implement sub-agents), NOT Steps 6-9. Added explicit note that
Steps 6-9 are mandatory for ALL strategies.
- Step 6: Clarify full-solution build must use no --framework flag to
catch multi-target framework issues (e.g., net472 errors).
- Step 7: Require fresh build for final test validation (no --no-build).
- New Step 9: Explicit git commit with verification that the diff is
non-empty.
- New rules 9-11: Read extensions first, always validate+commit,
preserve existing tests.
Motivated by MSBench investigation showing:
- Agent used Direct strategy and skipped final validation entirely
- Agent never read extensions/dotnet.md (missing dotnet sln add)
- Agent forgot to git commit (300 tests created, Changes +0 -0)
- Agent deleted pre-existing tests while adding new ones
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The test-anti-patterns skill was not reliably activated in plugin mode
because sibling descriptions overlapped:
- test-quality-auditor agent: narrowed to multi-skill audit pipelines
only, explicitly excluding single-file reviews
- writing-mstest-tests skill: removed 'review' and 'anti-patterns' from
description, added explicit exclusion for test quality auditing
- test-anti-patterns skill: expanded trigger phrases to better match
review prompts (flaky tests, duplication, tests that pass without
verifying)
- Add 'Use when:' trigger phrases to all 7 sub-agent descriptions
so parent agents can reliably discover and delegate to them
- Add .testagent/ cleanup rule to generator agent to prevent
ephemeral pipeline state from being committed
* Add test-quality-auditor hub agent for .NET test suite analysis
Introduces a new test-quality-auditor agent that triages test quality
concerns to specialized analysis skills: assertion quality, test smells,
anti-patterns, mock usage, test gaps, maintainability, coverage risk,
and test tagging. Includes a comprehensive audit pipeline that chains
multiple skills for broad health checks.
* Address Copilot review: clarify diagnostic role and coverage routing
- Reword intro to acknowledge file-modifying workflows (test tagging)
are only used with explicit user confirmation
- Clarify coverage-analysis vs crap-score: prefer coverage-analysis for
project-wide analysis, reserve crap-score for targeted method/class scope
* Add test-migration hub agent for .NET test framework and platform migrations
Introduces a new test-migration agent that auto-detects the current test
framework, version, and platform, then routes to the appropriate migration
skill (MSTest v1/v2→v3, v3→v4, xUnit v2→v3, VSTest→MTP). Handles
multi-step migrations with commit boundaries between steps.
* Align framework detection with platform-detection skill
Address Copilot review: MSTest requires metapackage, Sdk, or both
TestFramework+TestAdapter (not either alone). Add missing xUnit MTP
package variants.
* Add 'dotnet sln add' step to code-testing-implementer
When a new test project is created, register it with the solution
file using 'dotnet sln add' so that 'dotnet test <solution>'
discovers the tests.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: improve dotnet sln add guidance
- Use solution identified in research/plan rather than searching
- Account for .slnf solution filter scenarios
- Prefer 'dotnet test --solution' over 'dotnet test <solution>'
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move dotnet sln add guidance to dotnet.md extension
The implementer is polyglot, so 'dotnet sln add' doesn't belong
there. Replace with a generic 'register with build system' step
that defers to extensions/, and add the .NET-specific detail
(sln/slnx/slnf handling) to extensions/dotnet.md.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: fix solution registration guidance in dotnet.md
- Include .slnf in 'don't substitute' warning alongside .sln/.slnx
- Qualify '--solution' flag as SDK 10+/MTP-only; fall back to
positional form for older SDKs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add testability migration skills and agent for dotnet-test
Introduces three skills and one orchestrator agent to help developers
incrementally replace hard-to-test static dependencies with injectable
abstractions:
- detect-static-dependencies: scans C# code for DateTime.Now, File.*,
Environment.*, HttpClient, Console.*, Process.* and ranks by frequency
- generate-testability-wrappers: generates wrapper interfaces or guides
adoption of built-in abstractions (TimeProvider, IHttpClientFactory,
System.IO.Abstractions)
- migrate-static-to-wrapper: mechanical codemod-style bulk replacement of
static call sites with wrapper calls, scoped per project/namespace
- testability-migration agent: orchestrates the Detect-Generate-Migrate
pipeline end-to-end
Includes eval.yaml tests and fixture projects for all three skills and
the agent.
* Fix activation, token overhead, and eval timeouts
- Expand skill descriptions with more trigger phrases for
migrate-static-to-wrapper and generate-testability-wrappers
to fix NOT ACTIVATED issues
- Collapse verbose replacement mapping tables and constructor
examples to reduce token overhead (~30% fewer tokens)
- Increase generate-testability-wrappers eval timeouts from
120s to 240s to prevent timeout-impacted scoring
- Soften detect-static-dependencies scenario 2 rubric to
reduce run-to-run variance in plugin mode
- Increase migrate-static-to-wrapper non-activation timeout
Amaury Levé