* Add VS Code subagent metadata to dotnet-test code-testing agents
Enable the code-testing-* Research-Plan-Implement pipeline to fan out as
subagents in VS Code while keeping the GitHub Copilot CLI working.
Frontmatter (VS Code "coordinator/worker" pattern; portable tool aliases that
map in both VS Code and the CLI):
- code-testing-generator: tools: [agent, read, search, edit, execute] + an
agents: list (researcher, planner, implementer, builder, tester, fixer,
linter); softened the three runSubagent({ agent, prompt }) blocks to
tool-agnostic delegation prose.
- code-testing-implementer: tools: [agent, read, search, edit, execute] +
agents: (builder, tester, fixer, linter).
- Leaf agents (researcher/planner/builder/tester/fixer/linter):
tools: [read, search, edit, execute].
Why explicit tools (not ["*"]): VS Code has no all-tools wildcard for ools:
and a subagent's ools: overrides its inherited set, so ["*"] matched nothing
and stripped subagents of file tools (they ran without read/edit/search). The
CLI treats ["*"] as all-tools, so this was VS-Code-specific. The portable
aliases agent/read/search/edit/execute map to real tools in both environments;
agents: is ignored by the CLI.
README: document the VS Code chat.subagents.allowInvocationsFromSubagents
setting (off by default) needed for the nested implementer->builder/tester/
fixer/linter layer to fan out on large scopes; the CLI has no such gate.
Validated end to end: VS Code shows researcher->planner->implementer fanning out
with real file I/O (no "subagents lack file tools" warning); CLI fan-out intact
with skills still loading and tests passing. An all-tools baseline used only
tools within this enumerated set, confirming no CLI capability is restricted.
* Include skill tool in code-testing agent tool allowlists
Address PR review: the explicit `tools:` allowlists omitted the `skill`
tool, but every code-testing-* agent's prompt instructs calling skills
(e.g. `code-testing-extensions` for per-language guidance, `test-gap-analysis`,
`assertion-quality`). Because `tools:` is an override, omitting `skill` can
prevent the agents from loading those skills in environments that gate skill
invocation by the allowlist.
Add `skill` to all eight agents:
- orchestrators (generator, implementer): [agent, skill, read, search, edit, execute]
- workers (researcher/planner/builder/tester/fixer/linter): [skill, read, search, edit, execute]
Re-verified in the Copilot CLI: full fan-out (researcher -> planner ->
implementer/tester), the `code-testing-extensions` skill is invoked, and the
generated tests pass.
The code-testing-generator/implementer agents could treat an unusual or
scaffolded workspace (e.g. a gutted repo with an injected synthetic module)
as corruption and "repair" it with git checkout/restore/reset/clean or rm,
restoring deleted tracked files and testing the wrong code.
- Replace generator Rule 5 ("Clean git first - stash changes") with an
explicit "Treat the workspace as delivered" rule, and add a "Never mutate
version control" rule. Output must be purely additive test files.
- Add a no-revert/no-clean invariant to the implementer's edit boundaries.
- Add a 'workspace integrity' eval to the code-testing-agent suite
(eval.yaml + eval.vally.yaml). The fixture looks gutted: a metricsd project
whose real core/io modules are committed at HEAD but deleted from the
working tree, leaving only a synthetic 'synthstr' decoy. A git restore would
resurrect the deleted sentinel files; graders fail if they reappear and
require passing pytest tests for the module as delivered.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds a new `Write Tests That Pin Down Behavior'' section to
`unit-test-generation.prompt.md'' covering five universal
test-craft principles:
1. Mutation thinking - each assertion would fail under a plausible bug
2. Property intersections - test combinations, not only coordinate axes
3. Behavior radius - assert on at least one secondary observable
4. Fixture realism - never set the parameter under test to a degenerate value
5. Quick self-review before declaring a test method done
Mirrors the same depth requirements in
`code-testing-implementer.agent.md'' Step 4 as a cross-language
invariant block alongside the existing `Edit boundaries'' rules.
Extends the existing `code-testing-agent'' eval rubrics (yaml and
vally) with one or two depth-oriented bullets per scenario:
* ContosoUniversity: minimal IsNotNull-only assertions + secondary
observable check on controller actions
* python-flask-tasks: minimal `is not None''-only assertions + at
least one combined-property TaskService validation test
* typescript-vitest-cart: minimal `toBeDefined''/`toBeTruthy''-only
assertions + at least one intersection test (discount + tax +
shipping together)
Rationale and prior-art references in PR description.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* code-testing-implementer: add mandatory harness-discovery check
Generic CI/benchmark verifiers run the framework's default discovery from the repo root. Tests that pass via a scoped command (e.g. 'dotnet test MyProject.Tests.csproj', 'bundle exec rspec subgem/spec', 'Invoke-Pester -Path ./tools') but are invisible to the harness count as 0 generated tests.
Repeatedly observed in msbench cta-nightly runs:
- new C# test project never 'dotnet sln add'ed (ocelot)
- Pester tests placed under custom directories invisible to default Invoke-Pester (homebrew, ruby/ruby)
- RSpec specs placed in a sub-gem's spec/ dir invisible from repo root (fastlane)
Changes:
- code-testing-implementer: capture baseline test count in Step 2, new Step 7 'Verify Harness Discovery (MANDATORY)' with concrete failure examples, HARNESS_DISCOVERY line in report template, reminder after Step 3 to revisit registration if Step 4 creates a new project.
- code-testing-researcher: instruct to record BOTH scoped test command and harness-equivalent discovery command in .testagent/research.md.
- dotnet.md: strengthen 'Registering a new test project' heading to MANDATORY when dotnet new was used; add 'Harness Discovery Check' section with 'dotnet test <solution> --list-tests' from repo root.
- powershell.md: add 'Harness Discovery Check' section with default-config Invoke-Pester from repo root.
- ruby.md: add gem-monorepo trap guidance (fastlane, ruby/ruby) in Test Placement Contract; add 'Harness Discovery Check' section with 'bundle exec rspec --dry-run' from repo root.
* Address review feedback on harness-discovery check
- dotnet.md: replace stale `Step 8 cleanup` reference with the actual Step 3 (`Register Test Project with Build System`).
- dotnet.md: demote `Harness Discovery Check` to `###` so it nests under the parent `## .csproj / .sln Handling` section alongside `### Registering...`.
- dotnet.md: replace `\s\{4\}` (non-POSIX) in the grep regex with 4 literal spaces so the count works under BRE.
- ruby.md: group the rake/rails fallback with `{ ...; } | wc -l` so the pipe applies to both branches of `||`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* code-testing-agent: add polyglot pipeline examples for Python/TypeScript/Go/Java
The code-testing-agent skill family is polyglot in description but in
practice biased toward .NET because dotnet-examples.md was the only
filled-in pipeline walkthrough. The four sub-agents that participate in
the Research-Plan-Implement pipeline (researcher, planner, implementer,
generator) all pointed to dotnet-examples.md whenever they suggested a
concrete example, which made it harder for the agent to produce
idiomatic non-.NET tests (e.g. in msbench top5-* benchmarks for
Python/Flask, TypeScript/Express).
This change
* adds four new example files mirroring dotnet-examples.md format
(source → research → plan → generated test → fix cycle → final report):
- python-examples.md (pytest, unittest.mock, Mock(spec=...), parametrize)
- typescript-examples.md (Vitest with notes for Jest; it.each,
async tests, fake timers, ESM/CJS fix cycle)
- go-examples.md (standard testing package, table-driven subtests,
hand-written fake repository, injected clock)
- java-examples.md (JUnit 5 + Mockito on Maven, @ParameterizedTest +
@CsvSource, Clock.fixed, Surefire fix cycles)
* updates code-testing-extensions/SKILL.md TOC to list the new files
and clarifies usage instructions to read the matching <language>-
examples.md alongside the base extension
* makes the "Concrete example" pointers in code-testing-generator,
code-testing-implementer, code-testing-planner and
code-testing-researcher agents language-agnostic (list all available
example files instead of hard-coding dotnet-examples.md)
* expands code-testing-researcher project-structure detection list to
cover more Python (tox.ini, noxfile.py, requirements*.txt, uv.lock,
poetry.lock, pdm.lock), JS/TS (.mts/.cts/.jsx, vitest.config.*,
jest.config.*), C++ (CMakeLists.txt, BUILD.bazel, meson.build),
Java/Kotlin (pom.xml, build.gradle[.kts], wrappers), and other
ecosystem files; expands the Identify-Language section accordingly
* extends the "Language-Specific Examples" section in
code-testing-agent/SKILL.md to summarise each example file
Validated with: skill-validator check --plugin ./plugins/dotnet-test
(23 skills, 11 agents — all checks passed) and markdownlint-cli2.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #708
- go-examples.md: replace the hand-rolled `contains`/`stringIndex`
helpers with `strings.Contains` from the standard library. The hand-rolled
`contains` had subtly wrong semantics — `contains(""abc"", """")` returned
`false` while `strings.Contains` returns `true` — and the helpers are
unnecessary complexity for a code-generation example.
- go-examples.md: in the `go test -run` "wrong selection regex" sample fix
cycle, quote the test name and use `single_item` (matching the underscore
that the surrounding diagnosis text refers to) instead of the unquoted
`single item` which the shell would parse as two separate CLI arguments.
- java-examples.md: the source-tree file list described `Invoice.java` as a
`record` but the `InvoiceService.markAsPaid` example mutates the invoice
via `setStatus(...)` and `setPaidDate(...)` — records are immutable, so
the description was internally inconsistent. Re-describe it as a mutable
POJO with explicit mutators to match the service code.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The code-testing-* agent family generates tests for 12 languages (not just
.NET). Until now, no instruction told the implementer to keep its changes
additive: it would routinely modify existing test files non-additively
(deleting/reformatting lines) or edit non-test production code to make
something easier to test. Both behaviors cause the change to be rejected
by:
- the msbench 'simple' test verifier (any 'dels > 0' on a test file, or
any change to a non-test file, sets reward=0)
- most real-world test-quality gates, code-review policies, and CI
Add explicit cross-language rules in the implementer agent prompt:
1. Existing test files are append-only (no reformat/reorder/remove).
2. Do not modify non-test source files; surface untestable seams as
follow-ups for the testability-migration agent.
3. Prefer new test files over edits to existing ones when equivalent.
4. Build-system manifests may be edited only for project/dependency
registration.
Rule 6 in the Rules section now points at the Step 4 detail so the
implementer reads it on every phase.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* CTA: invocation-only baseline (dispatch mechanics, no content rules)
Experiment branch to isolate the impact of "invoke prompted subagents
more often" from the impact of "make those subagents do richer work."
Same baseline as dev/ykovalova/cta-prompt-tuning (main = 66628b6), but
strips out every content/quality rule and keeps only the dispatch
plumbing.
Comparison branch: dev/ykovalova/cta-prompt-tuning (HEAD: bd530be)
which contains both the dispatch mechanics AND content/quality rules.
Files modified (3 vs 5 in cta-prompt-tuning):
- code-testing-generator.agent.md +110 lines
- code-testing-implementer.agent.md +11 lines
- code-testing-fixer.agent.md +1 line
- code-testing-researcher.agent.md UNTOUCHED (baseline)
- code-testing-planner.agent.md UNTOUCHED (baseline)
KEPT (invocation / dispatch mechanics):
Generator:
- Rule 1: every task() call MUST use agent_type "dotnet-test:code-testing-..."
(without this, calls dispatch generic built-ins and never reach the
named CTA agents)
- Rule 2: routing table -- which named agent for which job
- Rule 3: prefer one named-agent dispatch over many tool calls
- Rule 4: orchestrator MUST NOT edit/create test files itself
(forces implementer dispatch)
- Rule 5: orchestrator MUST NOT run builds/tests via terminal
(forces builder/tester dispatch)
- Rule 6: every run MUST dispatch the planner (no exceptions for "small"
scope; Direct still goes through planner)
- Rule 7: every build/test failure MUST dispatch the fixer
- Step 1b: mandatory initial researcher dispatch (every strategy)
- Direct strategy rewritten: dispatches planner -> implementer -> builder
-> tester -> fixer -> linter (was "Skip Steps 3-5, write tests inline")
- All Step 3/4/5/6/7/8/9 dispatches converted from runSubagent({agent:...})
to task({ agent_type: "dotnet-test:code-testing-...", name:..., prompt:...})
- Step 9 validator dispatch (forces builder dispatch for cleanup)
- Steps 6/7 mandatory builder/tester dispatch wrapper
Implementer:
- Section 5: "you MUST dispatch fixer for build errors" + no-inline-edit
block (forces fixer dispatch on build failures)
- Section 6: "you MUST dispatch fixer for test failures" + no-inline-edit
block (forces fixer dispatch on test failures)
- Section 7: "Format Code (mandatory if a lint command exists)"
(was "Optional"; mandatory firing of linter)
- Rule 6: never declare SUCCESS while build/tests fail (gates SUCCESS on
fixer dispatch)
- Rule 7: no inline test-file edits between failed dispatch and fixer
Fixer:
- Frontmatter description widened to advertise handling of failing tests
(without this, the orchestrator's routing logic does not select the
fixer for test failures, so even Rule 7's mandate produces no firing
-- this is the change that took fixer firing from 0.00/inst to 0.39/inst
in earlier iterations)
DROPPED (content / quality rules -- in cta-prompt-tuning, NOT here):
Generator:
- Test-strength rules embedded in implementer dispatch prompt
- Test-design rules embedded in implementer dispatch prompt (OFAT,
mutation self-check, never mock subject under test)
- File-location rules embedded in implementer dispatch prompt
- TARGET ENTITIES / PHASE CHECKLIST / TEST TRACEABILITY blocks in
implementer dispatch prompt
- CHECKLIST format spec in planner dispatch prompt
- Step 9 validator's detailed cleanup classification
Implementer:
- Section 4b "Verify CHECKLIST coverage" pre-completion check
- Section 8 "CHECKLIST COVERAGE" report block
- "Honor the CHECKLIST" rule
Fixer:
- "Process -- Failing Tests" section (5-step diagnosis flow)
- All anti-weakening / anti-skipping rules
- "Re-derive expected from production source" guidance
Planner:
- CHECKLIST format ("one item per TARGET BEHAVIOR, Source/Variants/
Expected mandatory")
- "Test name from research.md conventions" rule
- "At least 2 phases" rule
Researcher:
- Section 8 "Extract Local Test Naming & Style Conventions"
- TARGET ENTITIES / TARGET BEHAVIORS / TEST INFRASTRUCTURE structure
in research.md
- Test naming pattern extraction
WHAT THE SUBAGENTS WILL ACTUALLY DO:
The researcher / planner / implementer / fixer all operate at baseline
behavior -- they receive the same prompts they receive in the upstream
"vanilla" runs. The only difference vs vanilla is that the orchestrator
ACTUALLY DISPATCHES THEM (where vanilla often inlines the work or skips
sub-agent dispatch entirely).
EXPECTED COMPARISON:
If quality on this branch is similar to or higher than dev/ykovalova/
cta-prompt-tuning (bd530be), then "more dispatches" is the dominant
quality lever and the content/quality rules in cta-prompt-tuning are
adding marginal or noise-level value.
If quality on this branch is materially lower than cta-prompt-tuning,
then the content/quality rules are doing the heavy lifting and the
dispatch mechanics alone are insufficient.
If quality on this branch matches or exceeds vanilla but trails
cta-prompt-tuning, then the dispatch mechanics provide a baseline lift
and the content rules add an incremental quality layer on top.
Rubber-duck check passed (validated dispatch-vs-content classification;
fixer frontmatter is routing metadata not a runtime gate; surviving
dispatch prompts contain no dangling references to removed CHECKLIST /
TARGET ENTITIES / TEST STRENGTH / naming-convention concepts).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* CTA: extract unit-under-test + behaviors quality cue (no caps)
* Address PR #646 review feedback
- Step 1b prompt: explicitly request unit-under-test (file:line) and behaviors
so the verification gate rarely needs re-dispatch.
- Step 3: rename to 'Deep Research Phase', mark skipped for Direct strategy,
and switch from overwriting research.md to extending it (no double-research).
- Step references: update '6-9' -> '6-10' and 'Step 9' -> 'Steps 9-10' in the
strategy table and the All-strategies-MUST line, since reporting is Step 10.
- Step 6 builder prompt: drop '*.sln' glob (could expand to multiple args);
use 'dotnet build --no-incremental' (auto-discovers .sln) per dotnet.md.
- Step 9: stop overloading the builder agent; perform diff/cleanup directly
in the orchestrator (Rule 5 forbids inline build/test, not git/fs hygiene).
- Fixer agent: update mission text to cover failing tests and assertion
correction (front-matter description already mentioned this; body now
matches), with explicit no-Ignore/no-Skip/no-production-rewrite guardrails.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Yuliia Kovalova <ykovalova@example.com>
* Improve test quality for benchmark performance
Based on SWE Atlas benchmark analysis (48.8% vanilla vs 19-34% CTA):
1. Default to Direct strategy for single-task requests — reduces
multi-agent overhead that costs tokens without improving results
2. Run tests immediately in Direct strategy — catches assertion
errors early instead of accumulating failures
3. Read source thoroughly before writing tests — trace actual logic
and return values, not just function signatures
4. Quality over Quantity guidelines — cover stated requirements first,
fewer focused tests beat many shallow ones
5. Verify tests are implementation-specific — tests that pass with
an empty function body aren't testing anything useful
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Combine redundant implementation-specificity bullets into one
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add end-to-end input/output examples to anchor expected behavior for the
LLM, addressing the Example Quality (2/5) review feedback.
Changes:
- New extensions/dotnet-examples.md with .NET-specific examples: sample
source code, research output, plan output, generated test file, fix
cycle walkthroughs, and final report
- SKILL.md: replace one-liner examples with strategy selection table,
pipeline walkthrough, and pointer to language-specific extensions
- Generator agent: add strategy decision examples and sample final report
- Researcher/Planner/Implementer agents: add references to extension
examples for concrete output shapes
Architecture keeps core agents language-agnostic — all language-specific
examples live in the extensions/ folder.
- Add 'Use when:' trigger phrases to all 7 sub-agent descriptions
so parent agents can reliably discover and delegate to them
- Add .testagent/ cleanup rule to generator agent to prevent
ephemeral pipeline state from being committed
* Add 'dotnet sln add' step to code-testing-implementer
When a new test project is created, register it with the solution
file using 'dotnet sln add' so that 'dotnet test <solution>'
discovers the tests.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: improve dotnet sln add guidance
- Use solution identified in research/plan rather than searching
- Account for .slnf solution filter scenarios
- Prefer 'dotnet test --solution' over 'dotnet test <solution>'
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move dotnet sln add guidance to dotnet.md extension
The implementer is polyglot, so 'dotnet sln add' doesn't belong
there. Replace with a generic 'register with build system' step
that defers to extensions/, and add the .NET-specific detail
(sln/slnx/slnf handling) to extensions/dotnet.md.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: fix solution registration guidance in dotnet.md
- Include .slnf in 'don't substitute' warning alongside .sln/.slnx
- Qualify '--solution' flag as SDK 10+/MTP-only; fall back to
positional form for older SDKs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Petr Pokorny