* CTA: invocation-only baseline (dispatch mechanics, no content rules)
Experiment branch to isolate the impact of "invoke prompted subagents
more often" from the impact of "make those subagents do richer work."
Same baseline as dev/ykovalova/cta-prompt-tuning (main = 66628b6), but
strips out every content/quality rule and keeps only the dispatch
plumbing.
Comparison branch: dev/ykovalova/cta-prompt-tuning (HEAD: bd530be)
which contains both the dispatch mechanics AND content/quality rules.
Files modified (3 vs 5 in cta-prompt-tuning):
- code-testing-generator.agent.md +110 lines
- code-testing-implementer.agent.md +11 lines
- code-testing-fixer.agent.md +1 line
- code-testing-researcher.agent.md UNTOUCHED (baseline)
- code-testing-planner.agent.md UNTOUCHED (baseline)
KEPT (invocation / dispatch mechanics):
Generator:
- Rule 1: every task() call MUST use agent_type "dotnet-test:code-testing-..."
(without this, calls dispatch generic built-ins and never reach the
named CTA agents)
- Rule 2: routing table -- which named agent for which job
- Rule 3: prefer one named-agent dispatch over many tool calls
- Rule 4: orchestrator MUST NOT edit/create test files itself
(forces implementer dispatch)
- Rule 5: orchestrator MUST NOT run builds/tests via terminal
(forces builder/tester dispatch)
- Rule 6: every run MUST dispatch the planner (no exceptions for "small"
scope; Direct still goes through planner)
- Rule 7: every build/test failure MUST dispatch the fixer
- Step 1b: mandatory initial researcher dispatch (every strategy)
- Direct strategy rewritten: dispatches planner -> implementer -> builder
-> tester -> fixer -> linter (was "Skip Steps 3-5, write tests inline")
- All Step 3/4/5/6/7/8/9 dispatches converted from runSubagent({agent:...})
to task({ agent_type: "dotnet-test:code-testing-...", name:..., prompt:...})
- Step 9 validator dispatch (forces builder dispatch for cleanup)
- Steps 6/7 mandatory builder/tester dispatch wrapper
Implementer:
- Section 5: "you MUST dispatch fixer for build errors" + no-inline-edit
block (forces fixer dispatch on build failures)
- Section 6: "you MUST dispatch fixer for test failures" + no-inline-edit
block (forces fixer dispatch on test failures)
- Section 7: "Format Code (mandatory if a lint command exists)"
(was "Optional"; mandatory firing of linter)
- Rule 6: never declare SUCCESS while build/tests fail (gates SUCCESS on
fixer dispatch)
- Rule 7: no inline test-file edits between failed dispatch and fixer
Fixer:
- Frontmatter description widened to advertise handling of failing tests
(without this, the orchestrator's routing logic does not select the
fixer for test failures, so even Rule 7's mandate produces no firing
-- this is the change that took fixer firing from 0.00/inst to 0.39/inst
in earlier iterations)
DROPPED (content / quality rules -- in cta-prompt-tuning, NOT here):
Generator:
- Test-strength rules embedded in implementer dispatch prompt
- Test-design rules embedded in implementer dispatch prompt (OFAT,
mutation self-check, never mock subject under test)
- File-location rules embedded in implementer dispatch prompt
- TARGET ENTITIES / PHASE CHECKLIST / TEST TRACEABILITY blocks in
implementer dispatch prompt
- CHECKLIST format spec in planner dispatch prompt
- Step 9 validator's detailed cleanup classification
Implementer:
- Section 4b "Verify CHECKLIST coverage" pre-completion check
- Section 8 "CHECKLIST COVERAGE" report block
- "Honor the CHECKLIST" rule
Fixer:
- "Process -- Failing Tests" section (5-step diagnosis flow)
- All anti-weakening / anti-skipping rules
- "Re-derive expected from production source" guidance
Planner:
- CHECKLIST format ("one item per TARGET BEHAVIOR, Source/Variants/
Expected mandatory")
- "Test name from research.md conventions" rule
- "At least 2 phases" rule
Researcher:
- Section 8 "Extract Local Test Naming & Style Conventions"
- TARGET ENTITIES / TARGET BEHAVIORS / TEST INFRASTRUCTURE structure
in research.md
- Test naming pattern extraction
WHAT THE SUBAGENTS WILL ACTUALLY DO:
The researcher / planner / implementer / fixer all operate at baseline
behavior -- they receive the same prompts they receive in the upstream
"vanilla" runs. The only difference vs vanilla is that the orchestrator
ACTUALLY DISPATCHES THEM (where vanilla often inlines the work or skips
sub-agent dispatch entirely).
EXPECTED COMPARISON:
If quality on this branch is similar to or higher than dev/ykovalova/
cta-prompt-tuning (bd530be), then "more dispatches" is the dominant
quality lever and the content/quality rules in cta-prompt-tuning are
adding marginal or noise-level value.
If quality on this branch is materially lower than cta-prompt-tuning,
then the content/quality rules are doing the heavy lifting and the
dispatch mechanics alone are insufficient.
If quality on this branch matches or exceeds vanilla but trails
cta-prompt-tuning, then the dispatch mechanics provide a baseline lift
and the content rules add an incremental quality layer on top.
Rubber-duck check passed (validated dispatch-vs-content classification;
fixer frontmatter is routing metadata not a runtime gate; surviving
dispatch prompts contain no dangling references to removed CHECKLIST /
TARGET ENTITIES / TEST STRENGTH / naming-convention concepts).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* CTA: extract unit-under-test + behaviors quality cue (no caps)
* Address PR #646 review feedback
- Step 1b prompt: explicitly request unit-under-test (file:line) and behaviors
so the verification gate rarely needs re-dispatch.
- Step 3: rename to 'Deep Research Phase', mark skipped for Direct strategy,
and switch from overwriting research.md to extending it (no double-research).
- Step references: update '6-9' -> '6-10' and 'Step 9' -> 'Steps 9-10' in the
strategy table and the All-strategies-MUST line, since reporting is Step 10.
- Step 6 builder prompt: drop '*.sln' glob (could expand to multiple args);
use 'dotnet build --no-incremental' (auto-discovers .sln) per dotnet.md.
- Step 9: stop overloading the builder agent; perform diff/cleanup directly
in the orchestrator (Rule 5 forbids inline build/test, not git/fs hygiene).
- Fixer agent: update mission text to cover failing tests and assertion
correction (front-matter description already mentioned this; body now
matches), with explicit no-Ignore/no-Skip/no-production-rewrite guardrails.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Yuliia Kovalova <ykovalova@example.com>
* Improve test quality for benchmark performance
Based on SWE Atlas benchmark analysis (48.8% vanilla vs 19-34% CTA):
1. Default to Direct strategy for single-task requests — reduces
multi-agent overhead that costs tokens without improving results
2. Run tests immediately in Direct strategy — catches assertion
errors early instead of accumulating failures
3. Read source thoroughly before writing tests — trace actual logic
and return values, not just function signatures
4. Quality over Quantity guidelines — cover stated requirements first,
fewer focused tests beat many shallow ones
5. Verify tests are implementation-specific — tests that pass with
an empty function body aren't testing anything useful
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Combine redundant implementation-specificity bullets into one
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add end-to-end input/output examples to anchor expected behavior for the
LLM, addressing the Example Quality (2/5) review feedback.
Changes:
- New extensions/dotnet-examples.md with .NET-specific examples: sample
source code, research output, plan output, generated test file, fix
cycle walkthroughs, and final report
- SKILL.md: replace one-liner examples with strategy selection table,
pipeline walkthrough, and pointer to language-specific extensions
- Generator agent: add strategy decision examples and sample final report
- Researcher/Planner/Implementer agents: add references to extension
examples for concrete output shapes
Architecture keeps core agents language-agnostic — all language-specific
examples live in the extensions/ folder.
- Add 'Use when:' trigger phrases to all 7 sub-agent descriptions
so parent agents can reliably discover and delegate to them
- Add .testagent/ cleanup rule to generator agent to prevent
ephemeral pipeline state from being committed
* Add 'dotnet sln add' step to code-testing-implementer
When a new test project is created, register it with the solution
file using 'dotnet sln add' so that 'dotnet test <solution>'
discovers the tests.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: improve dotnet sln add guidance
- Use solution identified in research/plan rather than searching
- Account for .slnf solution filter scenarios
- Prefer 'dotnet test --solution' over 'dotnet test <solution>'
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move dotnet sln add guidance to dotnet.md extension
The implementer is polyglot, so 'dotnet sln add' doesn't belong
there. Replace with a generic 'register with build system' step
that defers to extensions/, and add the .NET-specific detail
(sln/slnx/slnf handling) to extensions/dotnet.md.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: fix solution registration guidance in dotnet.md
- Include .slnf in 'don't substitute' warning alongside .sln/.slnx
- Qualify '--solution' flag as SDK 10+/MTP-only; fall back to
positional form for older SDKs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
YuliiaKovalova