* Fix dotnet-test-frameworks skill activation for evaluations
The skill had user-invocable: false and a description saying 'DO NOT USE
directly', which prevented the model from activating it during evaluation.
All 6 eval scenarios default to expect_activation: true, causing
SkillNotActivated failures on CI.
- Remove user-invocable: false from both dotnet-test and experimental copies
- Rewrite description with activation keywords matching eval prompts
(assertion mapping, framework identification, skip annotations, exception
assertions, lifecycle conversion, integration test markers)
- Preserve note that other analysis skills also load this as reference data
* Restore user-invocable: false, fix cross-refs, add expect_activation: false
Address review comment: remove exp-* skill references from dotnet-test-
frameworks description since those skills reference exp-dotnet-test-
frameworks instead.
Optimize eval results: this reference data skill's content is knowledge
the model already has (baseline scores 5.0/5 on most scenarios). Direct
activation adds token/time overhead without quality improvement:
- Restore user-invocable: false (correct for reference data skill)
- Shorten description to not trigger activation
- Add expect_activation: false to all 6 scenarios
The skill's value is realized when loaded by consumer skills (e.g.
test-anti-patterns), not via direct user queries.
Adds a new agentic workflow (.github/workflows/markdown-linter.md) that
runs Super Linter on weekdays at 2 PM UTC and on manual dispatch. The
agent analyzes linter output and creates GitHub issues with categorized
findings, severity breakdown, and fix recommendations.
Adapted from https://github.com/githubnext/agentics/blob/main/workflows/markdown-linter.md
with repo-specific conventions: PAT pool rotation, fork protection,
pinned action SHAs, scoped bash tools, and safe-outputs.
Add end-to-end input/output examples to anchor expected behavior for the
LLM, addressing the Example Quality (2/5) review feedback.
Changes:
- New extensions/dotnet-examples.md with .NET-specific examples: sample
source code, research output, plan output, generated test file, fix
cycle walkthroughs, and final report
- SKILL.md: replace one-liner examples with strategy selection table,
pipeline walkthrough, and pointer to language-specific extensions
- Generator agent: add strategy decision examples and sample final report
- Researcher/Planner/Implementer agents: add references to extension
examples for concrete output shapes
Architecture keeps core agents language-agnostic — all language-specific
examples live in the extensions/ folder.
The 'Migrate MSTest.Sdk v3 project using ManagedType and TestTimeout'
scenario times out on CI because MSTest.Sdk/4.1.0 resolution takes 40+
seconds on cold NuGet cache. Each agent session may call dotnet build
multiple times, easily exceeding the 240s timeout.
Add a 'dotnet restore' setup command that runs before the agent starts
(outside the timeout window), pre-warming the NuGet cache and obj/
directory so subsequent builds complete in ~2s.
* Remove agentic workflows
The agentic workflows were useful when CCA didn't support loading skills. That's now solved and supported. The agentic workflows come with a couple of issues (limited chars) which make it hard to use. Remove them in favor of the plugin system.
* Remove Agentic Workflows section from README
Removed section about Agentic Workflows from README.
* Remove stale /agentic-workflows references from CONTRIBUTING.md, CODEOWNERS, and docs
Agent-Logs-Url: https://github.com/dotnet/skills/sessions/8f129e2f-156e-4ee9-9897-896cd1981513
Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Harden the test generator agent to ensure final build validation, test
validation, and git commit always run — even when using the Direct
strategy for small requests.
Changes:
- Step 1: Mandate reading language-specific extensions (e.g., dotnet.md)
before writing any code. The extension contains critical project
registration (dotnet sln add) and build validation guidance.
- Step 2: Direct strategy now skips only Steps 3-5 (research, plan,
implement sub-agents), NOT Steps 6-9. Added explicit note that
Steps 6-9 are mandatory for ALL strategies.
- Step 6: Clarify full-solution build must use no --framework flag to
catch multi-target framework issues (e.g., net472 errors).
- Step 7: Require fresh build for final test validation (no --no-build).
- New Step 9: Explicit git commit with verification that the diff is
non-empty.
- New rules 9-11: Read extensions first, always validate+commit,
preserve existing tests.
Motivated by MSBench investigation showing:
- Agent used Direct strategy and skipped final validation entirely
- Agent never read extensions/dotnet.md (missing dotnet sln add)
- Agent forgot to git commit (300 tests created, Changes +0 -0)
- Agent deleted pre-existing tests while adding new ones
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix timeout in MSTest.Sdk v3-to-v4 eval scenario 9
The fixture csproj had MSTest.Sdk/3.8.0 while the prompt claimed the user
already upgraded to 4.1.0. This mismatch caused the model to edit the
csproj and run dotnet build, triggering slow MSTest SDK NuGet downloads
that exceeded the 240s timeout on CI.
- Update fixture csproj to MSTest.Sdk/4.1.0 to match the prompt
- Add specific CS error codes to the prompt so the model provides direct
fixes instead of building
- Update rubric item to reflect the project already being on 4.1.0
* Fix CI timeout in migrate-mstest-v3-to-v4 eval scenario
Reduce scope of the 'Fix multiple v4 breaking changes' scenario:
- Remove 3 lower-signal breaking changes from the fixture (AreEqual params,
IsInstanceOfType out param, Properties.Contains) to cut agent work
- Add max_turns: 5 to cap agent iterations
- Lower timeout from 240 to 180 seconds
* Strengthen net6.0 scenario prompt for reliable skill activation
The test-anti-patterns skill was not reliably activated in plugin mode
because sibling descriptions overlapped:
- test-quality-auditor agent: narrowed to multi-skill audit pipelines
only, explicitly excluding single-file reviews
- writing-mstest-tests skill: removed 'review' and 'anti-patterns' from
description, added explicit exclusion for test quality auditing
- test-anti-patterns skill: expanded trigger phrases to better match
review prompts (flaky tests, duplication, tests that pass without
verifying)
Five scenarios were performing at or below stock Copilot because prompts
gave away answers and rubric items tested knowledge the baseline LLM
can deduce from the code alone.
Changes per scenario:
- Fix DataRow type mismatch: added double-to-float mismatch, vaguer
prompt, rubric tests both mismatch types and latest-3.x knowledge
- Migrate .testsettings to .runsettings: added code coverage data
collector to fixture, rubric tests correct TestTimeout mapping
(<MSTest><TestTimeout> not <TestSessionTimeout>) and collector mapping
- Migrate MSTest v2 NuGet to v3: vaguer prompt (forces model to read
csproj), rubric tests metapackage/MSTest.Sdk recommendation and
Microsoft.NET.Test.Sdk dependency awareness
- Fix Assert.AreEqual overload errors: removed exact error code from
prompt, added mixed-type and interface-reference cases, rubric tests
correct generic type parameter selection
- Handle dropped TFM: changed to multi-target (net5.0;net462;
netcoreapp3.1), rubric tests that model knows which TFMs are still
supported and doesn't upgrade them indiscriminately
- Add 'Use when:' trigger phrases to all 7 sub-agent descriptions
so parent agents can reliably discover and delegate to them
- Add .testagent/ cleanup rule to generator agent to prevent
ephemeral pipeline state from being committed
GITHUB_TOKEN cannot push commits that modify files under .github/workflows/.
This is a hard GitHub platform restriction (requires workflows scope, which
is only available via PAT or GitHub App tokens).
The workflow now:
- Runs gh aw upgrade (without --create-pull-request)
- Categorizes changes into pushable (outside .github/workflows/) and
non-pushable (inside .github/workflows/)
- Creates a PR with only the pushable changes (e.g. actions-lock.json)
- Documents any workflow files that need manual recompilation in the PR body
with instructions for running gh aw compile locally
Tested successfully on https://github.com/JanKrivanek/skills2/pull/1
The ContosoUniversity test fixture had 17 unused NuGet packages (bootstrap,
jQuery, Modernizr, Antlr4, WebGrease, Microsoft.Build.Tasks.Core, etc.)
that bloated NuGet restore and build time. The multi-agent test generation
pipeline performs multiple build/test cycles, causing cumulative delays
that exceeded the 2400s scenario timeout on CI.
Changes:
- Remove 17 unused PackageReference entries from ContosoUniversity.csproj
- Remove stale ItemGroup entries (Global.asax, Scripts, compile removes)
- Remove NU1701 warning suppression (no longer needed)
- Increase command_timeout from 180s to 300s for dotnet test assertion
(accounts for first-time restore + build + test + coverage collection)
The 5 failing scenarios were timing out because expect_tools: [bash]
forced the agent to execute terminal commands. The hot reload workflow
involves 'dotnet run' which starts a test host that blocks indefinitely
waiting for file changes, causing the scenario to exhaust its timeout.
These scenarios test whether the model provides correct guidance (package
name, env var, commands), not whether it actually executes them. The
output_matches assertions already verify the response content.
Removed expect_tools from:
- Suggest hot reload for failing test in MTP project (SDK 9)
- Suggest hot reload for failing test in MTP project (SDK 10)
- Enable hot reload when package already installed
- Suggest launchSettings.json configuration for hot reload
- Negative: VSTest project cannot use MTP hot reload
Also reduced timeouts since scenarios no longer need to wait for command
execution.
* Add markdownlint analysis for local dev and CI
- Add .markdownlint.jsonc with rule configuration
- Add .markdownlint-cli2.jsonc with ignore patterns
- Add package.json with markdownlint-cli2 and lint scripts
- Add GitHub Actions workflow running on PRs touching *.md files
- Add VS Code extension recommendation for vscode-markdownlint
* Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Address review comments: use markdownlint-cli2-action, merge configs, lint only changed files
- Replace manual Node.js + npm setup with DavidAnson/markdownlint-cli2-action (pinned to SHA)
- Use tj-actions/changed-files to lint only PR-changed markdown files
- Merge .markdownlint.jsonc into .markdownlint-cli2.jsonc (single config file)
- Remove package.json and package-lock.json (no longer needed)
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Add rollForward: latestFeature to all global.json entries that pin SDK 9.0.200
so that any 9.0.xxx SDK satisfies the requirement. Without this, machines with
only 9.0.3xx installed fail all dotnet commands during evaluation, causing the
model to burn its entire timeout budget on failed bash retries.
The fixture csproj had MSTest.Sdk/3.8.0 while the prompt claimed the user
already upgraded to 4.1.0. This mismatch caused the model to edit the
csproj and run dotnet build, triggering slow MSTest SDK NuGet downloads
that exceeded the 240s timeout on CI.
- Update fixture csproj to MSTest.Sdk/4.1.0 to match the prompt
- Add specific CS error codes to the prompt so the model provides direct
fixes instead of building
- Update rubric item to reflect the project already being on 4.1.0
* Add test-quality-auditor hub agent for .NET test suite analysis
Introduces a new test-quality-auditor agent that triages test quality
concerns to specialized analysis skills: assertion quality, test smells,
anti-patterns, mock usage, test gaps, maintainability, coverage risk,
and test tagging. Includes a comprehensive audit pipeline that chains
multiple skills for broad health checks.
* Address Copilot review: clarify diagnostic role and coverage routing
- Reword intro to acknowledge file-modifying workflows (test tagging)
are only used with explicit user confirmation
- Clarify coverage-analysis vs crap-score: prefer coverage-analysis for
project-wide analysis, reserve crap-score for targeted method/class scope
* Add test-migration hub agent for .NET test framework and platform migrations
Introduces a new test-migration agent that auto-detects the current test
framework, version, and platform, then routes to the appropriate migration
skill (MSTest v1/v2→v3, v3→v4, xUnit v2→v3, VSTest→MTP). Handles
multi-step migrations with commit boundaries between steps.
* Align framework detection with platform-detection skill
Address Copilot review: MSTest requires metapackage, Sdk, or both
TestFramework+TestAdapter (not either alone). Add missing xUnit MTP
package variants.
* Add skill-coverage PR workflow with slash command support
- Auto-triggers on PRs that change plugins/** or tests/**
- Detects specific plugin/skill pairs from the PR diff
- Runs Measure-SkillCoverage.ps1 and posts a coverage report as a PR comment
- Supports /skill-coverage slash command on PR comments:
/skill-coverage (runs on changed skills in the PR)
/skill-coverage <plugin> (runs on all skills in the plugin)
/skill-coverage <plugin> <skill> (runs on a specific skill)
* Address Copilot review: security, permissions, and robustness fixes
- Add permission check: require write access before running slash command
- Tighten /skill-coverage matching to avoid false triggers on similar commands
- Add issues: write permission (PR comments use Issues API)
- Use refs/pull/<num>/head for checkout to support fork PRs
- Remove unused \ variable
- Fix multi-line JSON parsing with Out-String
* Address PR #538 review: fork PR handling, reactions, and sourcing note
- Skip PR comment posting for fork PRs (read-only token); coverage still runs in step summary
- Add continue-on-error to reaction step to avoid 403 failures
- Add header comment noting issue_comment trigger sources from default branch
- Add continue-on-error to no-changes comment step
* Split into analyze + comment jobs; fix pagination and dedup
Security:
- Split into two jobs: analyze (contents: read only) and comment (write only)
- Analyze job checks out base branch for trusted tooling, PR head into
a separate git worktree - untrusted PR code is never executed
- Comment job only reads an uploaded artifact, no PR code checkout
Fixes:
- Use github.paginate() for listComments to handle PRs with >100 comments
- Deduplicate targets: plugin-wide changes replace per-skill targets for
that plugin instead of producing duplicates
Introduce Measure-SkillCoverage.ps1 under eng/skill-coverage/ that
measures how much of a SKILL.md's teaching content is verified by
eval.yaml test scenarios — analogous to code coverage for skill files.
The script extracts testable 'coverage points' from four structural
sections of a skill file (Validation checklist, Common Pitfalls,
Workflow Steps, and code patterns from fenced blocks), then
cross-references them against eval.yaml assertions and rubric
criteria using regex matching and keyword overlap heuristics.
Features:
- Per-skill and aggregate (-All) analysis across all plugins
- Table output with colored pass/fail and per-category percentages
- JSON output for machine consumption (single object or array)
- Distinguishes deterministic assertions from rubric-only coverage
- MinCoverage threshold for CI gating
- Tracks source line numbers for all coverage point categories
* Add 'dotnet sln add' step to code-testing-implementer
When a new test project is created, register it with the solution
file using 'dotnet sln add' so that 'dotnet test <solution>'
discovers the tests.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: improve dotnet sln add guidance
- Use solution identified in research/plan rather than searching
- Account for .slnf solution filter scenarios
- Prefer 'dotnet test --solution' over 'dotnet test <solution>'
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move dotnet sln add guidance to dotnet.md extension
The implementer is polyglot, so 'dotnet sln add' doesn't belong
there. Replace with a generic 'register with build system' step
that defers to extensions/, and add the .NET-specific detail
(sln/slnx/slnf handling) to extensions/dotnet.md.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: fix solution registration guidance in dotnet.md
- Include .slnf in 'don't substitute' warning alongside .sln/.slnx
- Qualify '--solution' flag as SDK 10+/MTP-only; fall back to
positional form for older SDKs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add validator options
* Address PR review: validate options and add tests
- Reject --allow-repo-traversal with --plugin mode (plugins must be portable)
- Validate --max-declaration-lines and --max-agent-lines are positive integers
- Add unit tests for MaxDeclarationLines override in SkillProfiler
- Add unit tests for AllowRepoTraversal suppression and depth check preservation
- Add unit tests for MaxAgentLines override and fallback to MaxDeclarationLines
* Remove body line count checks and --max-declaration-lines/--max-agent-lines options
Line count checks are redundant with BPE token-based complexity classification,
which already provides better guidance. Keep --allow-repo-traversal only.
* Address review: remove unused CheckOptions from AgentProfiler, fix traversal test
- Remove unused CheckOptions parameter from AgentProfiler.AnalyzeAgent and
RunAgentsCheckCore since line checks were removed
- Fix AllowRepoTraversalSuppressesParentTraversalError test to use a non-deep
path (../SKILL.md) and assert no file-ref errors at all
dependabot[bot]