mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
eval-exec-opus48
589 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Copilot
|
0e2c1aad7d |
Reword high-variance hint and show CV as percentage (#674)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
92954153dd |
Bump GitHub.Copilot.SDK to 0.3.0 and address breaking changes (#659)
* Bump GitHub.Copilot.SDK to 0.3.0 and address breaking changes - GitHub.Copilot.SDK 0.2.2 → 0.3.0 - YamlDotNet 16.3.0 → 17.1.0 - Vecc.YamlDotNet.Analyzers.StaticGenerator 16.3.0 → 17.0.0 - Nerdbank.MessagePack pinned to 1.1.62 (GHSA-2cwq-pwfr-wcw3) Breaking changes addressed: - LocalSessionFsHandler now extends abstract SessionFsProvider (simplified method signatures with raw parameters) - McpServers config uses McpStdioServerConfig instead of Dictionary<string,object> - SessionFsSetProviderRequestConventions → SessionFsSetProviderConventions - PermissionRequestResultKind.DeniedByRules → .UserNotAvailable (obsolete) - Mtime/Birthtime now DateTimeOffset instead of string - Entry/EntryType → SessionFsReaddirWithTypesEntry/EntryType * Address review: validate MCP server type, suppress YamlDotNet trim warning - Reject non-stdio MCP server types with a clear error message instead of silently treating them as stdio. - Suppress IL2104 (YamlDotNet 17.x not fully trim-compatible) so that PublishAot native builds succeed. |
||
|
Amaury Levé
|
dde9f99cf0 |
dotnet-test: drop tools field from agents that declared it (#665)
The four agents (code-testing-generator, test-migration, testability-migration, test-quality-auditor) were the only dotnet-test agents declaring a 'tools:' list. Per Jan's suggestion on #660, drop the field so they inherit the runtime's default tool surface (matching the convention used by every other agent in the repo). Follow-up to #660. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
3c23523092 |
Fix vally eval isolation: pin baseline to 0 skills, skilled to 1 (#662)
* Fix vally eval isolation: pin baseline to 0 skills, skilled to 1 vally evaluations were not isolating skills properly. In the failing run https://github.com/dotnet/skills/actions/runs/26021981688, the dotnet-test `code-testing-agent` eval reported "Found 77 skill(s)" for baseline and "Found 22 skill(s)" for the skilled run instead of the expected 0 and 1. Root cause: `vally eval` derives skills via `discoverSkills(skillRoot)` where `skillRoot = opts.skillDir ?? cwd`: - Baseline omitted `--skill-dir`, so vally walked up from the repo root to `.vally.yaml` and loaded every skill under `plugins/` (77 total). - The CI workflow's skilled run passed `--skill-dir` pointing at the plugin-level `plugins/<plugin>/skills` (from `matrix.entry.skills_path`). That directory has no `SKILL.md` at its root, so vally's Step-4 child scan picked up every sibling skill in the plugin (22 for dotnet-test). Fix applied to both `eng/vally-adapter/run-vally-evals.sh` and `.github/workflows/vally-evaluation.yml`: - Baseline now passes `--skill-dir <empty mktemp dir>` so vally discovers 0 skills. - Skilled now uses `plugins/<plugin>/skills/<eval-name>` (the specific skill directory, not the plugin-level one) so vally matches Step-1 (SKILL.md at root) and discovers exactly 1 skill. - Evals with no matching `SKILL.md` (the `agent.*` orchestrator evals that declare their own `environment.skills`) are explicitly skipped with a warning -- baseline-vs-skilled isolation is not meaningful for them, since they require multi-skill setup by design. - `Find eval specs` now narrows to a single eval when `workflow_dispatch` is invoked with a specific skill, so single-skill dispatches do not accidentally run every eval in the plugin. - Empty temp dirs are created per-eval and cleaned up after each run to avoid any cross-run contention under parallel execution. After the fix the CI log should report `Found 0 skill(s)` for baseline and `Found 1 skill(s): <skill-name>` for skilled. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: trap-based cleanup for EMPTY_SKILL_DIR; fail-loud on missing dispatch skill - run-vally-evals.sh: register a RETURN trap immediately after creating EMPTY_SKILL_DIR so it is removed on every exit path, including aborts triggered by set -euo pipefail (e.g. if ode adapt.mjs exits non-zero inside the log-capture block). Removes the unconditional rm at the end of the function which would not run on errors. - vally-evaluation.yml: when workflow_dispatch supplies a skill input but no ests/<plugin>/<skill>/eval.vally.yaml exists, emit an explicit ::error:: that includes the missing candidate path and fail the job, instead of silently falling through to the generic `No eval.vally.yaml files found for <plugin>` warning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
7c876d45a3 |
skill-validator: serialize concurrent writes per path in LocalSessionFsHandler (#654)
During multi-model skill-validator evals, judge calls intermittently fail with: `Failed to persist session events: Error: Failed to append to JSONL` `file session-state\events.jsonl: Error: timeout while waiting for` `mutex to become available` The error wording (the word `mutex`) does not originate in our C# code and is reported by the GitHub Copilot SDK over RPC, so the underlying contention lives inside the SDK's session event-persistence layer. AgentRunner already gives each session its own `configDir` and LlmSession (judge) gives each call a unique temp root, so paths cannot collide ACROSS sessions — this points to intra-session contention on a single events.jsonl path. Mitigation: serialize concurrent writes per resolved path inside our handler with a `ConcurrentDictionary<string, SemaphoreSlim>` keyed by the absolute path. Both `WriteFileAsync` and `AppendFileAsync` route through a per-path lock helper. Reads remain unlocked. Even though the conflict surfaces inside the SDK, removing one variable (our handler issuing concurrent writes to the same file) is good defense and makes the `ISessionFsHandler` thread-safe in its own right. The dictionary is bounded by the number of unique files written within one session (typically <10), so leak risk is negligible for the per-session object lifetime. Validation: build passes; full xUnit suite 545/545 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
2a193300ca |
Replace 'terminal' with 'bash' and 'powershell' in agent tools (#660)
The skill-validator only recognizes 'bash' and 'powershell' as built-in shell tools, not 'terminal'. Update the four affected agents in the dotnet-test plugin to use both for cross-platform support. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
b6b53a268d |
skill-validator: bump per-plugin aggregate description cap from 15K to 20K (and document its informal origin) (#655)
* skill-validator: bump per-plugin aggregate description cap from 15K to 20K The dotnet-test plugin (22 skills) is now at ~14K aggregate description characters and bumping into the 15,000 cap on every meaningful description change. This blocks legitimate plugin growth on what is, on inspection, an informal repo guardrail rather than a documented Copilot constraint. Origin of the 15K number Traced via 'git log -S MaxAggregateDescriptionLength' to PR #238, which references issue #222. The issue verbatim says: `users may apply a limit to the total size of 'description' across` `all skills in a plugin (...). 15K characters was mentioned, we` `could choose smaller.` So the value is informal — picked because it `was mentioned` and explicitly leaving room to `choose smaller`. Spec status The agentskills.io specification (https://agentskills.io/specification) documents per-skill description (1024), compatibility (500), and name (64) limits. It does NOT define any aggregate per-plugin cap. The `MaxDescriptionLength = 1024` constant in this same file already cites the spec; the aggregate constant carried no citation comment. What this PR does - Bumps `MaxAggregateDescriptionLength` from 15,000 to 20,000. Picks a round value that gives ~30% headroom over the largest current plugin (dotnet-test ~14K) without making the cap meaningless. - Adds a comment documenting the origin (informal, issue #222), the absence of a spec source, and a TODO to validate this guardrail against literature on skill-routing degradation and to run experiments (skill-validator already has the infrastructure to A/B-measure description noise vs activation accuracy). What this PR does NOT do - It does not remove the cap. The cost concern in #222 is real (every description is loaded at agent startup for routing); we just don't yet have evidence for the right ceiling. - It does not change per-skill limits. Validation All 545 tests pass; the existing `CheckCommandAggregateDescriptionTests` reference the constant symbolically and continue to pass with the new value. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Pin Nerdbank.MessagePack to 1.1.62 and add spec URL to cap comment - Transitively pin Nerdbank.MessagePack from 1.0.2 (vulnerable via StreamJsonRpc 2.24.84 -> GitHub.Copilot.SDK) to 1.1.62, which contains the fix for the stackalloc DateTime vulnerability (GHSA-2cwq-pwfr-wcw3). Without this pin, restore fails with NU1903 since TreatWarningsAsErrors is true. - Add the agentskills.io specification URL to the MaxAggregateDescriptionLength comment so future readers can verify the claim that the per-plugin aggregate cap is a local repo policy rather than a spec-mandated limit (addresses PR #655 review feedback). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
04348a46ef |
tests/dotnet-test/code-testing-agent: fix coverage XML lookup path (#657)
The grader expected coverage.cobertura.xml under a workspace-root TestResults/ directory, but `dotnet test --collect:"XPlat Code Coverage"` writes the report under <TestProject>/TestResults/<guid>/coverage.cobertura.xml. With the previous path, `find TestResults …` failed with no-such-directory before even searching, and the glob `TestResults/**/coverage.cobertura.xml` could not match the nested location either, so the assertion failed on otherwise-successful runs. Search recursively from cwd in both eval.vally.yaml (`find . …`) and eval.yaml (`**/TestResults/**/coverage.cobertura.xml`) so the assertion matches the actual output location of XPlat Code Coverage regardless of the test project layout the agent chose. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
b0c2336d49 |
Tighten crap-score boundary to fix coverage-analysis plateau activation (#652)
Dashboard data showed coverage-analysis failing to activate in 8/10 recent scheduled plugin-mode runs for the 'Coverage plateau diagnosis' scenario, while activating reliably in isolated mode. PR #647 added positive triggers to coverage-analysis but did not address sibling attention competition. The crap-score description matched the plateau prompt almost as well as coverage-analysis (it advertised 'evaluate whether complex methods have sufficient test coverage' + 'Requires code coverage data (Cobertura XML)') without redirecting project-wide / stuck-coverage diagnosis to coverage-analysis. With 22 sibling skills competing for attention this overlap is enough to suppress activation altogether. Tighten the crap-score frontmatter to: - Scope positive triggers to a named method, class, or single source file (the actual eval surface — see tests/dotnet-test/crap-score/ eval.yaml, all 3 scenarios target OrderService.cs). - Add explicit DO NOT USE FOR redirects covering project-wide coverage analysis, coverage plateau / stuck coverage, what's blocking coverage, and where to add tests across a project — all of which point at coverage-analysis. skill-validator check passes (22 skills, 11 agents, 1 plugin). Aggregate dotnet-test description size: 14,932 chars (limit 15,000). markdownlint passes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
3d59e44c7e |
Fix coverage-analysis activation for plateau diagnosis prompts (#647)
* Fix coverage-analysis activation for plateau diagnosis prompts coverage-analysis SKILL.md: - Trim verbose implementation details (provider detection, ReportGenerator) that consumed description budget without aiding skill activation - Add explicit USE FOR keywords: coverage stuck, coverage plateau, can't increase coverage, what's blocking coverage code-testing-agent SKILL.md: - Add 'diagnosing coverage plateaus or CRAP score computation (use coverage-analysis)' to DO NOT USE FOR boundary to prevent test-generation skill from intercepting diagnostic prompts * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Strengthen code-testing-agent activation; harden coverage-analysis isolated mode Address eval regressions reported on PR #647 (run 25813728646): 1. code-testing-agent: `Generate tests for ContosoUniversity ASP.NET Core MVC app` was NOT ACTIVATED in plugin mode (detectedSkills=[], skillEventCount=0, invokedAgents=[]). The model bypassed the skill system entirely. - SKILL.md description: restructure to use the proven `Use when user says ...` pattern with quoted trigger phrases (matching the run-tests skill that consistently activates), make the link to the code-testing-generator sub-agent explicit, and tighten DO NOT USE FOR clauses. - eval prompt (eval.yaml + eval.vally.yaml): make the request pipeline-shaped (`project-wide, multi-file test generation task`, `scaffold a new test project`) so the model recognizes it as multi-step work that benefits from the orchestrated pipeline. Explicitly request coverlet.collector + a Cobertura XML run so rubric criterion 1 (`high line coverage as reported by the Cobertura XML in TestResults/`) becomes achievable without overfitting. 2. code-testing-tester agent + code-testing-extensions/dotnet.md: open a scoped exception to the `skip coverage tools` rule. Default behavior stays the same, but when the user/harness explicitly asks for a Cobertura/XML coverage artifact, the agent may add coverlet.collector to the generated test csproj so the harness's coverage command produces output. The agent still does not run the coverage command itself. 3. coverage-analysis SKILL.md: add a `User-visible output is mandatory` guard at the top of the Workflow section. The latest eval showed isolated mode producing literally `(no output)` in 2 of 3 scenarios — the agent ran Compute-CrapScores.ps1 / Extract-MethodCoverage.ps1 / ReportGenerator in parallel, then the session ended without ever surfacing findings. The guard tells the agent to always return a partial summary instead of ending silent, and to deprioritize ReportGenerator HTML when budget is tight. (Plugin-mode quality is already strong: 4.3 / 4.3 / 5.0 — no regression risk there.) Aggregate dotnet-test plugin description size: 14,925 chars (limit 15,000). skill-validator check passes (22 skills, 11 agents, 1 plugin); markdownlint passes for all 4 modified files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore MSTest modernization exclusion in code-testing-agent description * Fix isolated-mode coverage-analysis: emit summary before optional ReportGenerator The previous workflow encouraged the agent to run `dotnet tool install` for ReportGenerator in parallel with the CRAP scoring scripts (Phase 2 "Steps 3 and 4 in parallel" + Phase 3 "Steps 5 and 6 in parallel"). In isolated mode that pattern reliably crashed the session with "Failed to persist session events: timeout while waiting for mutex to become available" right after the scripts returned valid data, so the agent never produced the user-facing summary. Restructure the workflow into 5 phases: - Phase 1 (Setup) - unchanged - Phase 2 (Test execution) - skip when Cobertura XML already exists - Phase 3 (Analysis) - run only the two PowerShell scripts, no RG - Phase 4 (User-facing summary) - MANDATORY, must be the next assistant response after Phase 3, before any RG work; also save coverage-analysis.md as a secondary follow-up - Phase 5 (ReportGenerator HTML/CSV) - strictly optional, post-summary, skipped by default for existing-Cobertura and plateau-diagnosis paths Also update references/output-format.md so the Reports section marks RG artifacts as "Not generated (optional - request HTML reports to enable)" when Phase 5 has not run, and update references/guidelines.md so the "show and open the markdown report" rule explicitly defers to the user-facing assistant response. Targets the isolated-mode regressions in PR #647 eval: - Project-wide coverage with existing Cobertura: 1.0/5 -> expected 3+ - Coverage plateau diagnosis: 1.0/5 -> expected 3+ - Run coverage from scratch: 2.3/5 -> expected steady or up Verified: skill-validator check --plugin ./plugins/dotnet-test passes; markdownlint-cli2 clean on all 3 modified files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #647 review comments 1. Prerequisites: distinguish the from-scratch path (needs NuGet for the coverage-provider package install + optional internet for ReportGenerator) from the existing-Cobertura path (needs neither). 2. Add Step 2c "Discover or accept existing Cobertura XML" so the existing-data path actually has $coberturaFiles populated before Phase 3, instead of relying on Phase 2's discovery (which it skips). Also clarify Step 2's destructive Remove-Item only manages the skill-owned coverage-analysis/ subdirectory. 3. references/output-format.md: replace the unconditional "Reports saved to: <coverageDir>/reports/" line with one that always points at <coverageDir>/ (markdown summary + raw Cobertura) and only mentions reports/ if Phase 5 ran. 4. Have Compute-CrapScores.ps1 emit OVERALL_LINE_COVERAGE and OVERALL_BRANCH_COVERAGE from the Cobertura root attributes, and update Phase 4 to read those values directly from the script's output. The Phase 4 mandatory-summary rule no longer requires a separate XML parse before composing the response. Verified: skill-validator check --plugin ./plugins/dotnet-test passes; markdownlint-cli2 clean on all modified files; Compute-CrapScores.ps1 smoke-tested on a synthetic Cobertura XML (emits OVERALL_LINE_COVERAGE:75, OVERALL_BRANCH_COVERAGE:50 alongside HOTSPOTS). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply suggestion from @Evangelink * Address unresolved coverage-analysis and eval review comments * Refine follow-up review feedback from validation * Tighten coverage aggregation fallback notes and counters * Clarify pre-response save instruction wording --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
42afc2688d |
Add Go, Java, Rust, Ruby, Swift, and Kotlin extensions to code-testing-extensions (#650)
* Add Go, Java, Rust, Ruby, Swift, and Kotlin extensions to code-testing-extensions Adds six new language-specific guidance files to the code-testing-extensions skill, modeled on the existing python.md and typescript.md structure. Each new extension covers: investigate-first rule, toolchain detection, build commands, test commands (with framework variants), lint command, project layout/imports, common errors with fixes, mocking rules, dependency installation as last resort, and a Skip Coverage Tools footer. SKILL.md is updated to advertise the new files in the discovery table. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback and fix markdownlint issues - rust.md: replace # code span (MD038) with quoted text - java.md: clarify wildcard-import guidance to avoid contradicting the explicit-imports template - java.md: scope 'class must be public' to JUnit 4 only (JUnit 5 allows package-private) - java.md: correct Mockito final-class fix to inline mock maker (default in 5+, mockito-inline for 3.x/4.x); remove incorrect mockito-subclass suggestion - kotlin.md: same Mockito final-class correction - kotlin.md: replace nonexistent testScheduler.testDispatcher() with StandardTestDispatcher(testScheduler) - kotlin.md: rephrase coroutines-test dependency guidance to avoid contradiction - kotlin.md: correct cross-module 'internal' fix; @VisibleForTesting does not widen Kotlin visibility - ruby.md: replace numbered block parameter '_1' with '|f| require f' (works on Ruby < 2.7) - swift.md: Swift Testing requires Swift 6/Xcode 16, not 5.10 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
f532cf1d94 |
Add coverage-touching and self-referential assertion heuristics (#635)
* Add coverage-touching and self-referential assertion heuristics
- test-anti-patterns: add 'Coverage touching' and 'Self-referential assertion'
as named Critical-severity anti-patterns in the detection catalog
- assertion-quality: add 'Tests with self-referential assertions' metric and
a calibration rule distinguishing tautological from valid round-trip assertions
- Add eval scenarios for both new heuristics in test-anti-patterns (coverage-
touching ProductCatalog facade, self-referential SerializerTests)
- Add eval scenario and fixture for self-referential assertions in
assertion-quality (ConfigTests with identity/round-trip patterns)
* Fix test-anti-patterns activation: rewrite description with USE FOR keyword list
The skill was not activating in any eval scenario because the description
used narrative prose instead of the explicit 'USE FOR:' / 'DO NOT USE FOR:'
keyword list pattern that well-activating sibling skills use (run-tests,
writing-mstest-tests). Restructured the description with trigger phrases
matching common user prompts.
* Fix eval activation, timeouts, and add missing vally scenarios
- test-anti-patterns/eval.vally.yaml: increase timeout 2m→5m, add
coverage-touching and self-referential scenarios with constraints
- assertion-quality/eval.vally.yaml: add self-referential scenario
with environment files, broaden assertion pattern
- test-anti-patterns/eval.yaml: add setup.files to all 6 scenarios
to create workspace files that trigger skill discovery/activation
- assertion-quality/eval.yaml: increase timeouts 120→300 for
self-referential and decline-request scenarios, broaden
self-referential assertion pattern
- Create fixture files for mixed-severity, well-written,
coverage-touching, and self-referential test-anti-patterns scenarios
* Address review comments: fix test names and use source references
- Rename ThrowsArgumentException to ThrowsArgumentOutOfRangeException in
PaymentProcessorTests.cs fixture and inlined code in eval.yaml and
eval.vally.yaml to match the actual exception type being asserted.
- Replace inline csproj content blocks with source references to existing
fixture files for UserRepo, Payments, and CatalogService scenarios in
eval.yaml to avoid drift between fixtures and eval setup.
* Address review comments: pin MSTest versions, use source refs, fix naming
- Pin MSTest to 3.8.2 in all fixture .csproj files, add standard
properties (ImplicitUsings, Nullable, IsPackable)
- Replace inline csproj content blocks with source references to
fixture files to avoid content drift
- Create missing .csproj fixtures for flaky-coupled, duplicate-magic,
and assertion-quality self-referential scenarios
- Fix test method names: ThrowsArgumentException ->
ThrowsArgumentOutOfRangeException to match the actual assertion
* Fix activation, eval timeouts, and per-skill description limits
Address remaining issues from PR #635 eval scoreboard
(run 25677669933) and skill-validator failures:
1. test-anti-patterns activation (was NOT ACTIVATED in BOTH plugin and
isolated mode for ALL 6 scenarios — model answered everything from
inline code without ever needing the skill):
- eval.yaml + eval.vally.yaml: drop the giant inline `csharp` blocks
from every prompt. Each prompt now names the workspace files
explicitly (`Audit `Foo.Tests/FooTests.cs` for .NET test
anti-patterns ...`) and asks for the skill's signature deliverable
(severity-ranked list with concrete code-level fixes). The model
must read the workspace, which gives it a chance to consult the
skill — and the skill's structured-findings format is the actual
value-add over inline analysis.
- All 6 scenarios consistently get reject_tools (read-only review)
and timeout 180s.
- eval.vally.yaml: also added missing environment.files for each
scenario (was prompt-only before, no fixture on disk).
2. assertion-quality `Flag assertion-free tests` plugin-mode activation
(was bouncing to other skills because the prompt was generic `check
whether these tests verify anything useful`):
- eval.yaml + eval.vally.yaml: rewrite the prompt to name the file
(`SmokeTests/ApiEndpointSmokeTests.cs`) and explicitly request
`assertion quality and depth`, `assertion-free tests`, and
`trivial assertions like Assert.IsNotNull` — direct trigger
phrases for assertion-quality.
- assertion-quality SKILL.md: extend description with explicit USE
FOR keywords for `assertion-free tests`, `self-referential`, and
`tautological` so plugin-mode activation isn't won by sibling
skills like test-anti-patterns.
3. test-anti-patterns description size — was 1,301 chars (over the
1,024 per-skill limit) AND pushed dotnet-test plugin aggregate to
15,312 (over the 15,000 limit), failing skill-check on PR #635.
Trimmed to 779 chars; aggregate now at 14,983.
4. Vally job-level timeout (60m → 180m) was already raised on main
in commit
|
||
|
YuliiaKovalova
|
e1e8568375 |
Revert "dotnet-test: add unit-under-test + behaviors quality cue to code-test…" (#651)
This reverts commit
|
||
|
Amaury Levé
|
de0a97c551 |
Fix migrate-vstest-to-mtp activation for Handle exit code 8 scenario (#648)
The `Handle exit code 8 when migrating from VSTest to MTP` scenario was
not activating the migrate-vstest-to-mtp skill in plugin mode in the
latest scheduled eval (skillActivationPlugin.activated=false,
detectedSkills=[], skillEventCount=0, vally row showed `--`), even
though all 10 other scenarios in this eval activated cleanly.
Three layered fixes:
1. Prompt (eval.yaml + eval.vally.yaml): switch from past-tense
`We migrated`/`How do I fix this behavioral difference` to
present-tense migration framing — `We're migrating ... After
enabling MTP ... How do I handle this MTP behavioral difference?`.
The previous past-tense framing read as a generic post-migration CI
troubleshooting question, which the model felt confident answering
without consulting any skill.
2. SKILL.md description: add an explicit `USE FOR:` clause for MTP
behavioral differences (exit code 8, zero tests discovered,
--ignore-exit-code, TESTINGPLATFORM_EXITCODE_IGNORE), matching the
pattern that fixed coverage-analysis activation in
|
||
|
Amaury Levé
|
0f31228b58 |
Fix eval timeouts and skill activation for dotnet-test evals (#641)
* Fix evaluation timeouts and improve prompts for dotnet-test skills
- writing-mstest-tests: increase global vally timeout from 3m to 5m to match
the 300s eval.yaml timeout for the complex 'Modernize legacy test patterns' stimulus
- code-testing-agent: increase timeout from 40m/2400s to 60m/3600s
- mtp-hot-reload: increase timeout from 6m to 10m and remove overly strict
expect_tools constraint on launchSettings stimulus
- migrate-mstest-v1v2-to-v3: rephrase prompt for assertion overload errors to
mention CS1501 and cover AreNotEqual/AreSame alongside AreEqual
* Increase eval timeouts for mtp-hot-reload and migrate-mstest-v1v2-to-v3 scenarios
* Strengthen activation prompts and add missing vally scenarios
- migrate-mstest-v1v2-to-v3: Add explicit 'MSTest v2 to v3' migration
language in .testsettings and DataRow prompts (both eval.yaml and
eval.vally.yaml)
- writing-mstest-tests: Add 'MSTest' keyword and skill-specific
terminology to all prompts that lacked activation signal
- writing-mstest-tests: Add 5 missing vally scenarios (string
assertions, comparison assertions, collection/null/reference
assertions, conditional execution, parallelization)
- Add service-registry fixture files for the collection assertions
scenario
* Improve skill activation via SKILL.md descriptions and prompt alignment
Three-pronged fix for systematic NOT ACTIVATED failures:
1. SKILL.md descriptions: Restructure writing-mstest-tests, migrate-
mstest-v1v2-to-v3, and code-testing-agent descriptions to use the
proven 'Use when user says' pattern with quoted trigger phrases
(matching the working run-tests pattern)
2. Vally prompts: Each prompt now contains at least one exact quoted
trigger phrase from its SKILL.md description to ensure strong
description-to-prompt matching
3. Specific fixes:
- writing-mstest-tests: All 15 prompts now include MSTest-specific
trigger phrases (Assert.ThrowsExactly, sealed test class, DataRow,
DynamicData, Assert.HasCount, Assert.IsInstanceOfType, etc.)
- migrate-mstest-v1v2-to-v3: Add '.NET 5 dropped' and 'framework
compatibility issues' to trigger phrases; strengthen dropped-TFM
prompt
- code-testing-agent: Add 'generate comprehensive unit tests' and
'achieve high code coverage' to trigger phrases
* Revert SKILL.md description changes that caused activation regression
The description restructuring in
|
||
|
Amaury Levé
|
f8cae94781 |
Add test-tagging evals for uncovered skill aspects (#639)
* Add test-tagging evals for uncovered skill aspects Add 3 new scenarios (6-8) to test-tagging eval.yaml: - Scenario 6: Partially-tagged MSTest suite — verifies existing traits are preserved without duplication, only taxonomy values are used, and framework detection works correctly. - Scenario 7: Misleading method names (NUnit) — verifies the agent reads test bodies instead of guessing from method names, uses NUnit [Category] syntax, and classifies security/concurrency/boundary tests accurately. - Scenario 8: Build verification — verifies the project still compiles after tagging changes and only valid taxonomy values are applied. New fixtures: - fixtures/mstest-partially-tagged: OrderService suite with 3 pre-tagged and 12 untagged tests. - fixtures/nunit-misleading-names: TaskManager suite where method names deliberately contradict test body behavior. * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Add missing vally stimuli for test-tagging scenarios 6-8 --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
f1b09eba79 |
dotnet-test: add unit-under-test + behaviors quality cue to code-testing-generator (#646)
* CTA: invocation-only baseline (dispatch mechanics, no content rules)
Experiment branch to isolate the impact of "invoke prompted subagents
more often" from the impact of "make those subagents do richer work."
Same baseline as dev/ykovalova/cta-prompt-tuning (main =
|
||
|
Amaury Levé
|
4c40b873f2 |
Add detect-static-dependencies evals for report structure, obj exclusion, and lambda statics (#640)
Cover previously uncovered skill aspects: - [Validation] All .cs files in scope were scanned (check count) - [Validation] Report includes category totals, top patterns, and affected files - [Validation] obj/ and bin/ directories were excluded - [Pitfall] Scanning obj/ or generated code - [Pitfall] Missing statics inside lambdas/LINQ - [WorkflowStep] Step 4: Present the report Three new scenarios with supporting fixtures: - Structured report with file count, categories, and top patterns - Exclude obj/bin directories (with Generated.cs decoy in obj/) - Detect statics inside lambdas and LINQ expressions |
||
|
Jakub Jareš
|
6e7adc5e32 |
Add extension files for Python, TypeScript, and PowerShell (#617)
* Add Python and TypeScript extension files for code-testing pipeline First-draft language extension files following the dotnet.md pattern. These guide the polyglot test agent on build/test/lint/fix for each language. Python: pytest-focused, environment/runner detection (Poetry/PDM/uv/Hatch), public-API testing philosophy, common errors, mocking guidelines. TypeScript: package-manager detection (npm/pnpm/yarn/bun), Jest+Vitest+Mocha support, ESM/CJS guidance, TS-specific considerations, framework detection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revise python.md and typescript.md based on 10-agent review Key changes: - Add 'Rule #1: Investigate the repo first' as top section - Remove generic templates (anchor to wrong conventions) - Remove testing philosophy (unactionable for agent) - Fix command inconsistency (parameterize with <prefix>/<exec>) - Move dependency install to 'last resort' section - Add error-driven fixer playbook with concrete fixes - Fix Mocha --grep (test name, not file filter) - Fix ESM guidance (don't change package.json type field) - Add monorepo/workspace guidance (Nx, Turborepo) - Add framework notes (React, Express, NestJS) - Add Jest mock hoisting warning - Cut from 224+289 lines to 117+136 lines (~57% reduction) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add powershell.md extension for Pester v5 test generation Key sections: - Repo investigation first (shell target detection) - Pester v5 discovery vs run phase rules - Import patterns for modules, library scripts, and executable scripts - Cross-platform guidance (pwsh vs powershell.exe, Join-Path, casing) - Non-terminating error handling with Should -Throw - Mock scoping, -ModuleName, PesterBoundParameters - TestDrive: for file-based tests - Non-obvious assertion gotchas (Contain vs Be, Throw needs scriptblock) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reword test infra guidance; add framework detection to dotnet.md - Reword 'do not add' to 'use existing, only introduce if none exist' - Add test framework detection table to dotnet.md (MSTest/xUnit/NUnit) - Tailor wording per language (Python defaults to pytest, TS follows scripts.test, PS defaults to Pester) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make Python extension framework-adaptive instead of pytest-centric Rule #1 now searches for ALL test file formats (not just test_*.py), explicitly calls out custom frameworks like UTscapy, and emphasizes adopting repo conventions fully rather than layering pytest on top. Test Commands section now has a custom framework block before the pytest block, and Test File Naming defers to repo conventions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
github-actions[bot]
|
0521ae0ac6 |
chore: upgrade gh-aw runtime (#606)
* chore: upgrade gh-aw runtime * chore: recompile gh-aw workflows --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Jan Krivanek <jankrivanek@microsoft.com> |
||
|
dependabot[bot]
|
3a323e4b9b |
Bump github/gh-aw-actions in the github-actions-dependencies group (#642)
Bumps the github-actions-dependencies group with 1 update: [github/gh-aw-actions](https://github.com/github/gh-aw-actions). Updates `github/gh-aw-actions` from 0.72.1 to 0.74.1 - [Release notes](https://github.com/github/gh-aw-actions/releases) - [Changelog](https://github.com/github/gh-aw-actions/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/gh-aw-actions/compare/bc56a0cad2f450c562810785ef38649c04db812a...b07cf98ac5874e8f51c34ba52099d8a6fac2ef93) --- updated-dependencies: - dependency-name: github/gh-aw-actions dependency-version: 0.74.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: github-actions-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Dan Moseley
|
fe11dec69d |
Update .NET 10→11 migration skill for Preview 2 and 3 (#628)
* Update .NET 10→11 migration skill for Preview 2 and 3 Add breaking changes from .NET 11 Preview 2 and Preview 3 to the migrate-dotnet10-to-dotnet11 skill. The skill previously covered only Preview 1. Changes: - Add 15 new breaking changes across all reference files - Create new aspnetcore-dotnet10to11.md reference (OpenAPI v3, Blazor) - Update SKILL.md workflow steps, assessment checklist, and patterns - Add 4 new eval scenarios covering high-impact P2/P3 changes - Add github.com/dotnet/aspnetcore to known-domains.txt Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback - Replace prompt-echoable 'corrupt' in ZipArchive eval regex with 'checksum' - Update EF Core reference note to say 'Preview 1 through Preview 3' - Add missing Source link for Zstandard entry Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: tighten eval regex, fix raw HTML - Tighten OverscanCount grader regex to require 'OverscanCount' with a number, instead of matching bare '3' anywhere - Wrap Blazor component snippet in backticks so it renders as code Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
e6cf308357 |
Add eval scenario for filter-syntax [trait] query language pattern (#638)
* Add eval scenario for filter-syntax [trait] query language pattern Add a run-tests scenario that exercises the xUnit v3 query filter language with [traitName=traitValue] syntax (--filter-query), covering the uncovered [CodePattern] [trait] at line 113 of filter-syntax SKILL.md. New fixture: mtp-xunit-v3-query with multiple test classes and traits (Smoke, Regression, Unit) to test combined class-pattern + trait filtering. * Address review feedback for xUnit v3 query eval coverage --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
205e9f54e2 |
fix(groom): prune resolved rows, add First Seen column, handle duplicate sections (#637)
* fix(groom): prune resolved rows, add First Seen column, handle duplicate sections - Rename 'Status' column to 'Investigation' for clarity - Add 'First Seen' column showing when finding was first detected - Remove resolved investigation rows instead of keeping them forever - Handle duplicate Investigation Results sections by merging into one - Update health-check workflow to use matching column schema - Drop resolved rows from health-check table preservation logic * fix(health-check): remove duplicate Investigation Results section The health-check and groom workflows both created a '## Investigation Results' section, causing duplicates in issue #288. The groom owns this section via its replace-island block — remove it from the health-check template and update the guideline accordingly. * fix(health-check): restore Investigation Results as placeholder section The health-check replaces the entire issue body, so removing the section entirely caused it to disappear. Instead, keep it as a placeholder that: - Preserves existing rows from the groom's island markers - Appends new 'Dispatched' rows for findings dispatched in this run - Does NOT add island markers itself (the groom adds those) This ensures the section survives health-check body rewrites while the groom still owns the detailed content via replace-island. * fix: address PR review comments on groom/health-check workflows - Fix duplicate section guidance: clarify replace-island only updates first occurrence, doesn't auto-remove later duplicates - Match example dispatched row text with health-check template - Fix First Seen source: parse from summary line (first seen YYYY-MM-DD) instead of non-existent **First seen:** field - Preserve all non-resolved rows (not just Done rows) in health-check - Fix Step 4.2 to align with prune behavior in Step 4.3 - Add schema migration for legacy 4-column tables in health-check - Add link to dispatched investigation rows |
||
|
Amaury Levé
|
5856011d8f | Add README.md for dotnet-test plugin (#636) | ||
|
Amaury Levé
|
d575b1faa8 |
Add eval scenarios for writing-mstest-tests skill coverage (#593)
* Add eval scenarios for writing-mstest-tests skill coverage * Fix invalid YAML and remove build-validation claim from rubric * Fix timeouts and low quality * Improve further * Increase timeout for string assertions scenario from 120s to 180s Agent-Logs-Url: https://github.com/dotnet/skills/sessions/9943a60f-070b-4bea-9912-c994d1bed428 Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
cf831597ff |
fix(groom): use MCP tools with min-integrity:none instead of unauthenticated gh CLI (#633)
* fix(groom): use MCP tools with min-integrity:none instead of unauthenticated gh CLI The previous fix (PR #627) instructed the agent to use 'gh api' for reading issue comments. However, the gh CLI is NOT authenticated inside the sandbox — the framework deliberately strips credentials via the 'Clean git credentials' step. The system prompt also explicitly states 'gh is not authenticated'. Root cause analysis of run 25481332080 (pre-fix) shows the MCP issue_read tool DID return comments — only 1 item out of many was filtered by integrity policy. The agent overreacted to the [Filtered] warning and called report_incomplete instead of working with the available data. This fix: - Sets min-integrity: none under tools.github so the integrity filter does not remove any comments. This is safe because the workflow only processes comments from github-actions[bot] (Step 2 filters by author). - Reverts to MCP issue_read with get_comments (the proper tool for reading comments in the sandbox). - Removes gh from bash tools (not usable without auth). - Adds explicit guidance: if filtered items appear, continue working with available data — do NOT call report_incomplete or missing_tool. - Documents that gh CLI is NOT authenticated in the sandbox. Closes #625 * fix: address review comments - scope repos, bound comment fetch, fix jq guideline |
||
|
Amaury Levé
|
cfc293f8fe |
Fix skill activation for Assert.AreEqual overload eval scenario (#589)
The 'Fix Assert.AreEqual object overload errors after v3 upgrade' evaluation scenario was failing because: 1. The prompt said 'latest major version' without mentioning 'v3' or 'MSTest v3', missing the skill's activation keywords. 2. The project file showed MSTest 3.8.0 (post-upgrade state) with no v2 context, while the SKILL.md 'When Not to Use' said 'Project already uses MSTest v3'. 3. The prompt's 'fix compilation issues' phrasing competed with writing-mstest-tests skill activation. Changes: - Updated prompt to explicitly mention upgrading from v2 to v3 and reference MSTest v3 compatibility issues - Added XML comment in csproj showing the pre-upgrade v2 package versions - Clarified SKILL.md 'When Not to Use' to distinguish fully-migrated projects from projects with active migration errors |
||
|
Amaury Levé
|
9f531203ec |
Promote 4 experimental test skills to dotnet-test plugin (#634)
Move assertion-quality, test-gap-analysis, test-smell-detection, and test-tagging from dotnet-experimental to dotnet-test. These skills have demonstrated sufficient eval coverage, unique value beyond baseline model capabilities, and are already integrated into the test-quality-auditor agent pipeline. Changes: - Move skills and tests, dropping the exp- prefix - Update all cross-references in skill descriptions and routing tables - Update test-quality-auditor agent and eval scenarios - Update CODEOWNERS for the moved paths - Delete duplicate exp-dotnet-test-frameworks (identical to dotnet-test-frameworks) - Fix stale exp-dotnet-test-frameworks reference in exp-test-maintainability Skills held back for further iteration: - exp-mock-usage-analysis: baseline model scores 5/5 on key scenarios - exp-test-maintainability: quality gains offset by token overhead |
||
|
Amaury Levé
|
97b23ae750 |
Fix timeout for Modernize legacy test patterns evaluation (#590)
The scenario was timing out on CI because: - The 180s timeout left insufficient headroom for CI machines (local runs take 120-135s, CI machines are slower due to load, network NuGet restores, and parallel execution) Changes: - Add 'dotnet restore' setup command to pre-warm NuGet cache before the agent starts (same pattern used by migrate-mstest-v3-to-v4) - Increase timeout from 180s to 300s to accommodate CI variability |
||
|
Jan Jones
|
3aefa0f10a |
Update file-based app skill to mention features from 10.0.300 (#602)
Co-authored-by: Copilot <copilot@github.com> |
||
|
dependabot[bot]
|
4a4c424ee4 |
Bump the github-actions-dependencies group across 1 directory with 3 updates (#631)
* Bump the github-actions-dependencies group across 1 directory with 3 updates Bumps the github-actions-dependencies group with 3 updates in the / directory: [github/gh-aw-actions](https://github.com/github/gh-aw-actions), [actions/upload-artifact](https://github.com/actions/upload-artifact) and [actions/setup-node](https://github.com/actions/setup-node). Updates `github/gh-aw-actions` from 2fe53acc038ba01c3bbdc767d4b25df31ca5bdfc to ea222e359276c0702a5f5203547ff9d88d0ddd76 - [Release notes](https://github.com/github/gh-aw-actions/releases) - [Changelog](https://github.com/github/gh-aw-actions/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/gh-aw-actions/compare/2fe53acc038ba01c3bbdc767d4b25df31ca5bdfc...ea222e359276c0702a5f5203547ff9d88d0ddd76) Updates `actions/upload-artifact` from 7.0.0 to 7.0.1 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/v7...043fb46d1a93c77aae656e7c1c64a875d1fc6a0a) Updates `actions/setup-node` from 4.4.0 to 6.4.0 - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](https://github.com/actions/setup-node/compare/v4.4.0...48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e) --- updated-dependencies: - dependency-name: github/gh-aw-actions dependency-version: ea222e359276c0702a5f5203547ff9d88d0ddd76 dependency-type: direct:production dependency-group: github-actions-dependencies - dependency-name: actions/upload-artifact dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions-dependencies - dependency-name: actions/setup-node dependency-version: 6.4.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> * Update gh-aw-actions from v0.68.1 to v0.72.1 in agentics-maintenance.yml Agent-Logs-Url: https://github.com/dotnet/skills/sessions/2c61295e-713c-4b82-9681-339f0923989d Co-authored-by: danmoseley <6385855+danmoseley@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: danmoseley <6385855+danmoseley@users.noreply.github.com> |
||
|
Dan Moseley
|
e7ec5a77a0 | Fix markdownlint CI workflow (startup_failure since inception) (#629) | ||
|
Jan Krivanek
|
b306b4f11d |
Fix groom workflow: bypass MCP integrity filter with gh api (#627)
* Fix groom workflow: use gh api to bypass MCP integrity filter The MCP gateway's integrity policy intermittently blocks access to issue comments (report_incomplete: 'Resource issue:dotnet/skills#288 has lower integrity than agent requires'). This caused issue #625. Fix: Add 'gh' to allowed bash tools and instruct the agent to use 'gh api --paginate' for fetching comments. This bypasses the MCP gateway entirely and also provides node_id fields required by hide-comment (which the MCP issue_read tool did not return). * Address PR review: fix --paginate JSON, add author filter, narrow guideline scope |
||
|
Jan Krivanek
|
0bdf428da0 |
Fix triage workflow: separate label and assignee operations (#626)
* Fix triage workflow: separate label and assignee operations - Use add_labels tool for labeling (Step 6) instead of update_issue - Reserve update_issue exclusively for assignee assignment (Step 4) - Increase update_issue max to 2 for assignment flexibility - Add explicit workflow_dispatch guidance: agent must pass issue_number/item_number on all safe-output calls when there is no triggering issue context - Fixes issues where owners were mentioned in comments but not actually assigned to the issue * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Aditya Mandaleeka
|
3cba22740b | Include Vally job in parallel with existing eval (#622) | ||
|
Aditya Mandaleeka
|
6a6340d82b | fix: use 'vally' binary name in CI workflow (#621) | ||
|
Aditya Mandaleeka
|
9433a31e67 | Initial port of evals to Vally (#615) | ||
|
Jan Krivanek
|
66628b651e | Add triage workflows (#612) | ||
|
Jan Krivanek
|
b9d343f3b8 | Isolate failures during eval and judging (#611) | ||
|
Jan Krivanek
|
b774eff859 |
Fix deploy-dashboard condition for workflow_dispatch (#591)
* Dashboards series fix (followup) * Fix deploy-dashboard condition for workflow_dispatch The deploy-dashboard job required needs.discover.result == 'success', but the discover job's condition does not include workflow_dispatch, so it is always skipped for manual triggers. This made workflow_dispatch unable to deploy the dashboard. Split the condition so workflow_dispatch from main bypasses the discover dependency, while scheduled runs still require it. * Update deploy-dashboard comment to reflect workflow_dispatch support |
||
|
Copilot
|
2f9922e073 |
Remove LangVersion=latest from skill examples and test fixtures (#608)
* Initial plan * Remove LangVersion=latest from skill docs and test fixtures Per official C# docs, LangVersion should not be set to `latest` as it causes builds to silently vary across machines with different SDKs. Best practice is to omit LangVersion entirely and let the TFM determine the correct C# version automatically. - Update msbuild-modernization/SKILL.md: remove LangVersion=latest from example and add explicit guidance to omit LangVersion - Update directory-build-organization/SKILL.md: remove from example - Update directory-build-organization/references/multi-level-examples.md: remove from all 4 examples - Update msbuild-antipatterns/SKILL.md: remove from AP-08 examples - Remove LangVersion=latest from 11 test fixture project files Agent-Logs-Url: https://github.com/dotnet/skills/sessions/7a3d2722-770b-4010-a6e8-70c92668ad4c Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Fix LangVersion guidance and stale eval rubric reference Agent-Logs-Url: https://github.com/dotnet/skills/sessions/e89afcd8-7227-424b-9c8c-9ec0ce7113c2 Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> |
||
|
Manish Jayaswal
|
2633d3d482 |
add a new dotnet11 plugin and a new system text json skill to it. (#535)
* add a new dotnet11 plugin and a new system text json skill to it. * Update plugins/dotnet11/skills/system-text-json-net11/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * added the option to run dotnet11 skills. by default they don't get included * Update tests/dotnet11/system-text-json-net11/eval.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add JSON output assertions to system-text-json-net11 eval.yaml Agent-Logs-Url: https://github.com/dotnet/skills/sessions/8e302f3f-4328-44a0-845d-4eb0c930af63 Co-authored-by: ManishJayaswal <9527491+ManishJayaswal@users.noreply.github.com> * addressed CR comments * addressed more CR comments * fixed agument to pass while enabling dotnet11 skills * Remove dotnet11-specific filtering from skill-validator and evaluation workflow Now that .NET 11 is the default SDK for the repo (global.json updated on main), there is no need for special dotnet11 exclusion logic. The dotnet11 plugin skills are treated like any other plugin and included in all evaluation runs. Changes: - Remove --include-dotnet11 CLI option and filtering logic from EvaluateCommand.cs - Remove IncludeDotnet11 from ValidatorConfig model - Remove --dotnet11 flag parsing from evaluation workflow gate job - Remove dotnet11 exclusion logic from discover job - Remove conditional .NET 11 preview SDK install step (global.json handles it) - Remove dotnet11 from excludeFromSchedule list - Remove all dotnet11-specific warning/status messages - Remove section 9 from InvestigatingResults.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * address code review comments after rebase * added the plugin to cursor and added MIT license to skill --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: [Manish Jayaswal] <[manishj@microsoft.com]> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ManishJayaswal <9527491+ManishJayaswal@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Tomas Grosup
|
1613c98fe5 |
Honor --allow-repo-traversal for /-prefixed file references (#604)
/-prefix paths bypassed the flag and hit the depth check unconditionally. Now treated symmetrically with .. traversal: allowed only with the flag, otherwise produce a clear absolute (repo-rooted) path error. Also classify // as a protocol-relative URL. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
22575b2ac0 | Update SDK to .NET 11 preview in global.json (#603) | ||
|
YuliiaKovalova
|
809d0b180c |
Improve test generation quality based on SWE Atlas benchmark analysis (#599)
* Improve test quality for benchmark performance Based on SWE Atlas benchmark analysis (48.8% vanilla vs 19-34% CTA): 1. Default to Direct strategy for single-task requests — reduces multi-agent overhead that costs tokens without improving results 2. Run tests immediately in Direct strategy — catches assertion errors early instead of accumulating failures 3. Read source thoroughly before writing tests — trace actual logic and return values, not just function signatures 4. Quality over Quantity guidelines — cover stated requirements first, fewer focused tests beat many shallow ones 5. Verify tests are implementation-specific — tests that pass with an empty function body aren't testing anything useful Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Combine redundant implementation-specificity bullets into one Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Tomas Grosup
|
a55227f030 |
Fold F# .fsproj ordering into msbuild-antipatterns AP-05 (#577)
* Add fsharp-project-structure skill for .fsproj file ordering and .fsi files Teaches agents F# compilation order rules, signature file placement, and common FS0039/FS0010/FS0034 fixes. Includes eval with three scenarios (add module, fix broken order, add signature file) and test fixtures. Validated +64.6% improvement on gpt-4.1 (significant, CI [+17.2%, +63.0%]). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add fsharp-project-structure skill for .fsproj file ordering and .fsi files Teaches agents F# compilation order rules, signature file placement, and common FS0039/FS0010/FS0034 fixes. Includes eval with three scenarios (add module, fix broken order, add signature file) and test fixtures. Validated +64.6% improvement on gpt-4.1 (significant, CI [+17.2%, +63.0%]). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/fsharp-project-structure/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fold F# project-structure into msbuild-antipatterns AP-05 Scraps the standalone fsharp-project-structure skill and instead adds a short F# exception to AP-05 (Manual File Listing in SDK-Style Projects): F# compilation is order-dependent, so .fsproj files must list Compile items in dependency order (utilities first, entry point last), and .fsi signature files must appear immediately before their companion .fs. The three F# eval scenarios (add module, fix broken order, add signature file) are ported into tests/dotnet-msbuild/msbuild-antipatterns/eval.yaml with fixtures isolated under a fsharp/ subfolder (with empty Directory.Build.{props,targets} to prevent the existing C#-antipattern fixture files from leaking into the F# builds). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
github-actions[bot]
|
dbeee31b23 |
chore: upgrade gh-aw runtime (#592)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Jan Krivanek
|
3eb7206231 | Dashboards series fix (followup) (#587) | ||
|
dependabot[bot]
|
59c795c896 |
Bump the microsoft-and-system group with 4 updates (#574)
Bumps Microsoft.Bcl.Memory from 10.0.6 to 10.0.7 Bumps Microsoft.Data.Sqlite from 10.0.6 to 10.0.7 Bumps Microsoft.Extensions.FileSystemGlobbing from 10.0.6 to 10.0.7 Bumps System.CommandLine from 2.0.6 to 2.0.7 --- updated-dependencies: - dependency-name: Microsoft.Bcl.Memory dependency-version: 10.0.7 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Data.Sqlite dependency-version: 10.0.7 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Extensions.FileSystemGlobbing dependency-version: 10.0.7 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: System.CommandLine dependency-version: 2.0.7 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Bcl.Memory dependency-version: 10.0.7 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Data.Sqlite dependency-version: 10.0.7 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Extensions.FileSystemGlobbing dependency-version: 10.0.7 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: System.CommandLine dependency-version: 2.0.7 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |