Resolve bundled files from each skill root, use one bounded directory listing when direct reads fail, and make reduced reference coverage visible.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9c391191-97d7-4a4b-bb79-1e279c37d095
Use the exact MEVD package name and mark the provider-dependent query example as pseudocode.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 20291ed7-7a54-43f4-ab73-0772fb4f7352
Restore implementation guidance for Copilot SDK, ONNX Runtime, and OllamaSharp. Fix sample compile issues and align the churn evaluation prompt with its implementation rubric.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 20291ed7-7a54-43f4-ab73-0772fb4f7352
* Support classic .NET test projects
Teach dotnet-test skills and agents to preserve non-SDK projects, packages.config dependencies, explicit compile registration, legacy runners, and version-compatible MSTest APIs. Add regression evals for generation, execution, coverage, and authoring.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fdfec89f-b610-479c-a6c7-c2936b300e68
* Address classic test fixture review
Tighten the MSTest version grader, make the runner fixture assertion behavioral, and use nameof for the guarded parameter.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fdfec89f-b610-479c-a6c7-c2936b300e68
---------
Copilot-Session: fdfec89f-b610-479c-a6c7-c2936b300e68
* Add golden test generation scenarios
Add reusable scaffolding and testability skills plus eval fixtures for zero-to-one, partial-suite gap, and ambient-dependency workflows.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7
* Address golden scenario review feedback
Use the SDK-style project type GUID in the solution fixture and keep the discount lower bound explicitly decimal.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7
* Fix eval grep alternation
Use extended regular expressions so forbidden-call and fake-name graders reliably match each alternative without YAML escape ambiguity.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7
* Increase golden eval discrimination
Bundle isolated-run guidance, sharpen plugin routing, and add solution-filter, nested-scope, and fake-time delay decisions. Repeat the expensive end-to-end scenarios twice to clear the observed tie rate.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7
* Specialize golden test workflows
Restore code-testing and migration skills to main, promote zero-to-one scaffolding to its own skill, extend verified gap analysis through focused test additions, and concentrate testability evaluation on nested ambient-scope behavior.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7
* Increase specialized eval margin
Fix xUnit v3 discovery and no-op scaffolding, constrain smoke-test scope, double the clean gap-analysis evidence, and add two deterministic static-identity scenarios matching the strongest testability delta.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7
* Remove unstable eval scenarios
Drop consistently losing or inert scenarios, add SDK solution registration and empirical retry-policy mutation coverage, and sharpen activation for specialized workflows.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7
* Align testability workflow routing
Narrow scaffold no-op routing, separate targeted obstacle handling from migrated-code test generation, and add an agent scenario covering migration plus deterministic tests.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7
---------
Copilot-Session: d71b84f3-c114-48c5-9da0-59dd981916c7
* Revert "Resolve binlog MCP from the dnceng dotnet-public feed"
This reverts commit 805a42a675.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3b2899e9-48f8-4a2a-a19f-1827d08cbfc4
* ci(eval): configure NuGet source explicitly
Install a repository-owned NuGet configuration at the runner user level so temporary Vally workspaces resolve packages from nuget.org without plugin-specific source arguments.
* build: use dotnet-public for repository NuGet configs
Route evaluation and version-stamping tool restores through the locked-down dotnet-public source mapping.
* fix(msbuild): enable prerelease BinlogMcp versions
Match the Roslyn LSP dnx configuration and keep all mirrored MCP manifests consistent.
---------
Co-authored-by: Jan Krivanek <jankrivanek@microsoft.com>
Copilot-Session: 3b2899e9-48f8-4a2a-a19f-1827d08cbfc4
* Resolve binlog MCP from the dnceng dotnet-public feed
The `binlog` server ran `dnx Microsoft.AITools.BinlogMcp --yes` with no feed pinned, so resolution depended on whatever ambient NuGet configuration the agent happened to run under. Pin it to the dnceng dotnet-public feed with `--source` so the server resolves deterministically.
Applied to all four manifests that embed the server definition (plugin.json, .claude-plugin/plugin.json, .codex-plugin/.mcp.json, and .claude-plugin/marketplace.json) so the mirrors stay in sync. Also corrects two stale feed names in comments/docs (dotnet-eng, dotnet-tools).
Verified: `skill-validator check` passes (exit 0, no new warnings), and `dnx --yes --source <dotnet-public> Microsoft.AITools.BinlogMcp` starts aitools-binlog-mcp 2.0.1 and completes an MCP initialize handshake.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: eef3a5cc-7c9b-41b2-88e5-fbc7e786f3f2
* Keep the package ID first in the dnx args
Addresses review feedback. Both orderings resolve and start the server correctly (verified), but keeping `Microsoft.AITools.BinlogMcp` immediately after `dnx` matches the ordering already used in this repo and turns the change into a pure insertion of `--source` with no reordering noise.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: eef3a5cc-7c9b-41b2-88e5-fbc7e786f3f2
---------
Co-authored-by: Amaury Levé <amauryleve@microsoft.com>
Copilot-Session: eef3a5cc-7c9b-41b2-88e5-fbc7e786f3f2
`disable-model-invocation: true` drops a skill from the CLI's
`<available_skills>` menu, so the experiment's skilled arm -- which loads
exactly one skill -- cannot reach it either. The gate already reasoned about
this, but `report_uncovered()` skipped any skill that had an eval, so it only
reported the *better* case (no eval, visibly zero evidence) and stayed silent on
the worse one (an eval scoring baseline against baseline and labelling the
result a pass or a fail).
Two evals landed in that blind spot after the reasoning was written down:
tests/dotnet-test/platform-detection (#974) and tests/dotnet-test/filter-syntax
(#976). Both READMEs still described them as deliberately un-evaluated.
- report reference skills that carry a direct eval, naming the spec
- self-test both sides of the trigger via a new silent_case helper
- correct eng/eval-quality/README.md and plugins/dotnet-test/README.md
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3d666b74-a00f-4ed0-a3ce-2d3427cf0d23
* Close the remaining dotnet-test eval follow-ups from #899
Every non-agent dotnet-test eval now clears the 5-trial floor, the two cost
P1s are addressed, and the reference-skill coverage gap is recorded as a
decision instead of a standing warning.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 024b3241-d7af-418a-b9cd-3bb9ee9bf0ee
* Remove the duplicate block that cloned a grade-tests scenario, and gate it
The 'production code available' scenario carried a leftover tail from the
edit that moved the 'production code unavailable' one. YAML keeps the last
duplicate key, so every field of the new scenario was silently overwritten:
it shipped as a byte-identical rerun of its predecessor and never loaded the
dotnet-production-available fixture it was built around.
Parsing the spec and counting stimuli - which is what verified this PR -
returns the intended 5 names either way, so only the parser can see it. The
eval-quality gate now loads specs with a duplicate-key-strict loader
(failing check 9), with a self-test case and the incident recorded.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 024b3241-d7af-418a-b9cd-3bb9ee9bf0ee
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 024b3241-d7af-418a-b9cd-3bb9ee9bf0ee
* Fix migrate-mstest-v1v2-to-v3 skill activation
The frontmatter description said DO NOT USE FOR: ... projects already on
MSTest v3+, which blocked the skill on every scenario where the packages had
already been bumped to 3.x and only the source or settings still needed the
v1/v2-to-v3 fixes (Assert object overloads, DataRow strict typing,
.testsettings -> .runsettings). It also gated the whole skill behind "the user
asks to upgrade MSTest", so a standalone .testsettings conversion never matched.
- Rewrite the description around both entry points (pre-upgrade migration and
post-upgrade breaking-change fixes) and add the concrete trigger keywords
those prompts contain: CS1501/CS1503/CS0121, MSTEST0014, LegacySettings,
DeploymentEnabled, per-test TestTimeout, net5.0. Note that the current runner
is preserved so "migrate to v3 but keep VSTest" isn't poached by
migrate-vstest-to-mtp.
- Narrow migrate-mstest-v3-to-v4, which claimed the generic "tests don't
compile after upgrading MSTest" phrasing and competed for the same prompts.
- Widen the Boundary Gate: a 3.x package version alone no longer ends the
migration when v1/v2-era settings or errors remain, so the skill actually
performs the requested edits instead of reporting "already migrated".
- Add a routing row to the test-migration agent for the same case.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ad6ff32a-d441-4a7b-b474-2bfaee764740
* Clear the eval-quality gate's test-skill findings
The gate reported three classes of debt against the dotnet-test and
dotnet-test-migration plugins. All three are addressed here; the four ERRORS it
also reports are dotnet-maui allowlist lines and are untouched.
Underpowered evals (5). Below five trials the pass gate's sign test cannot reach
p <= 0.05 at any effect size, so these five evals could never return a verdict.
Each is now at or above the floor and its allowlist line is deleted in the same
change, as the ledger's shrink-only rule requires:
- coverage-analysis 3 -> 5: adds a refactoring-safety question (the "is this
safe to change?" use case named in the skill's Purpose but never exercised)
and a branch-vs-line coverage question. Both reuse the existing
partial-coverage fixture.
- find-untested-sources 4 -> 5: adds a mixed C#/TypeScript repository, which
is the only case that exercises the documented engine choice - polyglot
tree-sitter rather than the C#-only Roslyn engine. Composed from the two
existing fixtures.
- generate-testability-wrappers 4 -> 5: adds the ambient-context path (Step 5)
for a project with no DI container, where AsyncLocal<T> and scoped disposal
are the distinguishing content.
- grade-tests 4 -> 5: adds a C# case with the production code present. Every
prior C# scenario hides it, so "Unverified" was never tested as a negative,
and the D band and the swallowed-exception F were never graded at all. New
production-available fixture.
- code-testing-agent 3 -> 6 via defaults.runs=2. Scenarios are preferred over
runs, but each of these drives a full generate-build-test pipeline (npm ci
plus two Vitest runs, pip install plus pytest, a dotnet test build) under a
60m budget, which is the documented case for buying trials with runs.
Orphaned fixtures (5). v3-sealed-timeout, mtp-mstest-sdk9, mtp-mstest-sdk10,
mtp-mstest-hotreload-installed and vstest-mstest are all superseded first-
generation copies: their per-scenario successors differ only in whitespace, a
dropped rollForward, or a package version. Both evals are already well above the
floor, so wiring them up would add no power. Deleted.
Skills with no eval (2 of 4). platform-detection and filter-syntax carry real
checkable rules that nothing measured, and several are counterintuitive enough
that a baseline is likely to get them wrong - global.json test.runner outranking
TestingPlatformDotnetTestSupport on .NET 10+, Microsoft.NET.Test.Sdk not being a
VSTest signal, MTP properties living in Directory.Build.props, xUnit v3 dropping
VSTest --filter while MSTest on MTP keeps it. Both get a 5-scenario eval with
small fixtures and no build step. code-testing-extensions and
test-analysis-extensions are left flagged on purpose: their bodies are tables of
paths to extension files, so a head-to-head eval would score path recall rather
than user value. The content those files hold is already exercised through
code-testing-agent's three-language pipeline.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ad6ff32a-d441-4a7b-b474-2bfaee764740
* Fix the two skill defects behind the v1v2-to-v3 eval losses
The first eval run reached 7W/2T/2L, p=0.090, short of the p<=0.05 gate. Both
losses trace to skill content that actively misled the agent, and the session
transcripts show exactly how.
Loss 1 -- 'Migrate MSTest v1 project with assembly reference', skilled scored
0.00 against a 4.17 baseline. The transcript shows the skill loading correctly
and the agent then replying, in full: 'To give you specific migration steps, I
need to see your project file. Could you share the path to your .csproj?' The
project was already in the working directory. Cause: the Inputs table marked
'Project or solution path' as Required=Yes, which reads as a precondition the
agent must obtain before doing anything. This is the worst kind of failure for a
real user - they describe their project in prose and get a question back instead
of an answer. Path is now optional and discovered by globbing, Step 1 leads with
locating the project, and a note forbids opening with a request for the path.
The same Required=Yes trap was present in migrate-mstest-v3-to-v4 and
migrate-vstest-to-mtp, so both are corrected too.
Loss 2 -- 'Fix DataRow type mismatch errors', skilled 3.96 against a 5.00
baseline. The skill's breaking-change table said the 16-argument DataRow cap was
'fixed in later v3 versions' and suggested 'refactor test / wrap extra params in
array'. On a project already at MSTest 3.8, the agent concluded the valid
17-argument row exceeded the limit and rewrote it - first as new object[] { 17 },
which failed, then second-guessing itself mid-run ('let me check if the latest
3.x actually fixed the 16-arg limit'), finally settling on a (object)17 cast.
Churn plus wasted turns on code that was already correct.
The vague wording was the problem, so it is replaced with the fact: the cap was
introduced in 3.0.1 and removed again in 3.0.3 (microsoft/testfx#1554 and the
maintainer's 'please feel free to update to 3.0.3'). On 3.0.3+ a longer row is
valid and must be left alone. A general guideline is added alongside it -
confirm the diagnostic before editing, because rewriting valid code to dodge a
limit the project is not subject to is a defect rather than caution.
Both fixes are about what the skill tells a real user, not about the graders;
no eval prompt, fixture, or grader is touched. Skill grows ~480 tokens and stays
in the 'standard' tier, below the 5,000-token warning threshold.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ad6ff32a-d441-4a7b-b474-2bfaee764740
* Correct the Assert/DataRow facts and make the fixtures reproduce them
The two eval runs on this PR compared byte-identical skill content: run 1
(f9dc25d) and run 2 (2fc8ab8) differ in tests/ and eng/ only, git diff on
plugins/ between them is empty, and the run-2 artifact confirms the loaded
SKILL.md lacks the fix from 6b11ad9. So 7W/2T/2L -> 4W/5T/2L is judge noise, not
a regression. The v1-assembly-ref scenario scored 0.00 in BOTH runs with the same
'I need to see your project file' reply, which 6b11ad9 addresses.
Investigating the remaining scenarios against a real MSTest 3.8 project turned up
something worse than a scoring problem: the skill was teaching two things that
are not true, and one eval fixture could not reproduce the bug it was named for.
1. Assert. The skill said the removal of Assert.AreEqual(object, object) causes
'compile error on untyped assertions'. It does not. MSTest v3 keeps
AreEqual<T>(T?, T?), so two object-typed arguments infer T = object and
compile untouched; verified by building the shipped fixture, which passes 3/3
tests unmodified. The break happens only where T cannot be inferred, and the
real diagnostics are CS0411 and CS1503 - not the CS1501/CS0121 the earlier
description claimed. Following the old text, an agent rewrites assertions that
were already correct, which is the same over-application defect as the DataRow
one. Table, Step 5 and the description now state the real trigger and codes,
and say to fix only the call sites the compiler rejects.
2. DataRow. The skill implied compile errors. Verified: a mismatched row builds
with analyzer warning MSTEST0014 and fails at run time with 'Test data doesn't
match method parameters'. Widening (int -> long) still binds; narrowing does
not. Stated explicitly, because a green build is exactly what misleads here.
3. Fixtures. fix-assert-.../ComparisonTests.cs compiled and passed as shipped, so
its scenario could never discriminate - it scored baseline 5.00/5.00 in both
runs. It now uses two unrelated interface-typed views of one instance, which
genuinely fails with CS0411 on all three assertions and passes 4/4 once the
<object> argument is added. It also gains two already-valid typed assertions
that must be left alone; widening them still compiles, so only judgement
prevents it, and two graders now check that.
v2-nuget/UserServiceTests.cs and v2-complex/InventoryServiceTests.cs had the
same problem: their graders demanded Assert.AreEqual<object> on assertions
that never needed it, which now directly contradicts the corrected skill.
Both fixtures were rebuilt the same way and verified in three states - build
clean on MSTest 2.2.10 as shipped, fail with CS0411 after the v3 bump, pass
(5/5 and 7/7) once migrated.
Prompts for the two affected scenarios now describe the real symptoms (CS0411,
and 'builds but fails at run time') instead of the invented CS1501 and 'no longer
compile'. Every claim above was verified by building and running against MSTest
3.8 and 2.2.10 rather than inferred.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ad6ff32a-d441-4a7b-b474-2bfaee764740
* Stop the complex-migration run from scaffolding a substitute project
Last eval: 6W/4T/1L, p=0.063. The three scenarios fixed in 6b11ad9 and 22103d2
all flipped to wins; one loss remains and the transcript shows a distinct bug.
The agent globbed correctly and got back ./TestProject.csproj,
./InventoryServiceTests.cs and ./local.testsettings. It then rebuilt those into
absolute paths under the skill's own base directory, all three reads failed with
'Path does not exist', it globbed that directory, found only SKILL.md, and
concluded 'There's no actual project on disk'. It then scaffolded a substitute
project from the prose description - fewer tests, no 17-parameter row, and a
self-introduced bug it had to debug. Judge scored it 0.23 against a 3.07
baseline.
Step 1 told it to glob but not what to do with the answer, so add that: open
paths exactly as the search returned them, treat a failed read of a just-found
file as a wrong constructed path, never conclude the project is missing while a
search is still listing it, and never scaffold a replacement. This matters
outside the eval too - a skill loaded from a plugin directory always has a base
directory that is not the user's repo.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ad6ff32a-d441-4a7b-b474-2bfaee764740
* Address Copilot review: fixture versions and an over-broad grader
Three findings from the latest Copilot review, all valid.
Two package versions in the platform-detection fixtures were invented rather
than copied from the repo: TUnit 0.6.0 and xunit.v3 1.0.0. Both existed nowhere
else in tests/ - the canonical versions are 1.45.8 and 1.0.1 - so they risked a
restore failure the moment anything builds these fixtures. Aligned to the
versions the rest of the repo uses.
The third is a grader I added in the merge commit. output-not-matches:
ThreadStatic forbids the substring anywhere in the response, so it would also
fail a correct answer that warns the user against [ThreadStatic] - which is
exactly the answer the rubric asks for, and the likeliest way a good response
mentions it. No regex separates 'recommends the attribute' from 'warns against
it' reliably, so the grader is removed and the rubric line kept: a judge can
draw that distinction, a substring match cannot.
That leaves main's original grader set for this scenario untouched, plus the one
rubric item this branch contributed.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ad6ff32a-d441-4a7b-b474-2bfaee764740
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ad6ff32a-d441-4a7b-b474-2bfaee764740
Removes the remaining three mcp-csharp-* skills from the dotnet-ai plugin.
Following the cut of mcp-csharp-debug (#938), this retires the rest of the
family based on the cross-family skills-eval (#889) and the 30-day telemetry
review (https://gist.github.com/AbhitejJohn/db57f38051ebf4e7de5556a4bd98ec15):
- mcp-csharp-test: STRENGTHEN 0/5 families; 2 real users, 0 with >=5 invocations.
- mcp-csharp-publish: STRENGTHEN 0/5 families; 0 real users.
- mcp-csharp-create: wins only 2/5 families (Sonnet, Haiku) and REGRESSES both
frontier models; impact 0.40 at avgN 2.6 (directional only); absent from the
telemetry report. Validating it would require a full 5-family eval sweep with
no evidence of real demand to justify the investment.
Changes:
- Delete plugins/dotnet-ai/skills/mcp-csharp-{create,test,publish}/ (SKILL.md + references)
- Delete tests/dotnet-ai/mcp-csharp-{create,test,publish}/ (eval.yaml)
- Remove the dotnet-ai CODEOWNERS block for the three skills and their tests
- Repoint the skill-validator doc example to a surviving skill (technology-selection)
dotnet-ai retains technology-selection. plugin.json auto-discovers ./skills/,
so no manifest change is needed. Verified no remaining mcp-csharp-* references
repo-wide (git grep).
Refs dotnet/skills#889. Supersedes #939, #940, #941.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6a8ce111-58e5-48a7-9f7f-8de972bdc693
* Fix dotnet-test findings from the refreshed cross-family eval (#899)
Every change below is driven by judge evidence from the losing trials of the
refreshed 5-family dotnet-test matrix (runs 30108473397 + recovery runs), not by
style preference.
Eval measurement fix — the "discovery" P2s were an artifact:
- assertion-quality, test-gap-analysis, test-smell-detection, and test-tagging
each have a decline stimulus with `constraints.reject_skills: ["*"]`, so the
skill cannot activate there by construction. Without `expect_activation:
false` the adapter counted those dormant runs as missed activations, which is
exactly the 75-88% invocation rates reported in the scorecard. Annotating them
(the convention already used by agent.test-quality-auditor) removes the false
signal; the non-activations were the only ones observed for these skills.
Skill fixes:
- test-gap-analysis: baselines won by actually running the suite while the skill
reasoned statically and reported survivors that the tests in fact kill. Added
Step 4b: confirm every reported survivor by applying it, re-running the
covering tests, and reverting; fall back to reasoning only when the suite
cannot run, labelled unverified. Calibrated severity down for strong suites.
- test-anti-patterns: baselines won on depth, not polish. Added a depth bar —
account for every test in scope, give exact expected values in fixes, name the
adjacent error-path/boundary gaps, and keep counts consistent. Trimmed three
pitfall rows that duplicated the calibration step so the skill stays under the
profiler's "comprehensive" threshold.
- detect-static-dependencies: losses were all counting accuracy. One
authoritative total (no findings parked outside it), classify by the resource
touched rather than by the `static` keyword, exclude pure helpers such as
Path.Combine from the needs-wrapping total, require file:line, and add the
missing randomness/culture/serialization categories.
- test-smell-detection: the calibration rule told models to downgrade Sleepy
Test for integration tests, which is what lost both losing scenarios. Fixed
sleeps now stay High in any category; Mystery Guest and Eager Test still
downgrade.
- crap-score: losses came from estimating coverage after collection failed.
Added the dotnet-coverage/ReportGenerator recovery path and a hard rule never
to publish a CRAP score built on assumed coverage.
- coverage-analysis: answer the asked question first, reconcile every number
against the script output, and list every below-threshold member instead of
declaring one method the entire gap.
- migrate-static-to-wrapper: migrate exactly what was requested (no adjacent
DateTime.Now rewrites, respect intentional-use comments) and never report
"build succeeded" when the build or restore failed.
- code-testing-agent: quote each requirement verbatim in the evidence table so
multi-condition requirements map to a test that covers the whole combination,
and cite a clean run rather than a coverage attempt that exited non-zero.
Validation: skill-validator check passes (20 skills, 10 agents); markdownlint
clean; eval specs parse and the adapter now reports all four decline stimuli as
expect-dormant.
Refs #899
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1947263a-0ef9-47bd-ac53-5af5afa3ddaa
* Strengthen underpowered dotnet-test evals from the PR 945 eval run
The PR eval reported 5 of 10 skills as "no credible improvement". Reproducing
the gate arithmetic from the artifacts shows the dominant cause is statistical
power, not skill quality.
The gate is `mean > 0 AND ci_low > 0` with a t-based CI over per-trial scores,
which reduces to `sqrt(n) * (mean/sd) > t(n-1)`. The required mean/sd ratio is
brutal at small n:
n=2 -> 8.98 n=3 -> 2.48 n=4 -> 1.59
n=5 -> 1.24 n=6 -> 1.05 n=8 -> 0.84
Recomputing each reported CI from the per-trial scores reproduces the published
numbers exactly, which confirms the mechanism:
crap-score [0.4,1.0,0.4] n=3 CI [-0.261, 1.461]
migrate-static-to-wrapper [1.0,0.4,0.4] n=3 CI [-0.261, 1.461]
test-gap-analysis [0.4,0,0.4,0.4] n=4 CI [-0.018, 0.618]
test-anti-patterns [0.4,0,0.4,0,0,0.4] n=6 CI [-0.030, 0.430]
code-testing-agent [0,0.4] n=2 CI [-2.341, 2.741]
crap-score and migrate-static-to-wrapper won 100% of their trials (3W/0T/0L)
and still failed: at n=3 nothing short of three identically-sized wins can
clear the gate. That is a property of a thin eval, not of the skill.
Scenario counts are raised with discriminating cases, four of them by wiring up
fixtures that were already committed but had no stimulus referencing them:
- test-gap-analysis 4 -> 6, using the orphaned `report-quality` fixture (trivial
auto-properties and an auto-generated .g.cs to skip, private helpers reachable
only through the public API, and a deliberately weak Assert.IsTrue that cannot
kill arithmetic mutations) and the orphaned `rust-error-propagation` fixture
(an untested `?` propagation path and an untested `<=` boundary).
- test-anti-patterns 6 -> 8, using the orphaned `pytest-mixed` fixture (which
also checks the calibration rule that pytest's bare `assert` must not be
flagged) and the orphaned `assertion-problems` fixture (which separates
Critical false-confidence assertions from a Low-severity message nit).
- crap-score 3 -> 6, with a new `refactor-required` fixture whose numbers are
self-consistent: ApplySurcharges has complexity 13 behind a stale
"Complexity: 4" comment (CRAP 28.4, needs 77.2% coverage), ClassifyAccount has
complexity 17 so coverage alone can never reach CRAP < 15, and RoundToCurrency
is 100% covered so its CRAP equals its complexity exactly.
- migrate-static-to-wrapper 3 -> 5, adding a DateTimeKind-preservation scenario
over the existing fixture and a new `static-helper` fixture where a static
class must gain an ambient TimeProvider seam without breaking its callers.
- code-testing-agent 2 -> 3, with a compact C# fixture that must extend an
existing suite to the untested method only. This eval stays the weakest: each
scenario is expensive, so raising `runs` is a better lever than adding more
heavyweight scenarios.
Verification:
- every eval spec parses and all 254 fixture references resolve
- the three new fixtures compile; the shipping-quotes fixture restores, builds
and its three seed tests pass under `dotnet test` in a clean workspace
- skill-validator check passes (20 skills, 10 agents)
- markdownlint clean
Refs #899
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1947263a-0ef9-47bd-ac53-5af5afa3ddaa
* Address review feedback on fixture and counting wording
- BillableWeightTests: the ZeroOrNegative test only asserted the zero case, so
its name overstated what it covered. Made it data-driven over 0 and -1 so the
name matches the assertions. This matters more than usual here: the file is
the seed suite for a test-quality eval, and a misleading test name is exactly
what these skills are supposed to flag.
- detect-static-dependencies: the Step 3 lead-in said to count each "static call
pattern", which contradicted the rule immediately below it that instance
members reaching the same untestable resource must also be counted. Reworded
to "call site" and made the instance-member inclusion explicit.
Verified: the fixture restores, builds and now passes 4 tests (was 3);
skill-validator check passes; markdownlint clean.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1947263a-0ef9-47bd-ac53-5af5afa3ddaa
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1947263a-0ef9-47bd-ac53-5af5afa3ddaa
The Copilot reviewer flagged this twice and was right both times:
label.SetAppTheme<FileImageSource>(Image.SourceProperty,
"logo_light.png", "logo_dark.png");
Two defects in one line:
- Image.SourceProperty is applied to a *Label* instance. The BindableProperty
must belong to the object it is set on.
- T is FileImageSource while Image.SourceProperty is ImageSource-typed, and the
arguments are bare strings.
Worse, it COMPILES - verified. SetAppTheme<T> takes a BindableObject and a
BindableProperty with no compile-time check that the property belongs to the
object, so a model copying this produces code that builds and silently does
nothing useful. That is precisely the failure class this PR exists to fix.
I introduced this. The earlier compile check used `var img = new Image();
img.SetAppTheme<FileImageSource>(...)` - I verified a variant rather than the
exact text I shipped, and the `label.` receiver slipped through.
Corrected to the right owner and type, with ImageSource.FromFile:
var image = new Image();
image.SetAppTheme<ImageSource>(Image.SourceProperty,
ImageSource.FromFile("logo_light.png"),
ImageSource.FromFile("logo_dark.png"));
This time the exact block was extracted programmatically from SKILL.md and
compiled verbatim in a net11.0-maccatalyst app rather than retyped. Also swept
the other seven skills and the theming reference file for the same
wrong-owner pattern - none found.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 58513e0c-4086-4112-9ea1-95dbd8e8e469
Latest gate run: maui-theming PASSED (+40%, 4/0/0) - the MergedDictionaries
fix worked. But maui-dependency-injection and maui-shell-navigation flipped to
❌, both at 3W/1T/0L with ZERO losses: at 4 trials a single tie puts the CI
lower bound at -1.8%, so any tie fails. Rather than argue noise, I diagnosed
the two specific tying scenarios.
maui-shell-navigation, "Set up Shell navigation with tabs and flyout":
Both answers satisfied all four rubric items and were near-identical - except
baseline set an explicit Route= on each ShellContent and the skill never
mentioned Route at all (zero occurrences in the file).
That omission is a real defect. Routing.cs:134 generates
"D_FAULT_{TypeName}{++s_routeCount}" when Route is unset. Verified at runtime
in a net11.0-maccatalyst app with three ShellContent elements:
ShellContent Title='' Route='D_FAULT_ShellContent2'
ShellContent Title='Active' Route='D_FAULT_ShellContent5'
ShellContent Title='Archived' Route='archived' <- explicit
The numbers are not sequential (2 and 5) because the counter is shared across
all Shell element types, so they shift when pages are reordered or added. No
stable absolute route ("//dashboard") or deep link can target them. Added the
rule to the workflow plus the explanation, and set Route= on every
ShellContent in the AppShell example.
maui-dependency-injection, "Shell navigation auto-resolves DI-registered
pages": baseline covered IQueryAttributable and the skill did not (the skill
covered ActivatorUtilities and ContentTemplate, which baseline lacked - so it
was trading one gap for another). Added a "Passing parameters to a DI-resolved
ViewModel" section showing the actual combination: constructor injection for
dependencies, IQueryAttributable for navigation parameters, and the fact that
Shell applies query attributes to the BindingContext so no page wiring is
needed.
Measured (n=4/scenario, 32 trials across both skills, executor
claude-opus-4.6, judge gpt-5.5):
32W / 0T / 0L mean +0.55 [95% CI +0.45, +0.65] 100% win rate
Both previously-tying scenarios now win 4/4, and neither skill has a single
tie left. Per-skill mean is +0.55 for each.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 58513e0c-4086-4112-9ea1-95dbd8e8e469
The gate had maui-theming at 3W/1T/0L, +30% [-1.8%, +61.8%] - one tie away
from passing, on "Avoid common theming mistakes". Comparing the trajectories,
baseline and skilled were near-identical and both satisfied all four rubric
items, so there was nothing to win on... except that BOTH emitted the same
real bug:
var merged = Application.Current.Resources.MergedDictionaries;
merged.Clear();
merged.Add(new DarkTheme());
The default MAUI template merges Resources/Styles/Colors.xaml and Styles.xaml
into Application.Resources. Clear() removes those too, so every implicit style,
brush and colour in the app silently disappears.
Verified at runtime in a net11.0-maccatalyst app built from the default
template:
before Clear(): merged=2 'Primary' resolves=True value=[Color Red=0.317...]
after Clear(): merged=1 'Primary' resolves=False value=<GONE>
The skill previously had a weak comment acknowledging this ("assumes theme
dictionaries are the only merged dictionaries... move them to
Application.Resources directly instead"), which is both easy to miss and poor
advice. Replaced across SKILL.md (both occurrences) and theming-api.md with a
tracked-reference swap that removes only the previous theme:
static ResourceDictionary? _currentTheme;
if (_currentTheme is not null) merged.Remove(_currentTheme);
merged.Add(theme);
_currentTheme = theme;
Also verified at runtime: app styles survive across two consecutive theme
swaps and the dictionary count stays flat (2 -> 3 -> 3, 'Primary' OK
throughout), so it neither leaks nor unstyles the app.
Measured (n=4/scenario, 16 trials, executor claude-opus-4.6, judge gpt-5.5):
before 14W/1T/1L +0.33 [+0.21, +0.44] 88% win
after 15W/1T/0L +0.38 [+0.32, +0.43] 94% win
"Avoid common theming mistakes" - the scenario that tied on the gate - now
wins 4/4, and the remaining single tie is on a different scenario. Tightest
confidence interval of any arm measured in this PR.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 58513e0c-4086-4112-9ea1-95dbd8e8e469
maui-theming was the last skill not passing the gate: 1W/3T/0L, +10%
[-21.8%, +41.8%]. No losses - three ties. I had put that down to a ceiling
effect (its baseline Quality is 5.0/5) and suggested accepting it. That was
wrong; the artifacts show a real, fixable gap.
What the eval artifacts actually show:
- Skilled answers were already LONGER than baseline on all three tied
scenarios (2466 vs 2100, 2195 vs 1972, 1797 vs 1590 chars), so this was not
the terseness problem fixed in the previous commit.
- On "Detect and respond to system theme changes" the skilled answer covered
strictly more than baseline (adding UiMode and Preferences) and still tied.
More words were not converting into wins.
- On "Add light/dark mode support using AppThemeBinding" BOTH answers scattered
hardcoded hex literals across every element - while that scenario's rubric
explicitly says "Does not suggest hardcoded color values as a final
solution". Baseline at least mentioned moving them to App.xaml; the skill
never demonstrated it.
So the differentiator was structural, not volume. Added a "Define the palette
once - don't scatter literals" section that makes the recommended final shape a
single App.xaml source of truth: a raw Color palette plus implicit Styles
(TargetType="ContentPage" ApplyToDerivedTypes="True", TargetType="Label") whose
setters carry the AppThemeBinding. Pages then need no theming markup at all.
Inline AppThemeBinding is reframed as the one-off case, and even then pointed at
StaticResource keys rather than literal hex.
The pattern is compile-verified in a net11.0-maccatalyst app (implicit styles
with nested AppThemeBinding + StaticResource in a merged ResourceDictionary).
Measured (n=4/scenario, 16 trials, executor claude-opus-4.6, judge gpt-5.5):
before this commit 10W/5T/1L +0.22 [+0.09, +0.36] 63% win
after 14W/1T/1L +0.33 [+0.21, +0.44] 88% win
Ties collapse from 5 to 1, and all four scenarios now win predominantly -
including "Add light/dark mode support using AppThemeBinding", which was the
tie that motivated the change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 58513e0c-4086-4112-9ea1-95dbd8e8e469
* adding scaffolding skill and test
* Update tests/dotnet-data/scaffold-generate-aspnet/fixtures/blazor-crud/Components/Routes.razor
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* updating pr based on copilot feedback
* updating pr based on copilot feedback
* Update tests/dotnet-data/scaffold-generate-aspnet/fixtures/blazor-crud/Components/Layout/ReconnectModal.razor.css
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update tests/dotnet-data/scaffold-generate-aspnet/fixtures/blazor-crud/Components/Layout/ReconnectModal.razor.css
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update tests/dotnet-data/scaffold-generate-aspnet/fixtures/blazor-crud/Components/Layout/ReconnectModal.razor.css
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update plugins/dotnet-data/skills/scaffold-generate-aspnet/SKILL.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update tests/dotnet-data/scaffold-generate-aspnet/fixtures/minimal-api/Program.cs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fixing warnings in test projects
* fixing ci build issues
* fixing eval.yaml
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Fix unresolved Copilot review comments on PR #329
- Add CODEOWNERS entries for scaffold-generate-aspnet skill and tests
- Fix navbar toggler aria-controls/id mismatch in razor-pages-crud fixture
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* updating code owners for scaffold-generate-aspnet
* Update tests/dotnet-data/scaffold-generate-aspnet/fixtures/blazor-crud/Components/Pages/Error.razor
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update tests/dotnet-data/scaffold-generate-aspnet/eval.yaml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update tests/dotnet-data/scaffold-generate-aspnet/fixtures/blazor-crud/Components/Pages/Error.razor
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update plugins/dotnet-data/skills/scaffold-generate-aspnet/SKILL.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update tests/dotnet-data/scaffold-generate-aspnet/eval.yaml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update tests/dotnet-data/scaffold-generate-aspnet/eval.yaml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Apply suggestion from @webreidi
* Apply suggestion from @webreidi
* Apply suggestion from @webreidi
* Add Blazor CRUD application structure and initial components
- Created Home, NotFound, and Weather pages with basic layout and functionality.
- Implemented routing in the Router component.
- Added models for Department and Employee with data annotations for validation.
- Set up Program.cs for Blazor server-side rendering and configured HTTP request pipeline.
- Included launch settings and appsettings for development environment.
- Established CSS styles for the application and added favicon.
- Created minimal API with TodoItem model and basic weather forecast endpoint.
- Developed Razor Pages CRUD structure with models for Category and Product.
- Implemented error handling and privacy policy pages in Razor Pages.
* Remove SKILL.md documentation for ASP.NET Core scaffolding
* Refactor eval.yaml to enhance scaffolding scenarios and improve grading configurations
* Add scaffolding for Minimal API and MVC with existing DbContext, including models, DbContexts, and configuration files
* Refine evaluation criteria for create-datadriven-aspnetcore skill by updating file paths and adding checks for Razor Pages and DbContext presence
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Dan Moseley <danmose@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Abhitej John <aajohn@microsoft.com>
Co-authored-by: Wendy Breiding <55603905+webreidi@users.noreply.github.com>
Co-authored-by: Wendy Breiding (She/Her) <webreidi@microsoft.com>
Follows @AbhitejJohn's review and the cross-family run
(actions/runs/30313685452: 23/35 pass, no regressions).
Guard the P0 regression (the review's highest-value item - "nothing stops the
broken markup from creeping back"):
- Added an `ItemSizingStrategy placement for uniform items` stimulus to
tests/dotnet-maui/maui-collectionview/eval.yaml with an `output-not-matches`
grader on `<(Linear|Grid)ItemsLayout[^>]*ItemSizingStrategy`. Verified the
pattern catches both broken forms (LinearItemsLayout and GridItemsLayout) and
passes the corrected form, so the exact MAUIX2002 build break this PR fixed
is now regression-guarded. A regex guard rather than a real compile: the eval
runner has no MAUI workload, and this catches the specific defect
deterministically.
- The new scenario initially LOST (-0.4, 0, -0.4): the guard passed in both
arms, but baseline gave richer sizing guidance. Enriched the Performance Tips
entry with the default (MeasureAllItems), a full XAML example, and the four
cases where MeasureFirstItem is wrong (variable heights, DataTemplateSelector,
atypical first item, size-depends-on-late-data). The scenario now wins 3/3.
maui-collectionview, n=3, 15 trials: 12W/2T/1L, mean +0.29 [95% CI +0.16,
+0.42] - up from +0.16 [-0.00, +0.32] before the enrichment.
Minor items from the review:
- Cross-skill tension: the CollectionView "Basic Setup" page sets BindingContext
by instantiating the ViewModel inline, which reads oddly next to
maui-dependency-injection's constructor-injection guidance. Added the
one-liner pointing DI apps at constructor injection.
- Version-pinned the SDK-band-sensitive claims: the XC0022-XC0025 table now
states it is verified against .NET 10/11 with a note to re-check
BuildException.cs on newer SDKs; the
MauiEnableXamlCBindingWithSourceCompilation default is pinned to .NET 10/11;
the ListView [Obsolete] note now says explicitly that it does not apply to
.NET 9 and earlier, so a model checks the target framework first.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 58513e0c-4086-4112-9ea1-95dbd8e8e469
The maintainer-triggered /evaluate run (actions/runs/30246707498) passed 4 of 7
skills. maui-collectionview - the P0 this PR exists to fix - passed 4W/0T/0L,
alongside maui-safe-area (+100%), maui-app-lifecycle (+55%) and
maui-data-binding (+40%). This commit addresses the other three. Only those
three files change; the four passing skills are untouched.
Diagnosis from the eval artifacts: in every failing scenario the skilled answer
was SHORTER than baseline (shell 1804 vs 2433 chars, theming 1347 vs 1541).
The scope-control wording added earlier in this PR - needed to stop
maui-collectionview over-applying - had over-corrected the other skills into
answering too narrowly. Baseline kept winning by including the follow-through
code that makes a fix actionable.
maui-shell-navigation (was 2W/2T/0L, +35% not credible):
- Added "answer narrowly, but completely": show AppShell markup *and*
RegisterRoute, GoToAsync *and* the receiving IQueryAttributable, and present
both parameter-passing approaches with when each fits.
- Local n=4: 16W/0T/0L, mean +0.66.
maui-dependency-injection (was 1W/2T/1L, +0.0%):
Two claims the judge repeatedly called "misleading"/"overconfident", both
verified wrong against source:
- "Unregistered page silently skips injection - dependencies are silently null,
no exception is thrown". Routed pages go through
ActivatorUtilities.GetServiceOrCreateInstance (Routing.cs:336), which injects
registered dependencies and THROWS when one cannot be resolved.
ContentTemplate pages go through Activator.CreateInstance
(ElementTemplate.cs:33) - no DI at all, and a MissingMethodException rather
than null dependencies. Section rewritten around that real distinction.
- "must be AddTransient" softened to "prefer", with the case where Singleton is
defensible.
- Also added the alternatives the judge rewarded baseline for: explicit
IServiceScopeFactory.CreateScope() and the factory pattern.
- Local n=4: 15W/1T/0L, mean +0.49 (the 3 consistent losses are gone).
maui-theming (was 3W/0T/1L, +20% not credible):
- All losses were the same complaint: baseline also showed how to swap merged
dictionaries and hook RequestedThemeChanged. An instruction to "be complete"
did not stick, so the code now lives inline in the DynamicResource pitfall
where the model actually reads it.
- Scoped the Android UiMode rule to runtime/system theme switching - it was
being bolted onto narrow AppThemeBinding questions and judged "ancillary and
slightly overstated as required".
- Expanded the C# helpers so both SetAppThemeColor and generic SetAppTheme<T>
are shown; all three overloads compile-verified in a net11.0-maccatalyst app.
- Local n=4: 10W/5T/1L, mean +0.22 [95% CI +0.09, +0.36] - CI now above zero.
All three were measured with the same harness as the official run (vally,
executor claude-opus-4.6, judge gpt-5.5), at n=4 per scenario rather than the
official n=1, since single-trial cells cannot separate signal from noise -
maui-shell-navigation originally "failed" with zero losses purely on CI width.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 58513e0c-4086-4112-9ea1-95dbd8e8e469
The data-binding "Common Pitfalls" row said "enable XC0022 as error" while the
"Enforce binding warnings as errors" section above recommends XC0022;XC0025
paired with MauiEnableXamlCBindingWithSourceCompilation - two different
prescriptions for the same task. The pitfall row now points at that section as
the single source of truth instead of restating a partial version.
Copilot's four earlier comments on this PR (theming escaped pipe,
ObservableCollection contradiction, ShellContent eager form, QueryProperty
example) were already addressed in previous commits; the ShellContent and
QueryProperty findings matched gpt-5.6-sol's and gpt-5.5's independently.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 58513e0c-4086-4112-9ea1-95dbd8e8e469
Round-3 review by claude-opus-5, claude-opus-4.8, gpt-5.6-sol, gpt-5.5:
3 SHIP, 1 FIX. Both of gpt-5.6-sol's blocking items are fixed here, along
with the non-blocking findings from opus-5 and gpt-5.5.
gpt-5.6-sol (blocking):
- The Shell rules table used an undeclared `pages:` prefix, and its "Not this"
cell claimed `<ShellContent Content="..."/>` constructs pages eagerly.
`ShellContent` is `[ContentProperty(nameof(Content))]` with `Content` typed
`object` (ShellContent.cs:15,45), so the eager form is nested markup, not a
string attribute. Row rewritten with a declared `xmlns:views` and the real
eager form `<ShellContent><views:MyPage /></ShellContent>`.
- QueryProperty prose said "ViewModel" while the example was still a
ContentPage. Also flagged independently by gpt-5.5 (consensus).
opus-5 (non-blocking, both real):
- The rules-table cell I added in round 2 contained `</...>` — not valid XML.
A model lifting that cell verbatim emits an unparseable .csproj. Reworded to
prose so there is no truncated tag to copy.
- The QueryProperty ViewModel example used a plain auto-property. On a page
that was harmless (code-behind reads it); on a ViewModel it is the binding
target, and Shell applies query attributes *after* BindingContext is set, so
a non-notifying property leaves the binding on its initial value. Switched
to [ObservableProperty] and explained why.
- The "complete, copy-pasteable page" never set BindingContext, so pasting it
renders blank — the exact failure maui-data-binding lists first. Added it.
Both changed snippets were compile-verified in a net11.0-maccatalyst app
(flagship CollectionView page with root x:DataType + BindingContext, and the
[QueryProperty] + [ObservableProperty] ViewModel with CommunityToolkit.Mvvm).
Measured (maui-collectionview, paired base + current run per round, same
session, executor claude-opus-4.6, judge gpt-5.5, n=3/stimulus, 12 trials/arm):
round 3 base +0.00 this PR +0.23 delta +0.23
round 4 base -0.07 this PR +0.17 delta +0.24
round 5 base +0.00 this PR +0.13 delta +0.13
Round 5 is this commit. The delta is smaller than rounds 3-4 but the 95% CIs
overlap heavily at n=12 (base CI +/-0.24), so this is not a demonstrated
regression. The changes in this commit are correctness fixes - invalid XML,
a silently-broken binding, and an inaccurate Shell claim - which are worth
making regardless of a benchmark that cannot resolve a 0.1 difference. A
higher-n run follows to tighten the estimate.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 58513e0c-4086-4112-9ea1-95dbd8e8e469
Round-2 review by claude-opus-5, claude-opus-4.8, gpt-5.6-sol, gpt-5.5:
2 SHIP, 2 FIX. All blocking and consensus findings addressed.
Consensus finding (opus-5 + gpt-5.6-sol) — the flagship "complete,
copy-pasteable page" declared xmlns:models but had no root x:DataType, so its
outer ItemsSource binding was an uncompiled reflection binding. That undercut
the skill's own compiled-bindings guidance and left rubric credit unclaimed on
collectionview stimulus 1. Added xmlns:vm + root x:DataType.
opus-5 blocking claim — <WarningsAsErrors>XC0022;XC0025</WarningsAsErrors>
(pre-existing, line 110 on main) was said to break any build using Source= or
RelativeSource bindings, because XC0025 means "binding not compiled due to an
explicit Source" and MauiEnableXamlCBindingWithSourceCompilation defaults true
only under AOT/full-trim.
Tested empirically: a net11.0-maccatalyst app with that exact
WarningsAsErrors value plus both a Source={x:Reference} binding and a
RelativeSource binding BUILDS CLEAN. The XC diagnostics are not emitted on
the current SourceGen path (Descriptors.cs has XC0022/0023/0025 commented
out), so the claim does not reproduce on current MAUI.
It can still bite on the older XamlC path, and the reasoning is sound, so
the guidance now pairs the switch with the setting and states the caveat
rather than changing the rubric-rewarded value.
Other round-2 fixes:
- "A binding falls back to reflection (XC0022/XC0023) | Fix the path" still
carried pre-correction semantics; neither code concerns the binding path
(that is XC0045). Reworded. (opus-5)
- "Everything below is a suggestion the agent may decline" literally demoted
the Key Rules and the Validation checklist. Scoped it to the API reference
sections and promoted x:DataType to a fourth non-negotiable. (opus-5)
- AddScoped lifetime table row and checklist still implied a manual scope is
the only scope; both now reflect window lifetime. (gpt-5.6-sol)
- Shell "Option 2: QueryProperty — apply directly on the page class"
contradicted the new rule table and the eval rubric ("on the ViewModel, not
the page"). (opus-5)
Copilot review comment — "Always use ObservableCollection<T>, not List<T>.
Swapping a List forces a full re-render" contradicted this PR's own
"use ObservableCollection when the list mutates" rule, and the claim was
imprecise: replacing ItemsSource re-renders regardless of collection type.
Rewritten around INotifyCollectionChanged and in-place mutation.
Not changed: gpt-5.6-sol and opus-5 both suggested editing
tests/dotnet-maui/*/eval.yaml rubrics (safe-area "replaced by", DI "behaves
like Singleton") to match the corrected facts. Changing the benchmark in the
same PR that changes the skill would invalidate the comparison, so these are
left for maintainers and called out in the PR instead.
Measured (maui-collectionview, executor claude-opus-4.6, judge gpt-5.5,
n=3/stimulus, 12 trials/arm, base re-run in the same session each time):
round 3 base +0.00 this PR +0.23 delta +0.23
round 4 base -0.07 this PR +0.17 delta +0.24
Absolute numbers move with run-to-run noise (three independent base runs gave
-0.13, +0.00, -0.07), but the advantage over base is stable at ~+0.24 mean and
~+25pp win rate across every paired run.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 58513e0c-4086-4112-9ea1-95dbd8e8e469
Reviewed by claude-opus-5, claude-opus-4.8, gpt-5.6-sol and gpt-5.5.
Verdicts were split (2 SHIP / 1 FIX / 1 NEEDS_CHANGES), so the blocking and
consensus findings are addressed here. Every disputed factual claim was
re-verified against the dotnet/maui source before changing anything.
Factual corrections (reviewers were right, I was wrong):
- ListView IS marked [Obsolete] as of .NET 10 ("ListView is deprecated. Please
use CollectionView instead.", ListView.cs:19). Verified it is present on
main, net10.0 and release/10.0.1xx-sr7, and absent on release/9.0.1xx. The
previous "it is not removed or obsolete" wording was wrong. (gpt-5.6-sol)
- MAUI DOES create one IServiceScope per window
(MauiContextExtensions.MakeWindowScope, called from the iOS/Android/Windows/
Tizen ApplicationExtensions), so "a Scoped registration resolves from the
root scope and behaves like a Singleton" was over-broad. Reworded to keep the
rubric-rewarded "no built-in request scope like ASP.NET Core" phrasing while
stating the window-scope reality. (opus-5, gpt-5.6-sol)
- ItemTapped -> SelectionChanged is not behaviour-preserving (SelectionChanged
only fires when the selection changes), and RowHeight -> MeasureFirstItem is
not an explicit row height. Split into accurate rows. (gpt-5.6-sol)
Rules that could suppress rubric-rewarded answers (the main eval risk):
- The ListView stop-condition ("do not migrate unless the user asked to
migrate") fired on eval stimulus 4, which asks "Should I keep using ListView
in MAUI?" - the user asks *whether*, not *to*. Reworded to recommend
CollectionView when asked which control to use, while still forbidding
silent rewrites. Flagged independently by opus-5 and gpt-5.5.
- data-binding's blanket "do not convert a working binding" could talk the
agent out of the rubric-rewarded x:DataType recommendation. Scoped it. (opus-5)
- "Point out a concrete defect or say nothing" -> answer the question asked;
"say nothing" invited non-answers, which judges penalise. (opus-5)
Internal contradictions introduced by this PR:
- data-binding Common Pitfalls still said "enable XC0025 as error" for a
missing x:DataType; XC0022 is the missing-x:DataType code. (opus-4.8)
- The rules table said x:DataType "root element only" two rows above a row
requiring one per DataTemplate. Reworded to "wherever a binding scope
starts". (opus-5, gpt-5.6-sol)
- The Validation checklist re-armed items the Scope Control section had just
demoted; scoped it to markup the agent actually wrote. (opus-5)
- Dropped the "Using ListView instead of CollectionView" pitfall row, which
contradicted the new migration section. (opus-5)
Other:
- Removed the escaped pipe (\|) from the theming rules table - a model reading
raw Markdown could copy it into C#. (Copilot reviewer, opus-5)
- Softened "Apply these three rules to every theming answer" so narrow
questions don't attract unrelated UiMode/UserAppTheme guidance. (gpt-5.5)
- De-duplicated the AddScoped guidance that appeared 4x. (opus-5)
Measured effect (maui-collectionview, executor claude-opus-4.6, judge gpt-5.5,
n=3/stimulus, 12 trials/arm, base re-run in the same session):
base (main, no fix) 33% win 4 wins / 4 ties / 4 losses mean +0.00 [-0.22,+0.22]
previous commit 42% win 5 wins / 6 ties / 1 loss mean +0.13 [-0.03,+0.30]
this commit 58% win 7 wins / 5 ties / 0 losses mean +0.23 [+0.10,+0.36]
Zero losses, and the confidence interval is now entirely above zero - the
first arm in this series with a statistically significant positive impact.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 58513e0c-4086-4112-9ea1-95dbd8e8e469
Driven by a local skilled-vs-baseline run (vally 0.10, executor
claude-opus-4.6, judge gpt-5.5, n=3 per stimulus). The judge rationales
named two concrete, fixable causes for the remaining losses:
1. "Response B introduces an unresolved x:DataType prefix unless additional
namespace declarations are added" (penalised on 2 trials). The skill used
x:DataType="models:..." in six snippets but never declared xmlns:models,
so the examples were not copy-pasteable. Basic Setup is now a complete
ContentPage including the xmlns:models declaration, with an explicit
instruction to carry the matching xmlns when handing over a snippet.
2. Baseline beat the skill on the ListView/ViewCell migration stimulus by
supplying nuance the skill lacked - that ListView still exists, plus
event/API replacements. Added a "Migrating from ListView" section with a
verified mapping table (ViewCell, ItemSelected/ItemTapped ->
SelectionChanged, IsPullToRefreshEnabled -> RefreshView, IsGroupingEnabled
-> IsGrouped, HasUnevenRows -> ItemSizingStrategy, and the missing
separator APIs). Every member was confirmed against
Controls/PublicAPI.Shipped.txt.
Measured effect on the same eval (12 trials per arm, identical settings):
original skill 17% win rate 2 wins / 4 ties / 6 losses mean -0.13
this PR 42% win rate 5 wins / 6 ties / 1 loss mean +0.13
Losses drop 6 -> 1 and the mean impact flips from negative (the
FIX-REGRESSION signal in #895) to positive. n=12 on a single executor, so
directional rather than significant, but the loss collapse is the intended
effect of the regression fix.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 58513e0c-4086-4112-9ea1-95dbd8e8e469
Addresses the cross-family evaluation findings in dotnet/skills#895 for the
dotnet-maui plugin (8 skills, 48% pass, 1 P0).
Every claim below was validated empirically by building a real .NET MAUI app
against the current SDK, not by code reading alone.
FIX-REGRESSION - maui-collectionview (P0):
- ItemSizingStrategy was documented on LinearItemsLayout in both SKILL.md and
references/collectionview-api.md. It is declared on StructuredItemsView, so
the documented markup is a hard build break:
"error MAUIX2002: No accessible property, BindableProperty, or event found
for ItemSizingStrategy". Corrected to set it on CollectionView, with the
invalid form kept as an explicit counter-example.
- Added a "Scope Control" section with stop-conditions (do NOT act when...),
narrowed the frontmatter trigger, demoted the non-essential guidance to
suggestions the agent may decline, and reduced the three non-negotiable
rules to those that actually break code. This targets the ~30% of trials
where skilled output was worse than baseline through over-application.
- Added a validation checklist.
HIGH severity, non-compiling - maui-app-lifecycle:
- The iOS ConfigureLifecycleEvents examples called .DidBecomeActive() and
.WillResignActive(), which do not exist:
"error CS1061: IiOSLifecycleBuilder does not contain a definition for
DidBecomeActive". Corrected to .OnActivated() / .OnResignActivation() and
documented that the builder method names deliberately differ from the UIKit
selector names (the mapping tables now show both columns).
Correctness - maui-data-binding:
- The XC0022-XC0025 meanings table was wrong on all four rows. Corrected
against Build.Tasks/BuildException.cs and ErrorMessages.resx (XC0022 =
binding without x:DataType, XC0023 = x:DataType explicitly null, XC0024 =
x:DataType from outer scope, XC0025 = explicit Source).
Correctness - maui-safe-area:
- Claimed WindowSoftInputModeAdjust.Resize was "removed". It still compiles
and is not obsolete. Reframed as Android-only and superseded, preserving
the skill's brevity (EFFICIENT-WIN - deliberately not grown).
STRENGTHEN - decisiveness (ties dominated these skills):
- Added a compact "Rules That Change the Answer" table to maui-shell-navigation,
maui-data-binding, maui-dependency-injection and maui-theming, converting
reference prose into when-A-do-B guidance, each with an explicit "do not
churn working code" stop-condition.
Notably NOT changed:
- maui-theming's {AppThemeBinding Light={DynamicResource ...}} guidance. This
looked invalid from source reading, but a runtime probe proved it is fully
dynamic (value tracked a dictionary swap: #FF0000 -> #0000FF, while the
StaticResource control stayed #FF0000). The original guidance is correct.
- maui-shell-navigation fixture/SDK pinning. Per the #909 correction the
errored trial was a judge-side disabled-PAT failure, not fixture
nondeterminism, so the skill was treated as STRENGTHEN.
- dotnet-maui-doctor (EXEMPLAR) is unchanged.
Validation:
- skill-validator check --plugin ./plugins/dotnet-maui -> all checks passed
(8 skills, 0 errors, 26 reference files scanned).
- All corrected snippets compile in a net11.0-maccatalyst MAUI app; both
original buggy forms reproduce the exact errors quoted above.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 58513e0c-4086-4112-9ea1-95dbd8e8e469
The dotnet11/system-text-json-net11 skill scored 0%% pass across all five model families (issue #902, ADD-DECISIVENESS): it read as reference prose and the judge saw no behavior change vs baseline (7 ties, 67%% invocation).
Rewrite it into imperative when-A-do-B-verify-C guidance:
- Sharper frontmatter USE FOR / DO NOT USE FOR with trigger keywords to improve discovery.
- Decision table mapping each request to the exact API and the anti-pattern to avoid.
- Explicit DO/DON'T that target the eval's failure modes: no custom JsonNamingPolicy subclass, no cast of non-generic JsonTypeInfo, no try/catch to probe metadata.
- net11.0 run instructions (file-based app + project) so the program is actually executed.
- Verification checklist and common-pitfalls table.
Validated with skill-validator check (all checks passed).
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Abhitej John