mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
Amaury Levé
3b670a8276
* skill-validator: restore 15K aggregate cap, document it as the real Copilot CLI skill-menu budget The per-plugin aggregate description cap had been raised 15,000 -> 20,000 -> 22,000 under the belief that 15K was 'a local repo policy, NOT a documented Copilot constraint'. That belief was wrong: the GitHub Copilot CLI renders the model-facing <available_skills> menu under a hard 15,000- char budget (the agent SDK's SKILL_CHAR_BUDGET, default 15e3, confirmed in CLI 1.0.36 and 1.0.61). Skills are listed alphabetically and emitted with their full <description> only until the budget is exhausted; every skill past the cut-off collapses to a bare name with no description and can no longer be reliably model-activated. Raising the validator cap merely masked this silent menu truncation — e.g. dotnet-test's run-tests and test-* skills stopped activating in plugin eval runs because they fell into the name-only overflow. Changes: - SkillProfiler.MaxAggregateDescriptionLength: 22,000 -> 15,000, with the comment rewritten to document the real Copilot CLI budget (and correct the prior 'not a documented constraint' claim). - CheckCommand aggregate now excludes skills marked 'disable-model-invocation: true' — the CLI drops those from the menu, so they do not consume the budget. This makes the cap satisfiable by hiding reference / agent-orchestrated primitives rather than only by trimming. - InvestigatingResults.md: document plugin-arm-only non-activation caused by skill-menu budget overflow, and how to fix it. Note: dotnet-test currently exceeds 15K and must be slimmed below it (via disable-model-invocation on reference/primitive skills plus description trims) before this cap can go green repo-wide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * skill-validator: use source-generated regex for disable-model-invocation check Address review: replace Regex.IsMatch(pattern-string) with a [GeneratedRegex] partial method (AOT-friendly, no per-call cache lookup), matching FrontmatterParser's style. Runs once per skill during checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test-gap-analysis: cover validation safeguards Add focused evaluation scenarios for proportional scope, complete outcome inventory, mutation restoration, demonstrated recommendations, and canonical verifier usage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846 * test-gap-analysis: harden mutation verifier eval Anchor verifier paths to the script directory and require the eval response to report the supplied canonical verifier result. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846 * test-gap-analysis: distinguish verifier failures Require Python 3, preflight the test runner, build mutants explicitly, and treat only the MTP test-failure exit code as a killed mutation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846 * test-gap-analysis: tighten eval evidence Challenge proportional scope with an unrelated fixture and require affirmative canonical-verifier success wording. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846 * test-gap-analysis: name MTP failure exit code Document the Microsoft.Testing.Platform test-failure convention used by the mutation verifier. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846 * test-gap-analysis: require affirmative verifier result Constrain the output grader to a direct successful result for verify_mutations.sh so negative mutation summaries cannot pass. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846 * Improve MSTest migration skill activation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c9ea64d8-cced-442a-b558-2bb43579c316 * test-gap-analysis: cover proportional scope validation Name the validation outcome directly and prove the focused review excludes an unrelated AccessControl fixture. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846 * test-gap-analysis: classify killed mutants by test evidence Capture MTP output and require an xUnit execution summary with failed tests, avoiding runner-specific exit-code assumptions while preserving infrastructure diagnostics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846 * test-gap-analysis: preserve mutant build diagnostics Capture MSBuild output and print it when mutant compilation fails so verifier infrastructure failures remain actionable in CI. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846 --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81d0f63d-89cb-4802-a1da-82a9c4e5c846 Copilot-Session: c9ea64d8-cced-442a-b558-2bb43579c316