* dotnet-test: make code-testing agent tools declarations Claude Code-compatible
PR #847 added `tools: ["agent", "skill", "read", "search", "edit", "execute"]`
to the code-testing-* agents to enable VS Code / Copilot CLI subagent fan-out.
Those lowercase aliases map to real tools in VS Code and the Copilot CLI, but
Claude Code matches `tools:` against its own vocabulary (Task, Skill, Read,
Glob, Grep, Edit, Write, Bash). None of the aliases matched, so when these
agents are loaded into Claude Code via --plugin-dir and selected with
`claude --agent`, the agent was granted ZERO tools. A tool-less model asked
to generate tests emits a textual <tool_call> block and exits after one turn,
producing no file changes.
Append the Claude Code tool names to each agent's `tools:` list so the same
declaration works across all three runtimes (each honors the names it knows and
ignores the foreign ones):
- Orchestrators (generator, implementer): add Task, Skill, Read, Glob, Grep,
Edit, Write, Bash (Task is the Claude Code equivalent of the `agent`
fan-out tool).
- Workers (researcher, planner, builder, tester, fixer, linter): add Skill,
Read, Glob, Grep, Edit, Write, Bash.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* skill-validator: complete built-in tools + add cross-host tool portability check
Two related follow-ups to the agent tools fix:
1. Address the skill-check review feedback. The validator's BuiltInTools set was
missing three legitimate host tool spellings that are not case-insensitive
matches of existing entries, so they were flagged as non-built-in:
- "write" — Claude Code file-creation tool (Copilot CLI / VS Code: "create")
- "agent" — Copilot CLI / VS Code subagent fan-out tool (Claude Code: "task")
- "execute" — Copilot CLI / VS Code run-command tool (Claude Code: "bash")
"agent" and "execute" were already flagged before this branch (introduced by
the fan-out PR); adding them to BuiltInTools clears the pre-existing warnings.
2. Add a cross-host tool portability check (CheckAgentToolPortability) so an
agent that declares a capability for only one host is flagged. Tool names are
matched case-sensitively (hosts resolve tools by exact spelling), so an agent
that lists e.g. only "edit" (Copilot / VS Code) without "Edit"/"Write"
(Claude Code) is reported as working on one host and silently tool-less on the
other. Findings are advisory (do not fail CI) and allowlistable via
"agent-tool-portability:AGENT:capability". Wired into the agents loop in
CheckCommand and covered by unit tests.
Also make the one existing single-host agent (optimizing-dotnet-performance)
portable by adding its Claude Code tool spellings, so the new check reports a
clean tree.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* skill-validator: restore 15K aggregate cap, document it as the real Copilot CLI skill-menu budget
The per-plugin aggregate description cap had been raised 15,000 -> 20,000
-> 22,000 under the belief that 15K was 'a local repo policy, NOT a
documented Copilot constraint'. That belief was wrong: the GitHub Copilot
CLI renders the model-facing <available_skills> menu under a hard 15,000-
char budget (the agent SDK's SKILL_CHAR_BUDGET, default 15e3, confirmed in
CLI 1.0.36 and 1.0.61). Skills are listed alphabetically and emitted with
their full <description> only until the budget is exhausted; every skill
past the cut-off collapses to a bare name with no description and can no
longer be reliably model-activated. Raising the validator cap merely
masked this silent menu truncation — e.g. dotnet-test's run-tests and
test-* skills stopped activating in plugin eval runs because they fell
into the name-only overflow.
Changes:
- SkillProfiler.MaxAggregateDescriptionLength: 22,000 -> 15,000, with the
comment rewritten to document the real Copilot CLI budget (and correct
the prior 'not a documented constraint' claim).
- CheckCommand aggregate now excludes skills marked
'disable-model-invocation: true' — the CLI drops those from the menu, so
they do not consume the budget. This makes the cap satisfiable by hiding
reference / agent-orchestrated primitives rather than only by trimming.
- InvestigatingResults.md: document plugin-arm-only non-activation caused
by skill-menu budget overflow, and how to fix it.
Note: dotnet-test currently exceeds 15K and must be slimmed below it
(via disable-model-invocation on reference/primitive skills plus
description trims) before this cap can go green repo-wide.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* skill-validator: use source-generated regex for disable-model-invocation check
Address review: replace Regex.IsMatch(pattern-string) with a
[GeneratedRegex] partial method (AOT-friendly, no per-call cache lookup),
matching FrontmatterParser's style. Runs once per skill during checks.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* skill-validator: parse disable-model-invocation via YAML to avoid block-scalar false positives
The regex-based check matched any line in the frontmatter, so a block-scalar description that merely mentioned 'disable-model-invocation: true' on its own line was wrongly treated as disabling model invocation. Parse the frontmatter with the existing YAML deserializer (which correctly handles block scalars) by adding a DisableModelInvocation field to SkillFrontmatter, and drop the regex entirely.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add JSON output for skill-validator check
Fixes#600
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix empty check discovery results
Address PR feedback by failing when explicit skill or agent paths discover nothing, including the combined check path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address Copilot review comments
- Replace AddPlainError with AddGeneralError (remove duplicate method)
- Key pluginSkills by DirectoryPath instead of display name to avoid
mismatch when plugin.Name differs from directory name
- Use OS-aware path comparison in IsPathWithin (Ordinal on Unix,
OrdinalIgnoreCase on Windows)
- Pre-build name lookup dictionaries in CreateJsonOutput to eliminate
O(n*m) FirstOrDefault scans for external dependency attachment
- Extract CheckJsonSerializerContext scoped to check JSON output so
UseStringEnumConverter does not affect unrelated JSON payloads
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address latest Copilot PR feedback
- Attach external dependency warnings by stable path identifiers
(skill SKILL.md path, agent path, plugin directory path)
- Use OS-aware comparer for plugin directory path matching
- Restore console warning formatting via SkillProfiler.FormatProfileWarnings
- Move CheckJsonSerializerContext into SkillValidator.Check namespace
- Precompute reference-attachment targets to avoid repeated full-path
normalization and container recalculation per finding
- Add regression test covering duplicate skill names with external
dependency warnings in JSON output
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix flaky duplicate-skill JSON test path
- Run duplicate-name external dependency assertion through plugin mode,
which is where external dependency checks are executed
- Add plugin fixture helper for JSON output tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address latest Copilot follow-up feedback
- Reintroduce PluginValidationResult as obsolete compatibility type
while keeping PluginCheckResult as the primary model
- Avoid unnecessary profile-line formatting when verbose output is off
- Replace string discriminators for external dependency kind with enum
- Centralize JSON warning kind literals as constants
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
/-prefix paths bypassed the flag and hit the depth check unconditionally. Now treated symmetrically with .. traversal: allowed only with the flag, otherwise produce a clear absolute (repo-rooted) path error. Also classify // as a protocol-relative URL.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add validator options
* Address PR review: validate options and add tests
- Reject --allow-repo-traversal with --plugin mode (plugins must be portable)
- Validate --max-declaration-lines and --max-agent-lines are positive integers
- Add unit tests for MaxDeclarationLines override in SkillProfiler
- Add unit tests for AllowRepoTraversal suppression and depth check preservation
- Add unit tests for MaxAgentLines override and fallback to MaxDeclarationLines
* Remove body line count checks and --max-declaration-lines/--max-agent-lines options
Line count checks are redundant with BPE token-based complexity classification,
which already provides better guidance. Keep --allow-repo-traversal only.
* Address review: remove unused CheckOptions from AgentProfiler, fix traversal test
- Remove unused CheckOptions parameter from AgentProfiler.AnalyzeAgent and
RunAgentsCheckCore since line checks were removed
- Fix AllowRepoTraversalSuppressesParentTraversalError test to use a non-deep
path (../SKILL.md) and assert no file-ref errors at all
Reorganize the skill-validator project from layer-based grouping
(Commands/, Services/, Models/, Utilities/) to feature-based slices
(Check/, Evaluate/, Consolidate/, Shared/).
Key changes:
- Check/: CheckCommand, SkillProfiler, AgentProfiler, PluginValidator,
ExternalDependencyChecker, ReferenceScanner
- Evaluate/: EvaluateCommand, RejudgeCommand (now a subcommand of
evaluate), AgentRunner, Judge, PairwiseJudge, and other eval services
- Consolidate/: ConsolidateCommand
- Shared/: Models, SkillDiscovery, PluginParser (extracted from
PluginValidator), Reporter, and utilities
Split PluginValidator: parsing helpers (ParsePluginJson,
TryGetSafeSubdirectory) moved to Shared/PluginParser.cs; validation
logic stays in Check/PluginValidator.cs.
Move RawFrontmatter and RawAgentFrontmatter types from EvalSchema into
SkillDiscovery since they are only consumed by discovery.
Wire RejudgeCommand as subcommand: skill-validator evaluate rejudge.
Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Amaury Levé