mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
Dan Moseley
8e50fc8558
* Add agent and plugin spec conformance validation Extend the validation pipeline to check .agent.md files and plugin.json files alongside skills. Previously only SKILL.md files were validated. Agent validation (AgentProfiler): - Frontmatter required (error) - Name and description required (error) - Name format: same rules as skills (lowercase alphanum + hyphens, 64 max) - Name must match filename minus .agent.md suffix - Description max 1024 chars - Body max 500 lines Plugin validation (PluginValidator): - Name required, must match directory name - Version and description required - Description max 1024 chars - Skills path required and must exist - Agents path warned if specified but missing Discovery (SkillDiscovery): - DiscoverAgents() walks up from skill paths to plugin roots, scans agents/ directories for .agent.md files - DiscoverPlugins() finds plugin.json from skill paths Integration (ValidateCommand): - Runs after skill discovery, before evaluation - Errors block with exit code 1 (same as spec_conformance_failure) All 4 existing agents and 2 plugins pass cleanly. 25 new tests (13 agent, 12 plugin). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Code review fixes: spec comments, ValidateNameFormat, kind-specific messages - Refactor ValidateName into ValidateNameFormat (format-only) + ValidateName (format + directory match) so agent/plugin validation doesn't abuse the skill directory-match check by passing name twice. - Warning messages now say 'Agent name' or 'Plugin name' instead of 'Skill name' when called from AgentProfiler/PluginValidator. - Add spec URL comments to AgentProfiler and PluginValidator referencing agentskills.io/specification, VS Code agent-plugins docs, and Claude Code plugins-reference. - Add missing agent name edge-case tests: leading/trailing hyphen, consecutive hyphens, and message-prefix assertions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove unused agents path from plugins that have no agents The dotnet, dotnet-data, and dotnet-upgrade plugins declared an agents path but had no agents/ directory, producing validation warnings. Remove the field; it can be added back when agents are actually created. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Promote agent/plugin name validation from warnings to errors The agentskills.io spec uses 'Must' for all name constraints (length, character set, no leading/trailing hyphens, no consecutive hyphens). For a publishing validator these should block, not just warn. Agent filename mismatch is also promoted since the IDE won't discover the agent if the name doesn't match. Skill name validation stays as warnings (lenient loading, not part of this PR). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback - Fix FrontmatterRegex: remove RegexOptions.Multiline so a horizontal rule (---) in the body is not mistaken for frontmatter. Detection and stripping now both anchor to start of string. - Use filename as fallback identifier in AgentProfile when agent.Name is empty, so error output shows [agent:foo.agent.md] not [agent:]. - Add TryGetSafeSubdirectory to PluginValidator: rejects absolute paths and parent-directory traversal in skills/agents path fields. Used by both PluginValidator and SkillDiscovery.DiscoverAgents. - Use DirectoryName as fallback for plugin result name when Name is empty, so error output shows [plugin:my-dir] not [plugin:]. - Add tests for path traversal rejection and name fallback behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Detect orphaned test directories with no matching plugin/skill Add FindOrphanedTestDirectories to SkillDiscovery: checks that every directory under tests/{plugin}/{skill}/ has a corresponding directory under plugins/{plugin}/skills/{skill}/. Plugin-level orphans (tests/ directory with no matching plugins/ directory) are also flagged. Integrated into ValidateCommand after plugin/agent validation so it runs on every validation pass. Errors are blocking (exit 1). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #279 review feedback - Use case-sensitive (Ordinal) comparison for agent filename matching since agent names are required to be lowercase by spec. - Replace OrdinalIgnoreCase path-under-root check in TryGetSafeSubdirectory with Path.GetRelativePath + '..' prefix check for correct behavior on case-sensitive filesystems. - Let ParsePluginJson throw JsonException on malformed JSON instead of silently returning null. ValidateCommand catches it as a blocking error. - Add test for malformed plugin.json. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Increase eval timeouts for dump-collect and dotnet-trace-collect to 120s The with-skill runs for these skills were hitting the 60s scenario timeout in CI, causing the agent to be killed before producing output. Other skills in the same plugin already use 120s. The 30s decline/negative-test scenario is left unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix fixture paths in clr-activation-debugging and analyzing-dotnet-performance evals The source paths referenced tests/dotnet/ but the fixtures live under tests/dotnet-diag/. This caused file-not-found exceptions at runtime, which set hasRejections and caused unconditional exit 1 (bypassing --verdict-warn-only). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Surface execution errors in validation summary Previously, skills that threw exceptions (e.g. missing fixture files) were silently excluded from the verdict count. The summary showed '2/4 skills passed' with no indication that 2 other skills crashed entirely. Now the summary shows '2/6 skills passed (2 rejected due to execution errors)' and lists the specific error messages near the summary rather than buried in the middle of the log. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reduce parallelism for infra-change evaluation runs When infrastructure changes trigger evaluation of all plugins simultaneously, the high default parallelism (5x5x5) causes API rate limits (429s) and scenario timeouts from contention. Reduce to 2x3x3 for infra-change runs only; regular single-plugin PR evaluations keep the default 5x5x5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
170 lines
7.3 KiB
YAML
170 lines
7.3 KiB
YAML
scenarios:
|
|
- name: "Configure automatic crash dumps for CoreCLR app on Linux"
|
|
prompt: |
|
|
I have a .NET 10 CoreCLR web API running on Linux. I need to configure
|
|
automatic crash dump collection so that when the app crashes, a full
|
|
dump is written to /var/dumps. Show me the environment variables to set
|
|
and how to verify the configuration.
|
|
assertions:
|
|
- type: "output_contains"
|
|
value: "DOTNET_DbgEnableMiniDump"
|
|
- type: "output_contains"
|
|
value: "DOTNET_DbgMiniDumpType"
|
|
- type: "output_contains"
|
|
value: "DOTNET_DbgMiniDumpName"
|
|
- type: "output_contains"
|
|
value: "/var/dumps"
|
|
rubric:
|
|
- "Sets DOTNET_DbgEnableMiniDump=1 to enable crash dumps"
|
|
- "Sets DOTNET_DbgMiniDumpType=4 for full dumps"
|
|
- "Sets DOTNET_DbgMiniDumpName with the /var/dumps path"
|
|
- "Includes a verification step to confirm env vars are set"
|
|
- "Does not attempt to analyze dumps or install analysis tools"
|
|
timeout: 120
|
|
|
|
- name: "Set up NativeAOT crash dumps with createdump in Kubernetes"
|
|
prompt: |
|
|
I have a NativeAOT .NET 11 app running in Kubernetes. The createdump
|
|
tool is installed at /opt/tools/createdump. Configure automatic crash
|
|
dump collection and give me the pod spec changes I need.
|
|
assertions:
|
|
- type: "output_contains"
|
|
value: "DOTNET_DbgEnableMiniDump"
|
|
- type: "output_contains"
|
|
value: "DOTNET_DbgCreateDumpToolPath"
|
|
- type: "output_contains"
|
|
value: "SYS_PTRACE"
|
|
- type: "output_contains"
|
|
value: "/opt/tools/createdump"
|
|
rubric:
|
|
- "Uses DOTNET_DbgCreateDumpToolPath for .NET 11 NativeAOT"
|
|
- "Adds SYS_PTRACE capability to the container securityContext"
|
|
- "Mounts a volume for dump output"
|
|
- "Sets DOTNET_DbgMiniDumpType=4 since NativeAOT only supports full dumps"
|
|
- "Does not attempt to analyze dumps"
|
|
timeout: 120
|
|
|
|
- name: "Recover crash dump from macOS NativeAOT without createdump"
|
|
prompt: |
|
|
My NativeAOT app just crashed on macOS. I don't have createdump
|
|
available. How can I find the crash dump or crash report?
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "/cores"
|
|
- type: "output_matches"
|
|
pattern: "DiagnosticReports|crash"
|
|
rubric:
|
|
- "Checks /cores/ directory for core dump files"
|
|
- "Checks ~/Library/Logs/DiagnosticReports for .crash or .ips reports"
|
|
- "Explains that if ulimit -c unlimited was not set before the crash, the core dump may be lost"
|
|
- "Does not attempt to analyze the dump contents"
|
|
timeout: 120
|
|
|
|
- name: "Configure CoreCLR dump collection in Alpine Docker as non-root"
|
|
prompt: |
|
|
I need to set up crash dump collection for my CoreCLR app running in an
|
|
Alpine-based Docker container. The container runs as a non-root user.
|
|
Give me the Dockerfile changes and docker run command.
|
|
assertions:
|
|
- type: "output_contains"
|
|
value: "DOTNET_DbgEnableMiniDump"
|
|
- type: "output_contains"
|
|
value: "SYS_PTRACE"
|
|
- type: "output_matches"
|
|
pattern: "chown|chmod|fsGroup"
|
|
rubric:
|
|
- "Sets the DOTNET_DbgEnableMiniDump environment variable"
|
|
- "Adds SYS_PTRACE capability via --cap-add or securityContext"
|
|
- "Addresses dump directory permissions for non-root user"
|
|
- "Does not claim Alpine requires special workarounds for crash dumps"
|
|
- "Does not attempt to analyze dumps"
|
|
timeout: 120
|
|
|
|
- name: "Advisory: macOS NativeAOT crash dump recovery steps"
|
|
prompt: |
|
|
What steps would I follow to recover a crash dump on macOS if my
|
|
NativeAOT app crashed and I don't have createdump available?
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "/cores"
|
|
- type: "output_matches"
|
|
pattern: "DiagnosticReports|crash"
|
|
- type: "output_matches"
|
|
pattern: "ulimit"
|
|
rubric:
|
|
- "Describes checking /cores/ directory for core dump files"
|
|
- "Describes checking ~/Library/Logs/DiagnosticReports for crash reports"
|
|
- "Mentions that ulimit -c unlimited must have been set before the crash for a core dump to exist"
|
|
- "Provides guidance without attempting to execute commands or search the filesystem"
|
|
- "Does not attempt to analyze dump contents"
|
|
timeout: 120
|
|
|
|
- name: "Advisory: CoreCLR Alpine Docker non-root configuration"
|
|
prompt: |
|
|
If I had a CoreCLR app in an Alpine Docker container running as a
|
|
non-root user, what would I need to configure for crash dump collection?
|
|
assertions:
|
|
- type: "output_contains"
|
|
value: "DOTNET_DbgEnableMiniDump"
|
|
- type: "output_contains"
|
|
value: "SYS_PTRACE"
|
|
- type: "output_matches"
|
|
pattern: "chown|chmod|mkdir|permission"
|
|
rubric:
|
|
- "Explains DOTNET_DbgEnableMiniDump and related env vars"
|
|
- "Explains the need for SYS_PTRACE capability"
|
|
- "Explains dump directory ownership or permissions for non-root user"
|
|
- "Does not claim Alpine requires special workarounds for crash dumps"
|
|
- "Provides advisory guidance without attempting to execute commands"
|
|
timeout: 120
|
|
|
|
- name: "Advisory: NativeAOT Kubernetes dump collection setup"
|
|
prompt: |
|
|
Walk me through how I would set up crash dump collection for a
|
|
NativeAOT .NET 11 app in Kubernetes, assuming createdump is available
|
|
at /opt/tools/createdump. What pod spec changes are needed?
|
|
assertions:
|
|
- type: "output_contains"
|
|
value: "DOTNET_DbgEnableMiniDump"
|
|
- type: "output_contains"
|
|
value: "DOTNET_DbgCreateDumpToolPath"
|
|
- type: "output_contains"
|
|
value: "SYS_PTRACE"
|
|
rubric:
|
|
- "Uses DOTNET_DbgCreateDumpToolPath since this is .NET 11 NativeAOT"
|
|
- "Adds SYS_PTRACE capability to the container securityContext"
|
|
- "Describes a volume mount for dump output"
|
|
- "Sets DOTNET_DbgMiniDumpType=4 since NativeAOT only supports full dumps"
|
|
- "Provides advisory guidance without attempting to modify actual cluster resources"
|
|
timeout: 120
|
|
|
|
- name: "Detect runtime and configure crash dumps for unknown .NET app on Linux"
|
|
prompt: |
|
|
I have a .NET 10 app running on Linux (PID 12345) and I need to set up
|
|
automatic crash dump collection. I'm not sure if it's CoreCLR or NativeAOT.
|
|
Help me figure out which runtime it's using and configure dumps accordingly.
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "CorExeMain|Rhp|strings|readlink"
|
|
rubric:
|
|
- "Attempts to detect the runtime before configuring dumps"
|
|
- "Uses binary inspection (strings, nm) or process inspection to distinguish CoreCLR from NativeAOT"
|
|
- "Provides appropriate configuration based on the detected runtime"
|
|
- "Does not assume one runtime or the other without checking"
|
|
- "Does not attempt to analyze dumps"
|
|
timeout: 120
|
|
|
|
- name: "Decline dump analysis request"
|
|
prompt: |
|
|
I already have a .dmp crash dump file from my .NET app. Can you help
|
|
me analyze it to find the root cause of the crash?
|
|
expect_activation: false
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "out of scope|not cover|does not|cannot|only.*collect"
|
|
rubric:
|
|
- "Clearly states that dump analysis is out of scope for this skill"
|
|
- "Does not attempt to open or analyze the dump file"
|
|
- "Does not install analysis tools like dotnet-dump analyze, lldb, or windbg"
|
|
timeout: 30
|