mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
eval-exec-opus48
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aditya Mandaleeka
|
52ba152ed6 | Update to Vally 0.7.0 (#854) | ||
|
Aditya Mandaleeka
|
9433a31e67 | Initial port of evals to Vally (#615) | ||
|
Jeff Schwartz
|
55a7d80527 |
adding SIMD/TensorPrimitives to dotent-diag/analyzing-dotnet-performance (#330)
* incorporated feedback from previous pr, iterated with 100+ scalar loop tests to come to this skill, moved from skill to reference in the performance skill * Update tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address review feedback on SIMD vectorization reference - Check Span<T>/MemoryExtensions before TensorPrimitives (no extra dependency) - Add all supported types (sbyte, ushort, uint, ulong, nint, nuint, char via ushort) - TensorPrimitives: add constraint and applicable types columns (not just float/double) - Add FusedMultiplyAdd, clarify AddMultiply vs MultiplyAdd distinction - Prefer portable APIs over platform-specific intrinsics (allow when perf justifies) - Fix dispatch pattern to use if/else if to avoid pessimizing small inputs - Remove LINQ references Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * addressing more feedback * move to the experimental plugin * move the simd tests to exprimental * add codeowners --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
ab201ee691 |
Fix eval source file resolution to use eval directory (#374)
* Fix eval source file resolution to use eval directory The SetupWorkDir method resolved setup file 'source:' paths relative to the skill directory (plugins/.../skills/<name>/), but eval.yaml files live under tests/. The ../../../../ relative paths resolved correctly but were rejected by the path traversal security check because the resolved path escaped the skill directory boundary. Fix by preferring the eval directory as the base for source resolution when evalPath is available, falling back to skillPath for backward compatibility with scenarios embedded in SKILL.md files. Also update run-tests eval.yaml source paths from '../../../../tests/dotnet-test/run-tests/fixtures/...' to 'fixtures/...' (relative to the eval directory where the fixtures actually live). This was causing all 11 source-based run-tests scenarios to get zero fixture files, making the agent find an empty workspace and fail to evaluate. * Address review: extract ResolveSourcePath helper, add warning, fix remaining eval.yaml paths - Extract source-resolution and path-traversal check into internal ResolveSourcePath helper for testability - Emit a clear warning when file.Source is set but no base directory is available (instead of silently continuing) - Add unit tests covering: evalPath-based resolution, skillPath fallback, null paths warning, traversal rejection, and evalPath preference - Fix path traversal patterns in analyzing-dotnet-performance and clr-activation-debugging eval.yaml files (same issue as run-tests) |
||
|
Dan Moseley
|
8e50fc8558 |
Add agent and plugin spec conformance validation (#279)
* Add agent and plugin spec conformance validation Extend the validation pipeline to check .agent.md files and plugin.json files alongside skills. Previously only SKILL.md files were validated. Agent validation (AgentProfiler): - Frontmatter required (error) - Name and description required (error) - Name format: same rules as skills (lowercase alphanum + hyphens, 64 max) - Name must match filename minus .agent.md suffix - Description max 1024 chars - Body max 500 lines Plugin validation (PluginValidator): - Name required, must match directory name - Version and description required - Description max 1024 chars - Skills path required and must exist - Agents path warned if specified but missing Discovery (SkillDiscovery): - DiscoverAgents() walks up from skill paths to plugin roots, scans agents/ directories for .agent.md files - DiscoverPlugins() finds plugin.json from skill paths Integration (ValidateCommand): - Runs after skill discovery, before evaluation - Errors block with exit code 1 (same as spec_conformance_failure) All 4 existing agents and 2 plugins pass cleanly. 25 new tests (13 agent, 12 plugin). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Code review fixes: spec comments, ValidateNameFormat, kind-specific messages - Refactor ValidateName into ValidateNameFormat (format-only) + ValidateName (format + directory match) so agent/plugin validation doesn't abuse the skill directory-match check by passing name twice. - Warning messages now say 'Agent name' or 'Plugin name' instead of 'Skill name' when called from AgentProfiler/PluginValidator. - Add spec URL comments to AgentProfiler and PluginValidator referencing agentskills.io/specification, VS Code agent-plugins docs, and Claude Code plugins-reference. - Add missing agent name edge-case tests: leading/trailing hyphen, consecutive hyphens, and message-prefix assertions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove unused agents path from plugins that have no agents The dotnet, dotnet-data, and dotnet-upgrade plugins declared an agents path but had no agents/ directory, producing validation warnings. Remove the field; it can be added back when agents are actually created. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Promote agent/plugin name validation from warnings to errors The agentskills.io spec uses 'Must' for all name constraints (length, character set, no leading/trailing hyphens, no consecutive hyphens). For a publishing validator these should block, not just warn. Agent filename mismatch is also promoted since the IDE won't discover the agent if the name doesn't match. Skill name validation stays as warnings (lenient loading, not part of this PR). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback - Fix FrontmatterRegex: remove RegexOptions.Multiline so a horizontal rule (---) in the body is not mistaken for frontmatter. Detection and stripping now both anchor to start of string. - Use filename as fallback identifier in AgentProfile when agent.Name is empty, so error output shows [agent:foo.agent.md] not [agent:]. - Add TryGetSafeSubdirectory to PluginValidator: rejects absolute paths and parent-directory traversal in skills/agents path fields. Used by both PluginValidator and SkillDiscovery.DiscoverAgents. - Use DirectoryName as fallback for plugin result name when Name is empty, so error output shows [plugin:my-dir] not [plugin:]. - Add tests for path traversal rejection and name fallback behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Detect orphaned test directories with no matching plugin/skill Add FindOrphanedTestDirectories to SkillDiscovery: checks that every directory under tests/{plugin}/{skill}/ has a corresponding directory under plugins/{plugin}/skills/{skill}/. Plugin-level orphans (tests/ directory with no matching plugins/ directory) are also flagged. Integrated into ValidateCommand after plugin/agent validation so it runs on every validation pass. Errors are blocking (exit 1). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #279 review feedback - Use case-sensitive (Ordinal) comparison for agent filename matching since agent names are required to be lowercase by spec. - Replace OrdinalIgnoreCase path-under-root check in TryGetSafeSubdirectory with Path.GetRelativePath + '..' prefix check for correct behavior on case-sensitive filesystems. - Let ParsePluginJson throw JsonException on malformed JSON instead of silently returning null. ValidateCommand catches it as a blocking error. - Add test for malformed plugin.json. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Increase eval timeouts for dump-collect and dotnet-trace-collect to 120s The with-skill runs for these skills were hitting the 60s scenario timeout in CI, causing the agent to be killed before producing output. Other skills in the same plugin already use 120s. The 30s decline/negative-test scenario is left unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix fixture paths in clr-activation-debugging and analyzing-dotnet-performance evals The source paths referenced tests/dotnet/ but the fixtures live under tests/dotnet-diag/. This caused file-not-found exceptions at runtime, which set hasRejections and caused unconditional exit 1 (bypassing --verdict-warn-only). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Surface execution errors in validation summary Previously, skills that threw exceptions (e.g. missing fixture files) were silently excluded from the verdict count. The summary showed '2/4 skills passed' with no indication that 2 other skills crashed entirely. Now the summary shows '2/6 skills passed (2 rejected due to execution errors)' and lists the specific error messages near the summary rather than buried in the middle of the log. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reduce parallelism for infra-change evaluation runs When infrastructure changes trigger evaluation of all plugins simultaneously, the high default parallelism (5x5x5) causes API rate limits (429s) and scenario timeouts from contention. Reduce to 2x3x3 for infra-change runs only; regular single-plugin PR evaluations keep the default 5x5x5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Dan Moseley
|
042a9d2b2c |
Reorganize plugins into domain-specific groupings (#274)
* Reorganize plugins per agreed structure (#271) Split the dotnet plugin into domain-specific plugins per discussion in #271: - dotnet: narrowed to common everyday C#/.NET coding tasks - dotnet-diag: perf investigations, debugging, incident analysis - dotnet-data: data access and Entity Framework - dotnet-aspnet, dotnet-wpf, dotnet-winforms, dotnet-maui: scaffolded empty to convey intent for future framework-specific skills Moved skills, tests, and CODEOWNERS entries accordingly. No file content was changed; all skill/test moves are pure renames. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve dotnet-diag and dotnet-winforms descriptions - dotnet-diag: add '.NET' qualifier to clarify scope - dotnet-winforms: use official 'Windows Forms (WinForms)' name Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add .NET qualifier to dotnet-data description Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove empty plugins; list them in CONTRIBUTING.md as future options Remove the four empty placeholder plugins (dotnet-aspnet, dotnet-wpf, dotnet-winforms, dotnet-maui) that had only plugin.json and no skills. Add a 'Plugin organization' section to CONTRIBUTING.md that describes the domain-specific plugin groupings introduced in this PR and lists the four removed plugin names as reserved candidates for future contributions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Create dotnet-upgrade plugin with migration/upgrade skills Move migrate-nullable-references, thread-abort-migration, and dotnet-aot-compat from dotnet to the new dotnet-upgrade plugin. Update CODEOWNERS paths (same owners), CONTRIBUTING.md plugin table, and fix the thread-abort-migration SKILL.md link in CONTRIBUTING.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |