mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
eval-exec-opus48
589 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jeff Schwartz
|
55a7d80527 |
adding SIMD/TensorPrimitives to dotent-diag/analyzing-dotnet-performance (#330)
* incorporated feedback from previous pr, iterated with 100+ scalar loop tests to come to this skill, moved from skill to reference in the performance skill * Update tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address review feedback on SIMD vectorization reference - Check Span<T>/MemoryExtensions before TensorPrimitives (no extra dependency) - Add all supported types (sbyte, ushort, uint, ulong, nint, nuint, char via ushort) - TensorPrimitives: add constraint and applicable types columns (not just float/double) - Add FusedMultiplyAdd, clarify AddMultiply vs MultiplyAdd distinction - Prefer portable APIs over platform-specific intrinsics (allow when perf justifies) - Fix dispatch pattern to use if/else if to avoid pessimizing small inputs - Remove LINQ references Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * addressing more feedback * move to the experimental plugin * move the simd tests to exprimental * add codeowners --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
1d56634150 | Create experimental plugin (#446) | ||
|
Dan Moseley
|
b39fe8541b |
Add evaluation troubleshooting guide for AI agents (#441)
* Add evaluation troubleshooting guide and link from PR comments
Add InvestigatingResults.md with:
- How to find and download evaluation artifacts from workflow runs
- Full results.json schema documentation
- Common failure patterns (timeouts, empty output, variance, rubric
issues, skill not activated) with diagnosis steps and fixes
- Example analysis script for AI agents to parse results
Link the doc from:
- Reporter.cs: show a link in the PR comment when any skill fails
- skill-validator README.md: Output section
- CONTRIBUTING.md: CI evaluation section
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update InvestigatingResults.md
* Address review feedback: fix hard-coded URLs and optional plugin run docs
- Reporter.cs: Use repo-relative link instead of hard-coded GitHub URL
- evaluation.yml: Use github.sha instead of hard-coded 'main' branch
- InvestigatingResults.md: Clarify skilledPlugin run is optional
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* THROWAWAY: add trailing newline to trigger eval workflow
This change is intentionally trivial and will be reverted before merge.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR #441 review feedback (round 2)
- Reporter.cs: Remove leading / from relative link
- evaluation.yml: Use jq on results.json instead of grepping for emoji
- evaluation.yml: Use needs.gate.outputs.head_sha for revision-pinned URL
- InvestigatingResults.md: Mark pluginImprovementScore/pluginBreakdown as optional
- InvestigatingResults.md: Fix skill-not-activated symptoms to use
skillActivation fields instead of turnCount
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use specific skillActivation field names and fix timestamp description
- Use skillActivationIsolated/skillActivationPlugin instead of generic skillActivation
- Clarify timestamp is when results were written, not run start
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix timestamp field description (was lost in prior commit)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Revert "THROWAWAY: add trailing newline to trigger eval workflow"
This reverts commit
|
||
|
Viktor Hofer
|
1762a78fed |
Add stale PRs agentic workflow (#435)
* Add stale PRs agentic workflow * Add fork-guard to close-stale-prs workflow Skip scheduled runs on forked repositories to avoid consuming fork owner minutes and failing due to missing secrets/context, matching the pattern used by other scheduled workflows in this repo. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Dan Moseley
|
20ecb423c0 |
Include actionable guidance in scenario timeout warnings (#438)
* Include actionable guidance in scenario timeout warnings The timeout messages now show the actual timeout value and how to increase it via the 'timeout' field in eval.yaml. This applies to the console summary, per-scenario detail, and markdown/PR comment reporters. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Show actual timeout values in summary and markdown messages Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: nullable TimeoutSeconds, centralize default constant, fix grammar Co-authored-by: danmoseley <6385855+danmoseley@users.noreply.github.com> Agent-Logs-Url: https://github.com/dotnet/skills/sessions/478c8093-31bd-42df-b8d0-f75e4263d6d4 * Delete .nuget/nuget.exe --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: danmoseley <6385855+danmoseley@users.noreply.github.com> Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com> |
||
|
Jan Krivanek
|
c94bc5cdc3 |
Add Ability to author agent evals (#436)
* Add ability to author agent evals * Adjust |
||
|
Amaury Levé
|
17fe01321d | Promote crap-score and test-anti-patterns from experimental to dotnet-test (#434) | ||
|
Jan Krivanek
|
19b6cadddf |
Move token usage data into separate branch (#431)
* Move token usage data into separate branch * Address PR review: add help doc, fix title injection, add token-usage fallback * Add -SkipBenchmarkData switch, clean stale files on branch push |
||
|
Jan Krivanek
|
22542a5d75 |
Track and show subagents invocations (#427)
* Show subagents invocations * Address PR feedback: fix XML doc and sanitize agent names in markdown - Update ExtractSubagentActivation XML doc to accurately describe that all subagent event types are counted (not just Started/Completed) - Add markdown table cell sanitization for agent names in FormatSubagentCell to prevent injection via pipe chars or newlines in plugin-controlled content - Extract SanitizeMarkdownTableCell helper for reuse |
||
|
Viktor Hofer
|
c14e63ba6c |
Combine evaluation.yml and evaluation-run.yml into single workflow (#430)
* Combine evaluation.yml and evaluation-run.yml into single workflow Inline the three jobs from the reusable evaluation-run.yml workflow (build-validator, evaluate, comment-on-pr) directly into evaluation.yml and delete the now-unnecessary reusable workflow file. Changes: - Inline build-validator, evaluate, and comment-on-pr jobs - Eliminate 8-secret forwarding block (direct secrets.* access) - Replace inputs.* with direct needs.* references - Add MODEL/JUDGE_MODEL workflow-level env vars - Fix --verbose duplication in evaluate step - Simplify infra-change detection regex - Update dependency chains (report-status, publish-benchmark) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR feedback: fix consolidate subcommand, improve error reporting - Fix consolidate invocation: use 'evaluate consolidate' subcommand (was incorrectly changed to just 'consolidate' during inlining) - Add build-validator to report-status needs so build failures surface as a deterministic failure state instead of generic error - Add discover to comment-on-pr needs and gate on has_entries so /evaluate on a no-op PR doesn't post a misleading failure comment Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix matrix fallback and header comment - Use fromJson(... || '[]') fallback for matrix entries to prevent errors when discover is skipped on non-evaluation events - Update header comment to accurately describe the is_fork mechanism Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
b92dde1854 |
Refactor skill-validator to vertical slice architecture (#429)
Reorganize the skill-validator project from layer-based grouping (Commands/, Services/, Models/, Utilities/) to feature-based slices (Check/, Evaluate/, Consolidate/, Shared/). Key changes: - Check/: CheckCommand, SkillProfiler, AgentProfiler, PluginValidator, ExternalDependencyChecker, ReferenceScanner - Evaluate/: EvaluateCommand, RejudgeCommand (now a subcommand of evaluate), AgentRunner, Judge, PairwiseJudge, and other eval services - Consolidate/: ConsolidateCommand - Shared/: Models, SkillDiscovery, PluginParser (extracted from PluginValidator), Reporter, and utilities Split PluginValidator: parsing helpers (ParsePluginJson, TryGetSafeSubdirectory) moved to Shared/PluginParser.cs; validation logic stays in Check/PluginValidator.cs. Move RawFrontmatter and RawAgentFrontmatter types from EvalSchema into SkillDiscovery since they are only consumed by discovery. Wire RejudgeCommand as subcommand: skill-validator evaluate rejudge. Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
d97128e5d9 |
Move static analysis checks from eval to check command (#425)
* Move static analysis checks from eval to check command Remove SkillProfiler.AnalyzeSkill from the evaluate command's per-skill flow. All spec conformance checks (name, description, compatibility, body line count, file reference depth, token size warnings, etc.) are already enforced by the check command — running them again in eval was redundant. The eval-specific 'skill name in eval prompt' check is extracted into EvaluateCommand.ValidateEvalPrompts and kept in eval since it requires eval config context. The evalConfig parameter is removed from SkillProfiler.AnalyzeSkill along with the dead ResourceFileCount field. Tests updated to exercise ValidateEvalPrompts directly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [WIP] [WIP] Address feedback: Remove ProfileWarnings from vedict model (#426) * Initial plan * Remove ProfileWarnings from verdict model and reporter output Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> Agent-Logs-Url: https://github.com/dotnet/skills/sessions/9b7c3d0a-1693-4238-a397-2200f83634b5 --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> --------- Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
c4c26175e2 |
Integrate reference scanner into skill-validator CheckCommand (#422)
* Integrate reference scanner into skill-validator CheckCommand - Port reference scanner logic from scan.ps1 to C# ReferenceScanner service - Add --known-domains option to CheckCommand for reference scanning - Move known-domains.txt and allowed-external-deps.txt to eng/ - Remove standalone scan.ps1 and reference-validation.yml workflow - Reference scanning now runs as part of skill-check.yml via skill-validator - Add 34 unit tests for ReferenceScanner (all 463 tests pass) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR feedback: repoRoot derivation, early validation, .agents scanning, SRI policy - Derive repoRoot from --known-domains file path instead of cwd - Fail early with clear error when known-domains file is missing - Add .agents/**/*.md scanning to DiscoverFiles (regression fix) - Fix SRI policy loophole: SRI suppresses SCRIPT-NO-SRI only, domain and HTTPS checks still apply - Use actual known-domains file path in error messages instead of hardcoded eng/known-domains.txt - Update doc comments to reflect full set of scanned paths - Add tests for .agents discovery and SRI domain enforcement Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove repo-specific paths from ReferenceScanner DiscoverFiles now only scans directories it is given -- no hardcoded knowledge of .agents, agentic-workflows, eng, or README.md. The tool is repo-agnostic; callers control what gets scanned via --plugin dirs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Drop repoRoot from ReferenceScanner API ScanFile/ScanFiles no longer take a repoRoot parameter. Finding paths use the file path as provided by the caller, keeping the scanner completely decoupled from any repo layout assumptions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Prune stale domains and add json.schemastore.org to known-domains.txt Removed 10 entries no longer referenced by any plugin content: agentskills.io, visualstudio.com, mcr.microsoft.com, dotnet.github.io, github.com/dotnet/docs, github.com/dotnet/dotnet-api-docs, github.com/dotnet/skills, github.com/github/gh-aw, github.com/modelcontextprotocol/csharp-sdk, developers.openai.com. Added json.schemastore.org (referenced by dotnet-template-engine). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix case-insensitive regex, update docs, add tests - Make UrlRegex and PipeToShellRegex case-insensitive to match the original PowerShell behavior (HTTP://, CURL, Wget all now detected) - Update DiscoverFiles doc comment to remove stale 'individual files' claim - Narrow CONTRIBUTING.md scope description to match actual plugin-only scanning - Add tests for mixed/upper-case URL and pipe-to-shell detection Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove IPv6 loopback support, fix pipe-to-shell allowlist case sensitivity - Remove Ipv6LoopbackRegex and IPv6 test data (not needed) - Add case-insensitive + culture-invariant matching for pipe-to-shell allowlist - Add mixed-case allowlist test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
dependabot[bot]
|
b720aa06d3 |
Bump System.Text.Json from 8.0.4 to 8.0.5 (#411)
--- updated-dependencies: - dependency-name: System.Text.Json dependency-version: 8.0.5 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Amaury Levé
|
e354d43534 |
Improve test skills (#423)
* Increase timeout * Fix activation issues * Improve further |
||
|
Jan Krivanek
|
fcfa85a221 | Mock out exponential backoff unit test (#421) | ||
|
Viktor Hofer
|
a895c226b3 |
Fix agents field in plugin.json to use explicit file paths (#419)
* Fix agents field in plugin.json to use explicit file paths Claude Code's manifest validator rejects directory paths in the agents field. Replace directory references with explicit .agent.md file paths in all affected plugins (dotnet-diag, dotnet-msbuild, dotnet-template-engine). Fixes https://github.com/dotnet/skills/issues/416 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Validate that agents paths are explicit files, not directories Claude Code's manifest validator rejects directory paths in the agents field. Update PluginValidator to error when an agent path resolves to a directory, guiding authors to use explicit .agent.md file paths instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Youssef Victor
|
c4937f8f68 |
Add 'run_command_and_assert' assertion (#401)
* Add 'run_command_and_assert' assertion * Rename for clarity * Add tests * Fix race condition in command output handling * Fix async output race condition in RunCommandAndAssert WaitForExit(TimeSpan) does not guarantee that async output event handlers have completed. Add parameterless WaitForExit() call after the timed wait to ensure stdout/stderr are fully captured before checking their contents. Also handle the timeout case explicitly by killing the process. * Address PR review: extract CommandAssertionArgs, add regex matching, configurable timeout - Extract command-specific fields from Assertion into a dedicated CommandAssertionArgs record type for cleaner separation of concerns - Add ExpectedStdOutMatches/ExpectedStdErrorMatches for regex-based stdout/stderr assertions (matches type asserts) - Make command timeout configurable via assertion-level 'command_timeout' field, falling back to the scenario timeout when not specified - Show actual timeout value in timeout error messages - Update EvalSchema parsing, EvaluateCommand call sites, and all tests - Add new tests for regex matching and timeout configuration --------- Co-authored-by: Jan Krivanek <jankrivanek@microsoft.com> |
||
|
Amaury Levé
|
0846e6015d | Improve MSTest migration skills (#387) | ||
|
Amaury Levé
|
9adebf86a8 | Add experimental CRAP score skill for dotnet-test (#413) | ||
|
Amaury Levé
|
00c1e82006 |
Add exp-test-tagging skill to dotnet-test-experimental (#417)
* Add exp-test-tagging skill and evaluation Experimental skill for analyzing and tagging test suites with a standardized trait taxonomy (positive, negative, boundary, critical-path, smoke, regression, integration, end-to-end, performance, security, concurrency, resilience, destructive, configuration, flaky). Supports MSTest, xUnit, and NUnit with framework-specific attribute syntax. Includes 5 eval scenarios: tag MSTest/xUnit/NUnit, audit-only, non-activation. * Move exp-test-tagging to dotnet-test-experimental plugin |
||
|
Amaury Levé
|
6489fbee39 |
Add exp-test-anti-patterns skill for test quality review (#415)
New experimental skill that detects anti-patterns and code smells in .NET test suites. Covers assertion quality, test isolation, naming, flakiness indicators, over-mocking, and structural problems across MSTest, xUnit, NUnit, and TUnit. Key features: - 4-tier severity classification (Critical/High/Medium/Low) - False-positive avoidance: calibrates severity honestly and recognizes well-written tests without inventing problems - Positive observations alongside findings Evaluation results (3 runs, claude-opus-4.6): - Flakiness/coupling detection: 3.0 -> 5.0 (isolated), 4.0 (plugin) - Duplicates/magic values: 3.0 -> 5.0 (isolated), 4.7 (plugin) - Well-written test recognition: 2.0 -> 5.0 (both modes) - Mixed severity detection: 5.0 baseline maintained - Overfitting score: 0.06 across all scenarios |
||
|
Amaury Levé
|
e0e1a91d90 |
Extract shared references and add cross-refs for dotnet-test skills (#414)
* Extract shared references and add cross-refs for dotnet-test skills - Extract platform detection logic into run-tests/references/platform-detection.md (shared by run-tests, mtp-hot-reload, migrate-vstest-to-mtp) - Extract filter syntax reference into run-tests/references/filter-syntax.md (shared by run-tests, mtp-hot-reload, migrate-vstest-to-mtp) - Add cross-references between skills in When Not to Use / Related Skills sections - run-tests SKILL.md reduced from 364 to 191 lines * Fixes |
||
|
Copilot
|
a68fa8da18 |
Add Codex CLI installation docs to README (#408)
* Initial plan * Add Codex CLI installation section to README and allow developers.openai.com domain Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Remove Codex CLI example per review feedback Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Copilot
|
6face98ff6 |
Change agents and skills fields in plugin.json to array format for Claude Code CLI compatibility (#407)
* Initial plan * Remove agents string field from plugin.json and handle array format in parser Claude Code CLI validates `agents` as array-of-strings per its plugin manifest schema. Our plugin.json files had `"agents": "./agents/"` (a string), which caused "agents: Invalid input" validation errors on install. Remove the field from the 3 affected plugin.json files — agents are still discovered by convention from the `agents/` directory. Also make ParsePluginJson resilient to both string and array formats for forward-compatibility. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Remove `agents` string field from plugin.json for Claude Code CLI compatibility Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Change agents field to array format in plugin.json, update validator to read array-first with string fallback Plugin.json files now declare agents as an array of file paths: "agents": ["./agents/foo.agent.md", "./agents/bar.agent.md"] ParsePluginJson reads the array first (preferred, Claude Code schema), falls back to a string path (legacy). PluginInfo gains an AgentPaths field. Validator and discovery use the array when present, else fall back to directory-based convention. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Simplify agents to directory-path array form ["./agents/"], handle directories in array entries Per Copilot CLI spec, agents field values are "Path(s) to agent directories" — no need to list individual files. Use ["./agents/"] to mirror how skills uses "./skills/". Updated validator and discovery to handle both directory and file entries in the array. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Delete .nuget/nuget.exe * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Convert skills field to array format in plugin.json, update validator to handle both forms Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Align AgentPaths null checks to use { Count: > 0 } pattern for consistency Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Change `agents` and `skills` fields in plugin.json to array format for Claude Code CLI compatibility Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Delete .nuget/nuget.exe * Consolidate SkillsPath/SkillPaths and AgentsPath/AgentPaths into single array fields Remove dual string/array fields from PluginInfo. The parser now normalizes legacy string values into single-element arrays, so all downstream code uses only SkillPaths and AgentPaths (IReadOnlyList<string>). This eliminates all array-vs-string fallback branching. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
29a9dd1202 | Add eval test authoring skill (#409) | ||
|
Jan Krivanek
|
68d83b6ea8 | Add parallelism configuration options to eval.yaml (#410) | ||
|
Jan Krivanek
|
101a151e98 | Improve the build-perf-diag scenario test (#404) | ||
|
Jan Krivanek
|
cc8010f357 | Include judge tokens, fix summaries (#405) | ||
|
YuliiaKovalova
|
dcba4f3d8e |
Add dotnet-template-engine plugin (#341)
* Add dotnet-template-engine plugin Add template-engine skills for .NET project scaffolding powered by the DotnetTemplateMCP MCP server (v1.1.0). Includes three skills and one agent: Skills: - template-discovery: find, inspect, and compare templates with intent resolution (70+ keyword mappings) - template-instantiation: create projects with CPM adaptation, latest NuGet versions, parameter validation, and multi-template composition - template-authoring: create templates from existing projects and validate template.json for authoring issues Agent: - template-engine: expert agent that routes to the appropriate skill based on user intent Also adds CODEOWNERS entries, marketplace.json registration, eval tests, and .mcp.json for DotnetTemplateMCP server configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: mcpServers in plugin.json, expect_tools, path fix - Move MCP server config into plugin.json under mcpServers so SkillValidator discovers the template_* tools during evaluation runs - Add expect_tools constraints to all eval scenarios to verify actual MCP tool invocation instead of relying on rubric text alone - Fix frontmatter path: template.config → .template.config (leading dot) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update DotnetTemplateMCP to v1.2.0 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Include 'dotnet new' in agent description for discoverability Address review feedback from danmoseley: the frontmatter description is how the caller decides whether to load the agent tokens, so it needs to mention 'dotnet new' as a trigger keyword. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove MCP server configuration and references - Delete .mcp.json MCP server config file - Remove mcpServers section from plugin.json - Remove MCP/DotnetTemplateMCP references from agent, skills, and marketplace descriptions - Skills and eval files remain unchanged Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove ghost tool references and add CLI fallback guidance - Remove references to 4 non-existent tools: template_compose, template_suggest_parameters, template_validate, solution_analyze - Replace with real alternatives (template_inspect, manual review, sequential template_instantiate, direct file inspection) - Add CLI fallback guidance so LLMs degrade gracefully when template tools are unavailable - Fix authoring eval to not expect template_validate tool Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: CODEOWNERS naming, claude marketplace, README - Rename CODEOWNERS header to dotnet-template-engine for consistency - Mirror plugin entry in .claude-plugin/marketplace.json - Add plugin to README What's Included table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix PR review comments: align eval rubric with expect_tools and sync marketplace.json - Updated template-discovery eval.yaml rubric to match expect_tools (removed 'or template_search' since expect_tools enforces template_from_intent) - Made .claude-plugin/marketplace.json byte-for-byte identical to .github/plugin/marketplace.json (removed extra trailing newline) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review comments: strengthen eval assertions and fix rubrics - Add exit_success assertions to all eval scenarios - Add file_exists and file_contains assertions for template-instantiation - Add output_contains for shortName and datatype in template-authoring - Fix rubrics to reference actual tools (skill + dotnet new CLI) instead of non-existent MCP tools (template_from_intent, template_inspect, template_instantiate) which caused the judge to penalize quality scores - Update expect_tools from non-existent MCP tools to 'skill' - Change plugin version from 1.2.0 to 0.1.0 for consistency with other plugins - Remove extra blank line in README.md after plugin table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rewrite skills and agent to use dotnet new CLI instead of phantom MCP tools The SKILL.md files and agent.md referenced MCP tools (template_from_intent, template_inspect, template_instantiate, template_dry_run, etc.) that are not available at runtime since no MCP server is declared in plugin.json. This caused the agent to waste tokens attempting to find these tools before falling back to dotnet new CLI commands. Changes: - template-discovery/SKILL.md: Replace MCP tool calls with dotnet new search, dotnet new list, dotnet new --help, and dotnet new --dry-run. Add intent-to-template keyword mapping table. - template-instantiation/SKILL.md: Replace template_instantiate with dotnet new CLI commands. Add explicit CPM adaptation steps and multi-project composition with dotnet sln/dotnet add reference. - template-authoring/SKILL.md: Replace template_create_from_existing with manual template.json creation guidance including example JSON. Replace template_dry_run/template_instantiate with CLI equivalents. - template-engine.agent.md: Replace Available Tools table with CLI Commands Reference. Update all workflow steps to use CLI commands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix file assertion paths: remove ./ prefix for glob matcher compatibility Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> |
||
|
Jan Krivanek
|
defd58af6a |
Add proper skills isolation (copying and allowlisting) (#402)
* Add proper skills isolation (copying and allowlisting) * Address PR review: harden isolation, skip symlinks, precompute allowlist - Pass null skillPath in isolated mode so the original skill tree (including siblings) is not accessible; only staged copies are allowed. - Harden CopyDirectory to skip symlinks/reparse points and verify that recursed paths stay within the source root. - Precompute additionalAllowedDirs once at session creation instead of allocating on every permission request. - Add IsolatedStagingDoesNotExposeOriginalOrSiblingSkills test that verifies both directory-level and permission-level isolation. * Fix unit test |
||
|
Jan Krivanek
|
30c19eae13 | Add tokens usage dashboard and data collection (#385) | ||
|
dependabot[bot]
|
243f35a4bd |
Bump the microsoft-and-system group with 4 updates (#397)
Bumps Microsoft.Bcl.Memory from 9.0.14 to 10.0.5 Bumps Microsoft.Data.Sqlite from 10.0.0 to 10.0.5 Bumps Microsoft.Extensions.FileSystemGlobbing from 10.0.4 to 10.0.5 Bumps System.CommandLine from 2.0.4 to 2.0.5 --- updated-dependencies: - dependency-name: Microsoft.Bcl.Memory dependency-version: 10.0.5 dependency-type: direct:production update-type: version-update:semver-major dependency-group: microsoft-and-system - dependency-name: Microsoft.Data.Sqlite dependency-version: 10.0.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Extensions.FileSystemGlobbing dependency-version: 10.0.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: System.CommandLine dependency-version: 2.0.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Bcl.Memory dependency-version: 10.0.5 dependency-type: direct:production update-type: version-update:semver-major dependency-group: microsoft-and-system - dependency-name: Microsoft.Data.Sqlite dependency-version: 10.0.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Extensions.FileSystemGlobbing dependency-version: 10.0.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: System.CommandLine dependency-version: 2.0.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> |
||
|
dependabot[bot]
|
2531f0f374 |
Bump Azure.Identity from 1.10.0 to 1.11.4 (#399)
--- updated-dependencies: - dependency-name: Azure.Identity dependency-version: 1.11.4 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> |
||
|
dependabot[bot]
|
8e27c14ea4 |
Bump actions/download-artifact in the github-actions-dependencies group (#396)
Bumps the github-actions-dependencies group with 1 update: [actions/download-artifact](https://github.com/actions/download-artifact). Updates `actions/download-artifact` from 8.0.0 to 8.0.1 - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](https://github.com/actions/download-artifact/compare/70fc10c6e5e1ce46ad2ea6f2b72d43f7d47b13c3...3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c) --- updated-dependencies: - dependency-name: actions/download-artifact dependency-version: 8.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> |
||
|
Youssef Victor
|
f75fd8840c | Create SECURITY.md (#400) | ||
|
Viktor Hofer
|
c5435dd3da |
Make evaluate a subcommand with tests-dir as required argument (#398)
* Make evaluate a subcommand with tests-dir as required argument
Change EvaluateCommand from RootCommand to a regular Command('evaluate')
so it is a subcommand like check, consolidate, and rejudge. Convert
--tests-dir from an optional Option to a required positional Argument
placed after the skill paths.
Update .csproj RunArguments, evaluation-run workflow, README, and
CONTRIBUTING.md to reflect the new CLI syntax.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Default --parallel-skills/scenarios/runs to 3
Update the default from 1 to 3 for all three parallelism options and
remove the now-redundant overrides from RunArguments in the .csproj.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Replace RunArguments with launchSettings.json, publish in skill-check
Add Properties/launchSettings.json with default evaluate args for local
development. Remove RunArguments from .csproj. Update skill-check.yml to
publish the validator and invoke the binary directly instead of using
dotnet run.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Keep --tests-dir as a required option, not positional
A variadic paths argument greedily consumes all positional values,
making a trailing positional tests-dir impossible to parse. Keep
--tests-dir as a named required option which matches the previous UX.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move testsDirOpt after resultsDirOpt, drop evaluate from CONTRIBUTING examples
Reorder testsDirOpt declaration and registration to follow resultsDirOpt.
Remove the evaluate subcommand from CONTRIBUTING.md dotnet run examples
since the launch profile already provides it.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
Viktor Hofer
|
bd965cbd55 |
Require /evaluate command for PR evaluation (#394)
* Require /evaluate command for PR evaluation Update evaluation.yml to require an /evaluate command to trigger evaluation, matching the behavior in evaluation-fork-pr.yml. This change: - Adds pr-status job that posts initial commit status on PR open/sync: - 'success' if no skills changed (PR not blocked) - 'pending' if skills changed (maintainer must post /evaluate) - Adds gate job to validate /evaluate command (permissions, same-repo check) - Replaces evaluation-status check-run with report-status using commit status API - Scheduled runs continue to work unchanged (daily evaluation of all plugins) The evaluation-status required check still works because: - PRs without skill changes get immediate success status - PRs with skill changes get pending status until /evaluate is posted Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Merge evaluation and evaluation-fork-pr into single workflow Consolidate evaluation.yml and evaluation-fork-pr.yml into one unified workflow that handles both same-repo and fork PRs via conditionals: - pr-status: posts initial commit status for same-repo PRs (pull_request) - fork-pr-status: posts initial commit status for fork PRs (pull_request_target) - gate: unified /evaluate command validation with is_fork output - discover: unified skill discovery with conditional worktree for forks - report-status: single job for posting final commit status - Scheduled runs and publish-benchmark unchanged Also fixes: - run-evaluation now uses always() so scheduled runs aren't skipped - report-status scoped to issue_comment events only - Removed stale evaluation-fork-pr.yml reference from infra detection regex - Updated evaluation-run.yml comment Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback - Fetch PR head and use worktree for both fork and same-repo PRs in discover - Always pass checkout-ref (PR head SHA) to evaluation-run for /evaluate - Fix runs count: use event_name == 'schedule' instead of github.ref Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Require /evaluate for infra changes too Expand pr-status and fork-pr-status discovery to detect evaluation workflow and skill-validator changes in addition to skill/test changes. Both now post pending status requiring /evaluate when infra files are modified. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Build skill-validator from PR branch for non-fork PRs Add is-fork input to evaluation-run.yml. When false (same-repo PRs), the validator is built from checkout-ref so skill-validator changes are tested. When true (fork PRs), the validator is built from the base branch to prevent untrusted code from modifying tooling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: fix security comment, add issues:write - Update security model comment to reflect validator-ref behavior - Add issues: write permission to gate and report-status jobs for reaction and comment API calls Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Add prominent note about issue_comment trigger limitation Changes to this workflow file in a PR won't take effect until merged because issue_comment always runs from the default branch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Don't require /evaluate for workflow YAML changes Workflow YAML changes can't be tested via /evaluate anyway (issue_comment always runs from main). Only eng/skill-validator/ changes now require /evaluate in the pr-status jobs. The discover job still detects workflow changes to trigger all-plugin evaluation when /evaluate is run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Jeff Handley
|
646b5d93b0 |
Add Skill: Convert to Central Package Management (convert-to-cpm) (#75)
* Add convert-to-cpm skill for NuGet Central Package Management conversion Authoring process: - Researched the Agent Skills standard at agentskills.io and reviewed repo conventions - Studied the existing csharp-scripts skill and test structure as a reference implementation - Studied Microsoft's CPM documentation at learn.microsoft.com - Authored SKILL.md with a 9-step workflow covering scope determination through validation - Created 5 test scenarios across simple, moderate, and advanced tiers - Performed dry-run evaluation against all scenarios, identifying 15 findings - Applied all 15 fixes (4 gaps, 6 ambiguities, 5 improvements) to SKILL.md - Re-evaluated: 50 OK, 0 gaps, 0 ambiguities, 1 test fixture improvement - Updated test fixture to resolve the remaining finding - Final validation: 254 lines, 0 errors, 0 warnings Skill implementation (skills/convert-to-cpm/SKILL.md): - 9-step workflow: scope, CPM check, audit, create/update props, update projects, handle properties, restore/validate, cleanup, summary - Handles single projects, solutions, and full repositories - Audits packages with per-project detail table including version conflicts and security advisories - Flags CVE/security vulnerabilities and recommends patched versions - Supports MSBuild property-based versioning with inline vs keep decisions - Covers conditional PackageReference items across target frameworks - Discovers PackageReference in imported .props/.targets via Import chain scanning - Provides cross-platform commands (Unix and Windows/PowerShell) - Documents import order requirements for property references in Directory.Packages.props - Mentions dotnet new packagesprops as preferred creation method (.NET 8+) - Includes validation checklist and common pitfalls table Test scenarios (tests/convert-to-cpm/): - simple-single-project: 1 project, 3 packages, no conflicts - simple-solution: 3 projects via .sln, 5 packages, PrivateAssets preservation - moderate-version-conflicts: 4 projects, System.Text.Json version conflicts with CVE-2024-43485 advisory, AutoMapper major version conflict with VersionOverride - moderate-msbuild-properties: 2 projects using MSBuild properties for versioning in Directory.Build.props, inline vs keep decisions, property cleanup verification - advanced-multi-complexity: 5 projects combining version conflicts, MSBuild properties, conditional TFM PackageReference, shared Common.props imports, VersionOverride, and security advisory handling Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor SKILL.md with references, baseline build, and package comparison Condensed SKILL.md using progressive disclosure: - Reduced from 254 lines / ~3937 tokens to 120 lines / ~1778 tokens - Extracted detailed procedural instructions into 5 focused reference files - Main file retains workflow decision logic with links to references loaded on demand - Follows agentskills.io progressive disclosure pattern: metadata -> instructions -> resources Added baseline build verification (new Step 2): - Requires clean build (dotnet clean && dotnet build) before any changes - Captures baseline.binlog for structured build artifact - Captures baseline-packages.json via dotnet list package --format json - Blocks conversion if baseline build fails Added post-conversion baseline comparison (Steps 8 and 10): - Step 8 produces after-cpm.binlog and after-cpm-packages.json from clean build - Step 10 compares baseline vs post-conversion package lists per project - Changes table shows version bumps, VersionOverride entries, added/removed packages - Unchanged table confirms version-neutral conversion for unmodified packages - Binlog files produced as artifacts for user review in MSBuild Structured Log Viewer Reference files created: - references/audit-complexities.md: 7 complexity categories for package auditing - references/baseline-comparison.md: dotnet list package comparison procedure, binlog production, artifact guidance, table formats for changes and unchanged packages - references/directory-packages-props.md: File creation (dotnet new packagesprops), placement rules, conditional versions, VersionOverride patterns - references/msbuild-property-handling.md: Property usage search, inline vs keep decisions, import order requirements, cleanup verification - references/validation-and-errors.md: NuGet error codes (NU1008, NU1010, NU1507), clean build validation, multi-TFM guidance, common pitfalls Workflow is now 10 steps (was 9): 1. Determine scope 2. Establish baseline build (new) 3. Check for existing CPM 4. Audit package references 5. Create or update Directory.Packages.props 6. Update project files 7. Handle MSBuild version properties 8. Restore and validate (now with binlog + package list capture) 9. Clean up obsolete properties 10. Summary and baseline comparison (expanded with diff tables) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace third-party packages with Microsoft equivalents, add baseline/comparison steps, add .gitignore for test solutions Package replacements across all skill references and test scenarios: - Serilog.AspNetCore -> OpenTelemetry.Extensions.Hosting 1.15.0 - Serilog (standalone) -> OpenTelemetry.Extensions.Hosting 1.15.0 - Swashbuckle.AspNetCore -> Microsoft.AspNetCore.OpenApi 8.0.24 - SerilogVersion MSBuild property -> OTelVersion - DIVersion updated from 8.0.1 to 9.0.0 (DI/Abstractions alignment) - Microsoft.AspNetCore.OpenApi uses 8.0.24 (not 9.x) for net8.0 compat Test fixture improvements: - Added Step 2 (baseline build with binlog) to all 5 with-skill.md files - Added Step 10 (package comparison table + binlog artifact links) to all 5 with-skill.md files - Expanded simple-solution without-skill.md with more realistic agent behavior - Standardized security fix terminology: table markers cite CVEs, prose references CVE IDs - Removed space-padded table formatting in favor of compact markdown with emojis Test solution infrastructure: - Added tests/convert-to-cpm/.gitignore to exclude src/ build artifacts - Added generation prompt to test README for recreating buildable solutions - All 5 test solutions verified to restore and build successfully Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback on terminology, placement, and reference accuracy - Rename Step 10 and validation checklist from 'binlog comparison' to 'package list comparison' - Rewrite baseline-comparison.md: title to 'Baseline Comparison', lead with dotnet list package as primary comparison mechanism, position binlogs as supplementary artifacts for manual review/troubleshooting - Add brief definition of binlogs before the artifacts section - Fix Directory.Packages.props placement guidance: use first common ancestor of all .NET projects instead of assuming repository root (.git), since many repos nest source under src/ or similar - Update SKILL.md scope input and Step 1 to reference directory-scoped conversion - Standardize security fix terminology: table markers cite CVEs, prose uses 'security fix for CVE-...' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review findings: sort PackageVersion, add packages.config scenario, fix commands Review findings addressed: 1. Fixed BlobsVersion grep step description in advanced scenario 2. Sorted PackageVersion entries alphabetically in all Directory.Packages.props outputs 3. Consolidated baseline-comparison.md to single-table format 4. Added solution file arguments and multi-line commands in test fixtures 5. Added Directory.Packages.props placement question for single-project scenario 6. Created simple-packages-config test scenario (packages.config rejection) 7. Use dotnet new packagesprops in simple-single-project workflow 8. Fixed More Info link text to match reference file title Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix test scenario accuracy: sln/slnx mix, package counts, ImplicitUsings - Use mix of .sln and .slnx across scenarios (simple-solution and moderate-msbuild-properties use .sln; moderate-version-conflicts and advanced-multi-complexity use .slnx) - Fix simple-solution package count from 5 to 6 - Fix advanced user response #2 to reference Azure.Storage.Blobs version - Add ImplicitUsings to advanced Directory.Build.props in audit and cleanup - Complete the advanced unchanged table with all expected package entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add skill evaluation section to test README Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Inventory.sln references and make evaluation opt-in - Fix 4 occurrences of Inventory.sln to Inventory.slnx in moderate-version-conflicts/with-skill.md for consistency with the scenario README and test solution - Update test README to only run skill evaluation when the user explicitly requests it Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Condense SKILL.md with progressive disclosure - Remove duplicated bash code blocks from Steps 2 and 8, linking to baseline-comparison.md instead - Merge Steps 7 (MSBuild property decisions) and 9 (cleanup) into a single step since both reference msbuild-property-handling.md - Condense Step 10 summary to reference baseline-comparison.md for procedure details - Reduces from 120 to 97 lines (10 steps to 9 steps) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Enterprise.sln references to Enterprise.slnx in with-skill.md Address code review feedback: fix 4 occurrences of Enterprise.sln to Enterprise.slnx in advanced-multi-complexity/with-skill.md for consistency with the scenario README and test solution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve convert-to-cpm skill: preserve whitespace, save summary report, fix links - Instruct Step 6 to retain each file's existing indentation style and blank lines to keep diffs minimal - Instruct Step 9 to save the full summary as convert-to-cpm.md alongside binlog and JSON artifacts, suggesting its contents for a PR description - Strip en-us/ from learn.microsoft.com links in SKILL.md and validation-and-errors.md to make them culture-agnostic - Update all with-skill.md test expectations to include the convert-to-cpm.md create step and artifact mention Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Indicate that the binlog viewer is for the user; not the tool/skill * Move convert-to-cpm skill and tests to src/dotnet-msbuild layout Move skill from skills/convert-to-cpm/ to src/dotnet-msbuild/skills/convert-to-cpm/ and tests from tests/convert-to-cpm/ to src/dotnet-msbuild/tests/convert-to-cpm/ to match the repository layout established on main. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Convert tests from with/without-skill format to eval.yaml Replace manual test approach (README.md + with-skill.md + without-skill.md) with automated eval.yaml scenarios containing assertions and rubric items. Add committed test fixture files (.csproj, .sln, .slnx, .props) for each scenario with copy_test_files setup, replacing the git-ignored generated solutions. 6 scenarios converted: - simple-single-project: basic single project CPM conversion - simple-solution: multi-project solution conversion - simple-packages-config: negative test (packages.config rejection) - moderate-version-conflicts: version conflict resolution - moderate-msbuild-properties: MSBuild property version handling - advanced-multi-complexity: combined complexities (conflicts, properties, conditionals, shared props) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Consolidate eval.yaml files into single root eval.yaml Move all 6 scenario definitions from individual subdirectory eval.yaml files into a single src/dotnet-msbuild/tests/convert-to-cpm/eval.yaml file, matching the skill-validator discovery convention that expects {testsDir}/{skillname}/eval.yaml. Scenario prompts updated to reference subdirectory-prefixed paths since copy_test_files now copies all fixture subdirectories. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reorganize convert-to-cpm into plugins/tests folders * Add post-conversion reporting rubrics to CPM eval scenarios Add rubric items for baseline-vs-result comparison reporting, binlog capture and guidance, version reconciliation, and risk/reassurance messaging across all conversion scenarios. Increase complex scenario timeout to 360s. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve post-conversion reporting, conflict resolution, and skill activation SKILL.md: - Surface binlog capture commands explicitly in Steps 2 and 8 - Expand Step 4 with detailed version conflict presentation and resolution workflow - Restructure Step 9 into 5-section post-conversion report template: overview, conflict resolutions, baseline comparison, risk assessment, and artifacts/next steps with binlog education - Broaden activation triggers in description and When to Use to match update/bump/upgrade/align/sync terminology from ecosystem PRs - Add activation for out-of-sync/conflicting/mismatched version scenarios audit-complexities.md: - Expand version conflict section with per-conflict presentation format, 4 resolution options with trade-offs, and risk-level guidance eval.yaml: - Pre-specify conflict resolution strategy in prompts to avoid blocking on unanswered user prompts during single-turn evaluation - Add 3 new scenarios testing CPM recommendation when user asks to update/align packages without mentioning CPM - Update rubrics to match directed resolution strategy - Increase complex scenario timeout to 360s Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR feedback from kartheekp-ms - Use GitHub markdown URL for CPM docs reference (SKILL.md) - Use dotnet package list (renamed in .NET 10) with backward compat note for dotnet list package (SKILL.md, baseline-comparison.md) - Use dotnet package list --format json for auditing (SKILL.md Step 4) - Document Directory.Packages.props sub-folder support for test projects with different dependencies (directory-packages-props.md) - Remove security advisory version bumping from CPM conversion scope; instead surface advisories as follow-up items in the post-conversion report (audit-complexities.md, SKILL.md Step 9 Section 5) - Add Follow-up items section to report template covering security advisories, deprecated packages, version alignment opportunities, and test validation (SKILL.md Step 9) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve convert-to-cpm skill activation, artifact preservation, and eval assertions - Broaden skill description to trigger on version drift/sync scenarios and MSBuild property contexts even when CPM is not explicitly mentioned - Add explicit instructions to preserve binlog and JSON artifacts (not delete) - Make convert-to-cpm.md report a mandatory deliverable, not optional - Replace fragile output_contains assertions (ManagePackageVersionsCentrally, Directory.Packages.props) with file_contains checks on the actual generated files across 5 conversion scenarios Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Extend timeouts and add CODEOWNERS entry * Move convert-to-cpm skill into a new dotnet-nuget plugin * Fix erroneous plugin msbuild/nuget plugin split change. Add dotnet-nuget to marketplace. * Add NuGet package management skills to README * Address PR feedback: clarify CPM sub-folder semantics, remove security-fix example, require explicit conflict strategy - directory-packages-props.md: Clarify that sub-folder Directory.Packages.props files are independent and replace (not extend) the parent; require explicit MSBuild Import to chain files - baseline-comparison.md: Replace security-fix upgrade example with version conflict alignment example, consistent with audit guidance that security advisories are follow-up items - SKILL.md: Remove default 'use highest version' conflict strategy; require explicit user choice or confirmation when conflicts are detected Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Extend eval timeouts for scenarios hitting time limits - Convert single project: 180s -> 270s - Recommend CPM (version conflicts): 180s -> 270s - Recommend CPM (complex repo): 360s -> 540s - Convert complex repository: 360s -> 540s Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add missing CODEOWNERS for dotnet-nuget plugin. Reorder convert-to-cpm evals and expand timeouts. * Remove containing skill mentions from reference files; fix ordering * Replace output_matches with deterministic file assertions in eval.yaml Convert 6 non-deterministic output_matches assertions to 8 file-based assertions (file_contains, file_not_contains) across conversion scenarios: - Scenario 5: Verify PrivateAssets preserved in Tests.csproj - Scenario 6: Verify OTelVersion/DIVersion removed from Directory.Build.props - Scenario 7: Verify System.Text.Json and Azure.Identity in Directory.Packages.props - Scenario 8: Verify System.Text.Json in Directory.Packages.props and BlobsVersion/HostingVersion removed from Directory.Build.props Recommendation-only scenarios and conversational assertions (conflict detection, conditionals, Common.props awareness) remain as output_matches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reorganize skill description with USE FOR / DO NOT USE FOR markers Restructure the description to improve activation reliability by adopting the USE FOR / DO NOT USE FOR pattern used by other skills in the repo (thread-abort-migration, binlog-failure-analysis). Key changes: - Front-load activation cues before capability list - Add explicit USE FOR / DO NOT USE FOR section markers - Add .sln/.slnx file extensions for solution-path matching - Move DO NOT USE exclusions into description (was only in body) - Use YAML folded scalar for multi-line readability All existing activation keywords preserved. Length reduced from 882 to 849 chars (max 1024). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix issues for external link references * Increase timeouts * Handle packages.config non-activation in convert-to-cpm skill Set expect_activation: false on the packages.config eval scenario so non-activation is reported as expected rather than flagged as a warning. Add a packages.config guard in Step 1 of the skill workflow so that if the skill is activated for a packages.config project, it stops early and recommends migrating to PackageReference first. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove leading blank lines from .sln test files Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use imperative 'Read' instead of 'See' for reference file links Ensures the agent explicitly loads and reads reference files rather than treating them as optional 'see also' links. This aligns with the repo convention of explicitly instructing agents to load reference content. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Simplify dotnet-nuget CODEOWNERS * Simplify packages.config guard * Add minimal Program.cs entrypoints to Web/Worker test fixtures Test fixture projects using Microsoft.NET.Sdk.Web and Microsoft.NET.Sdk.Worker require a Program.cs entrypoint for dotnet build to succeed. The skill workflow instructs the agent to run dotnet restore and dotnet build for validation, so these fixtures must be buildable. Adds minimal entrypoints (with explicit using directives since ImplicitUsings is not enabled in these projects) to 6 projects: - simple-solution/Web - moderate-version-conflicts/Api, Worker - moderate-msbuild-properties/Api - advanced-multi-complexity/Api, Web Also adds Microsoft.Extensions.Hosting package reference to the Worker project, matching what the dotnet new worker template produces. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> |
||
|
Copilot
|
0bb3606942 |
Fix release job skipped by transitive dependency chain from discover (#395)
* Initial plan * Fix release job being skipped due to tainted dependency chain from discover The release job was always skipped for schedule and workflow_dispatch events because the discover job (a transitive dependency via build) is skipped for non-PR events, which taints the dependency chain. Without always() in its if condition, GitHub skips release without evaluating the expression. Add always() so the if expression is always evaluated, and add an explicit needs.build.result == 'success' guard so the release only runs when the build actually passed. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> |
||
|
Viktor Hofer
|
1a20e39a8d |
Refactor skill-validator: add check command, rename validate to evaluate (#388)
* Refactor skill-validator: add check command, rename validate to evaluate - Add new 'check' command for CI skill validation (frontmatter, structure) - Rename ValidateCommand to EvaluateCommand to clarify its purpose - Add skill-check.yml GitHub Actions workflow - Refactor SkillDiscovery for improved skill detection - Simplify OverfittingJudge and SkillProfiler - Update tests to match refactored APIs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix check subcommand: use EvaluateCommand as root command System.CommandLine 2.0.4 requires the primary command to be the RootCommand. Restores the upstream pattern where the evaluate/validate command is the root and check/consolidate/rejudge are subcommands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Pass all plugins in single check invocation Build plugin args dynamically and pass them all at once instead of looping per plugin. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments - Remove 'evaluate' subcommand token from README, evaluation-run.yml, and RunArguments in csproj (evaluate is now the root command) - Wrap ParsePluginJson in try/catch for JsonException with clean error - Add CheckCommandTests with aggregate description limit coverage (under/at/over limit and multi-plugin cases) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
ae0b6a1d86 | Add mtp-hot-reload skill for iterative test fixing with MTP hot reload (#389) | ||
|
YuliiaKovalova
|
14346ff3b2 |
Enrich build-perf-diagnostics skill with MSBuild team and community insights (#342)
* Enrich build-perf-diagnostics skill with MSBuild team and community insights Additions (pure knowledge, no MCP dependency): - RAR: dotnet/msbuild#2015 context, DisableTransitiveProjectReferences, ReferenceOutputAssembly - Copy tasks: SkipCopyUnchangedFiles, --artifacts-path, Copy batching (dotnet/msbuild#12884) - Dev Drive (ReFS): copy-on-write + reduced Defender scans, OrchardCore 7257 Copy tasks benchmark - Evaluation: NuGetSdkResolver 180-400ms overhead (dotnet/msbuild#4025), EnableDefaultItems - NuGet: RestoreUseStaticGraphEvaluation (20s+ savings in large builds) - Project graph shape: wide vs deep (40% faster clean, 20% faster incremental) - MSBuild Server: MSBUILDUSESERVER=1 for CLI caching - Inline task overhead: 1s vs 3ms (RoslynCodeTaskFactory vs compiled) - Misleading ResolveProjectReferences time (dotnet/msbuild#3135) - Incrementality anti-patterns (dotnet/msbuild#13206) - Impact categorization (HIGH/MEDIUM/QUICK WINS) - Quick Wins: expanded to 15 items (from 7) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix minor issues from review comments - Fix missing closing XML tag for CreateHardLinksForCopyFilesToOutputDirectoryIfPossible in Quick Wins checklist - Clarify confusing Copy task batching wording per reviewer feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Trim skill to reduce token overhead Remove standalone sections (MSBuild Server, Inline Tasks, ResolveProjectReferences, Incrementality Anti-patterns) that add ~1400 extra tokens without benefiting the evaluated scenario. Consolidate Quick Wins checklist accordingly. These sections can be contributed as separate skills in follow-up PRs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add separate skills for MSBuild server, inline tasks, and ResolveProjectReferences Extract content removed from build-perf-diagnostics into standalone skills to keep each skill focused and avoid token budget bloat: - msbuild-server: MSBUILDUSESERVER=1 for CLI incremental caching - inline-task-overhead: RoslynCodeTaskFactory ~1s overhead vs ~3ms compiled - resolve-project-references: misleading Target Performance Summary time (dotnet/msbuild#3135), guides users to Task Performance Summary instead Incrementality anti-patterns content was not extracted as a separate skill because the existing incremental-build skill already covers that area. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add github.com/dotnet/msbuild to known-domains.txt Required by reference scanner for MSBuild issue links in build-perf-diagnostics and resolve-project-references skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace full dotnet/msbuild URLs with shorthand notation The reference scanner reads known-domains.txt from upstream main, not from the PR branch. Use shorthand 'dotnet/msbuild#NNNN' format (consistent with the rest of the file) instead of full URLs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review comments and fix reference scanner errors - Remove https://aka.ms/devdrive URLs that trigger reference scanner - Soften unsourced benchmark claims (OrchardCore Copy tasks, 20s+ restore savings, 40%/20% graph shape improvements) per reviewer feedback to use workload-dependent language instead Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add eval tests for new skills; remove inline-task-overhead Add eval.yaml and test fixtures for msbuild-server and resolve-project-references skills: - msbuild-server: tests CLI build perf scenario with developer report - resolve-project-references: tests misleading target perf summary with multi-project setup and perf report Remove inline-task-overhead skill: failed 3 evaluation iterations (avg +2.7%) because the baseline LLM already knows about RoslynCodeTaskFactory overhead — the skill doesn't add enough unique value to meet the 10% improvement threshold. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Address review comments - Add caveat that DisableTransitiveProjectReferences may require adding direct references for consumed types - Fix escaped quotes in Quick Wins RunAnalyzers checklist item - Add OutputType=Exe and Main entry point to msbuild-server fixture - Change 'solution' to 'project' in resolve-project-references report - Trim trailing spaces in developer-report.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore accidentally removed known-domains.txt Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix remaining review comments - Quote -flp argument in resolve-project-references bash example to handle semicolons in POSIX shells - Format LibA/Class1.cs and LibB/Class1.cs across multiple lines for consistency with other test fixtures - Rename msbuild-server/Program.cs to Calculator.cs to match content Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
7e7dabd343 |
[WIP] Improve msbuild skills descriptions (#384)
* Improve msbuild skills descriptions * Improve descriptions |
||
|
Amaury Levé
|
c55e4f222b |
Add MSTest migration skills (v1/v2→v3, v3→v4) (#377)
* Add MSTest migration skills (v1/v2→v3, v3→v4) New dotnet-test plugin with two step-by-step migration skills: - migrate-mstest-v1v2-to-v3: covers assembly ref (v1) and NuGet (v2) projects, assertion overload changes, DataRow constructors, .testsettings replacement, timeout unification, and MSTest.Sdk adoption. - migrate-mstest-v3-to-v4: covers Execute→ExecuteAsync, CallerInfo constructors, ClassCleanupBehavior removal, TestContext.Properties generic change, Assert API changes, ExpectedExceptionAttribute removal, dropped <net8.0 TFMs, and behavioral changes (DisableAppDomain, TestCase.Id, TreatDiscoveryWarningsAsErrors). Includes eval.yaml test scenarios and fixtures for both skills, plugin.json, and CODEOWNERS entries. * Fixes * Improve results * Improve * Improve description * Copy fix for path * Bump timeout |
||
|
Amaury Levé
|
92d830ff48 |
Add writing-mstest-tests skill with eval scenarios (#378)
* Add writing-mstest-tests skill with eval scenarios Add a new skill for writing modern MSTest 3.x/4.x unit tests covering: - Test class structure (sealed, AAA pattern, naming conventions) - Modern assertion APIs (Assert.Throws, IsInstanceOfType, collection asserts) - Data-driven tests (DataRow, DynamicData with ValueTuples) - Async tests with cancellation and timeout patterns - Test lifecycle (constructor init, TestContext injection) - Anti-pattern detection and modernization Includes 10 eval scenarios across 4 goal categories with fixture files for both modern MSTest and legacy anti-pattern test projects. MSTest.Sdk version is recommended via global.json msbuild-sdks instead of inline in the project file, ensuring all test projects get bumped together. * Fixes * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Evangelink * Apply suggestion from @Evangelink --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
d5d2136157 | Update code owners to use team alias (#382) | ||
|
Larry Ewing
|
1f127d4239 |
Add --keep-sessions flag and rejudge command to skill-validator (#149)
* Add --keep-sessions flag and rejudge command - Add --keep-sessions flag to preserve agent session data in SQLite DB - Create SessionDatabase service with WAL mode and thread-safe writes - Update AgentRunner to separate config/work dir lifecycle - Track session metadata, RunMetrics, judge and pairwise results - Add rejudge command to re-run judges on saved sessions - Add 8 tests for SessionDatabase (CRUD, concurrency, isolation) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add prompt and skill_content columns to sessions.db Store the eval scenario prompt and SKILL.md content in the sessions table so they are available for browsing and replay without needing the original skill directory. Both columns are optional (nullable) for backward compatibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace skill_content with skill_sha for change detection Compute a SHA-256 hash (truncated to 12 hex chars) over all files in the skill directory, sorted by relative path for determinism. This gives lightweight change detection without storing potentially large skill content in the database. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR feedback: relative config_dir, schema_info table - Use session ID as config dir folder name (sessions/<id>) so the DB record links directly to its files on disk - Store config_dir as relative path for portable results directories - Add schema_info table with type='skill-validator' and version='1' for easy DB detection by external tools like dotnet-replay - Add GetSchemaInfo() method for consumers - Extract GetSessions(whereClause) for reuse - Add tests for schema_info and relative config_dir Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix: share timestamped results dir between sessions and reporters When --keep-sessions is active, pass the already-created timestamped directory to Reporter.ReportResults so sessions.db, session files, and report outputs (results.json, summary.md) all live in the same folder instead of creating two separate timestamped directories. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Adapt rejudge for current validator model Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #149 review feedback Persist the judge model for rejudge, use fresh judge workdirs, tighten session DB hashing/foreign key behavior, and cache skill SHA calculation during validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix rejudge rubric persistence Persist scenario rubrics in sessions.db so rejudge can reconstruct the original EvalScenario rubric, self-migrate older databases to the new schema, and validate --judge-mode inputs explicitly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Guard SessionDatabase reads with semaphore Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix validator session cleanup and reporting Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
2a8388dd71 |
Add VSTest to MTP migration skill (#373)
* Add VSTest to MTP migration skill New migrate-vstest-to-mtp skill in the dotnet-test plugin covering: - Framework-specific MTP runner enablement (MSTest, NUnit, xUnit.net) - Directory.Build.props centralization of OutputType, runner properties - MSTest.Sdk UseVSTest removal - NUnit3TestAdapter 5.0.0+ upgrade requirement - xunit v2 to xunit.v3 upgrade requirement - dotnet test CLI argument translation (--logger to --report-trx, --collect to --coverage, --blame-crash to --crashdump, --blame-hang to --hangdump) - xUnit.net --filter syntax migration to --filter-class/--filter-trait/ --filter-query - global.json native MTP mode (.NET 10+) vs TestingPlatformDotnetTestSupport VSTest-bridge mode (.NET 9 and earlier) - Azure DevOps VSTest@3 to DotNetCoreCLI@2 pipeline migration - MTP extension NuGet packages (TrxReport, CrashDump, HangDump, CodeCoverage) - Exit code 8 behavioral difference (--ignore-exit-code 8, TESTINGPLATFORM_EXITCODE_IGNORE) Includes 10 eval scenarios with tiered timeouts (240-360s), 5 fixture directories, plugin.json, marketplace.json entries, CODEOWNERS, and README.md updates. * Fix codeowners * Improve * Improve further * Improve prompt * Improve further the description * Address review comments |
||
|
Amaury Levé
|
ab201ee691 |
Fix eval source file resolution to use eval directory (#374)
* Fix eval source file resolution to use eval directory The SetupWorkDir method resolved setup file 'source:' paths relative to the skill directory (plugins/.../skills/<name>/), but eval.yaml files live under tests/. The ../../../../ relative paths resolved correctly but were rejected by the path traversal security check because the resolved path escaped the skill directory boundary. Fix by preferring the eval directory as the base for source resolution when evalPath is available, falling back to skillPath for backward compatibility with scenarios embedded in SKILL.md files. Also update run-tests eval.yaml source paths from '../../../../tests/dotnet-test/run-tests/fixtures/...' to 'fixtures/...' (relative to the eval directory where the fixtures actually live). This was causing all 11 source-based run-tests scenarios to get zero fixture files, making the agent find an empty workspace and fail to evaluate. * Address review: extract ResolveSourcePath helper, add warning, fix remaining eval.yaml paths - Extract source-resolution and path-traversal check into internal ResolveSourcePath helper for testability - Emit a clear warning when file.Source is set but no base directory is available (instead of silently continuing) - Add unit tests covering: evalPath-based resolution, skillPath fallback, null paths warning, traversal rejection, and evalPath preference - Fix path traversal patterns in analyzing-dotnet-performance and clr-activation-debugging eval.yaml files (same issue as run-tests) |