mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
Amaury Levé
33110eeba8
* code-testing-agent: fix workspace-integrity activation + stabilize Contoso rubric
Workspace-integrity scenario was NOT ACTIVATED in plugin mode: the prompt
("Generate unit tests for its core module") was terse and small-scoped, so
the runtime did not route to the code-testing-agent skill. Reframe it with
high-level test-generation language ("comprehensive pytest test suite",
"scaffold", "thorough unit tests") that matches the skill description,
while preserving the guardrail anchors: it still points at the on-disk module
without naming it and never implies restoring the gutted tree.
ContosoUniversity rubric item #5 (find-untested-sources) was conditional on
that skill being loaded — only true in plugin mode. In isolated runs the
agent cannot satisfy it, so the judge penalized it asymmetrically, injecting
isolated-vs-plugin variance. Make the conditional deterministic: explicitly
N/A when the skill is not loaded, without lowering the bar when it is.
Applied to both eval.yaml and eval.vally.yaml.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* code-testing-agent: make find-untested-sources rubric decidable from session timeline
Address review feedback: the 'treat as satisfied (N/A) when the skill is
not loaded' clause is not verifiable from the judge's inputs (which show
which tools were called, not which were available). Rewrite the criterion
to be decidable from the session timeline by requiring a source-to-test
pairing map recorded in .testagent/research.md that either cites
find-untested-sources output or documents the equivalent manual approach.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
18 KiB
18 KiB