mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
Amaury Levé
812565f8d1
* Add polyglot evals for code-testing-agent (Python Flask + TypeScript Vitest) Adds two non-.NET evaluation scenarios to the dotnet-test/code-testing-agent eval to measure how well the agent generates idiomatic tests in Python and TypeScript — the languages exercised by the msbench top5-* benchmarks. New fixtures (no pre-existing tests, so passing pytest/vitest proves the agent actually generated them): - fixtures/python-flask-tasks/ — small Flask app with TaskService (injected clock), TaskRepository Protocol + InMemoryTaskRepository, and a /tasks blueprint with HTTP 201/200/400/404/409 surfaces. Validated locally with `python -m pip install -e ".[test]" && python -m pytest`. - fixtures/typescript-vitest-cart/ — small shopping-cart library with an injectable DiscountPolicy seam (No/Percentage policies) and a Cart class with non-trivial merge / clamping semantics. Validated locally with `npm ci && npx vitest run`. package-lock.json committed so `npm ci` is reproducible in CI. New scenarios (added to both eval.yaml and eval.vally.yaml): - "Generate pytest tests for the Flask tasks API (Python polyglot)" — asserts that `python3 -m pip install -e '.[test]' && python3 -m pytest` exits 0 and that at least one `tests/**/test_*.py` file was produced. Rubric rewards using Flask's `test_client()`, mocking `TaskRepository` for service-level tests, asserting HTTP status + JSON body, and covering the empty-title / >200-char / not-found / already-done error paths. - "Generate Vitest tests for the shopping-cart library (TypeScript polyglot)" — asserts that `npm ci && npx vitest run` exits 0 and that at least one `tests/**/*.test.ts` file was produced. Rubric rewards mocking `DiscountPolicy` via `vi.fn()`, covering Cart merge semantics, `updateQuantity` zero-removes, `totals()` discount-clamping, and the `PercentageDiscountPolicy` constructor boundary. Companion to #708 (polyglot examples + sub-agent generification); these scenarios exercise the per-language guidance that PR adds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback for #709 - routes.py: guard `create_task` against non-dict JSON payloads (e.g. list) by returning 400 instead of letting `payload.get(...)` raise AttributeError and surface as a 500. This keeps the API behavior stable and makes it easier for generated tests to assert 400 on invalid input shapes. - tsconfig.json: drop `vitest/globals` from `compilerOptions.types`. The fixture's vitest.config.ts sets `globals: false`, so allowing TypeScript to assume globals (describe/it/expect) only enables tests that compile but fail at runtime. Removing it keeps the fixture aligned with the non-global Vitest API the eval is exercising. Both fixtures re-smoke-tested locally: vitest 1/1, pytest 2/2 (including a new test confirming the non-dict body returns 400). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix skill activation for polyglot scenarios in PR #709 Both polyglot scenarios (pytest + Flask, Vitest + TS) scored 5.0/5 quality but reported `⚠️ NOT ACTIVATED` because the SDK's skill router did not load `code-testing-agent` into the agent's context: `activated: false, detectedSkills: []` in skillActivationIsolated / skillActivationPlugin. The .NET ContosoUniversity scenario activates fine because its prompt has stronger `project-wide, multi-file ... high coverage` framing and SKILL.md already covers .NET keywords. Fixes per `eng/skill-validator/src/docs/InvestigatingResults.md` section "Skill not activated": - SKILL.md description: add framework-specific keywords (pytest, Flask/Django, Vitest, Jest, Mocha, JUnit, Node libraries, API, package, project-wide, multi-file) so the router has explicit hooks for polyglot prompts. Trimmed verbose `DO NOT USE FOR` clauses to stay under the 1,024-char skill spec limit (now 1,019 chars). - eval.yaml + eval.vally.yaml: rewrite the two polyglot prompts to mirror the .NET scenario's framing — add `project-wide, multi-file test generation task across the ... layers` and `achieve high coverage`, soften the library-prescriptive bullets (Mock(spec=...), vi.fn()) into capability-level requirements (`mocked or stubbed`, `mock or hand-written stub`). Rubric items unchanged — they remain flexible enough to grade either mock style. Validated locally: `dotnet run --project eng/skill-validator/src -- check --plugin ./plugins/dotnet-test` => ✅ All checks passed (23 skill(s), 11 agent(s), 1 plugin(s)). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #709 review + sharpen polyglot skill activation **Activation fix for polyglot scenarios (Python + TypeScript)** The previous attempt (e8e6feeb2) added pytest/Vitest framework names to a long capability list, but both polyglot scenarios still reported activated: false in skillActivationIsolated/skillActivationPlugin across two consecutive /evaluate runs. Rewrote the SKILL.md frontmatter description so trigger vocabulary moves into the "Use when asked to ..." intent phrase and mirrors the EXACT phrasing the polyglot prompts use (the .NET scenario already activates because it says "scaffold a new test project", which is in the description): - Action triggers: "generate pytest tests", "generate Vitest tests" alongside "generate tests" / "write unit tests". - Scaffolding triggers: "scaffold a new test project / test suite" (matches "scaffold a comprehensive pytest suite" and "scaffold a comprehensive Vitest suite"). - Surface nouns matching the polyglot fixtures: "web app", "REST API", "blueprint", "services, repositories, routes, and modules". - Up front: list what the skill scaffolds across all languages (".NET test projects, pytest test suites, Vitest/Jest test suites, Go test files, JUnit test suites"). Trimmed the "DO NOT USE FOR" clause to keep the folded description at 992 chars (under the 1,024 agentskills.io spec limit). **Review comment fixes** - fixtures/typescript-vitest-cart/tsconfig.json: drop types: ["node"] — fixture has no @types/node so tsc would fail with "Cannot find type definition file for 'node'". - fixtures/typescript-vitest-cart/src/pricing.ts: rewrite the DiscountPolicy.computeDiscountCents @returns docstring. Was "Must not exceed the subtotal" (contradicted by Cart.totals() which clamps both negative and over-subtotal values). Now clearly states policies should target [0, subtotalCents] but Cart.totals() clamps out-of-range values back into the range, so contract and implementation match. - fixtures/python-flask-tasks/src/tasks_api/models.py: the Task.created_at default now uses datetime.now(timezone.utc) via default_factory, matching TaskService's timezone-aware _now callable. Avoids mixing naive datetime.utcnow() with aware datetimes when Task is instantiated directly (e.g., in generated tests). - fixtures/python-flask-tasks/src/tasks_api/routes.py: add an explicit isinstance(title, str) guard in create_task before delegating to TaskService.create. Previously a JSON title value that was a number/null/object would fall through to title.strip() and crash with AttributeError -> 500. - fixtures/python-flask-tasks/src/tasks_api/service.py: defense-in-depth — TaskService.create now raises ValueError on non-string titles (instead of crashing) so non-Flask callers (and unit tests that exercise the service directly with bad inputs) get a clean validation error. Validated locally: - dotnet run --project eng/skill-validator/src -- check --plugin ./plugins/dotnet-test -> All checks passed (23 skill(s), 11 agent(s), 1 plugin(s)). - Python fixture: python -m compileall -q src clean. - TS fixture: npm ci && npx tsc --noEmit clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Simplify polyglot prompts to encourage skill activation The Python pytest and TypeScript Vitest scenarios were marked NOT ACTIVATED in evaluation runs even after rewriting the SKILL.md description with polyglot trigger vocabulary. Investigation of results.json showed the agent went straight to view/create without ever invoking the `skill` tool. Root cause: the polyglot prompts contained long bullet lists pre-specifying exact test cases (Cart.add merge semantics, updateQuantity zero-removes, PercentageDiscountPolicy boundary tests, TaskService validation errors, etc.). This gave the model a complete plan and removed any incentive to load a planning skill. Compare with the .NET ContosoUniversity scenario (which does activate the skill): a short open-ended prompt — "write comprehensive unit tests across the controllers, services, and the data layer. Achieve high code coverage." plus a real tool-config requirement (coverlet.collector for Cobertura XML). Mirror that style in both polyglot prompts: - Drop the prescriptive bullet lists; keep a short framing intro and a single open-ended ask. - Add a coverage tooling requirement (pytest-cov XML for Python, @vitest/coverage-v8 lcov for TypeScript) to give the skill real scaffolding value. - Rubric and assertions unchanged — they still grade the actual quality of the generated suite. Mirrors guidance in eng/skill-validator/src/docs/InvestigatingResults.md section 5 ("Check the scenario itself has sufficient information that the agent can reason that it needs the skill") and avoids the "illegitimate fixes" called out there. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Elaborate polyglot prompts and enforce coverage in assertions The previous prompt rewrite got the TypeScript Vitest scenario activating in isolated mode (2/3 runs) but the Python scenario still NEVER activated. Inspecting the session events showed the agent went straight to view+create on all 3 Python runs without invoking the `skill` tool. Theory: the Python prompt is too abstract (just "service layer, the repository, and the Flask routes") compared to the .NET prompt which enumerates Students/Courses/Departments/Instructors controllers. Make the polyglot prompts feel as substantial as the .NET one by: - Naming concrete symbols (TaskService with injected clock, InMemoryTaskRepository, /tasks, /tasks/<id>, /tasks/<id>/complete for Python; Cart.add/remove/updateQuantity/totals plus the two policy classes for TS) — mirrors the .NET prompt's named controllers list. - Pointing at the exact mocking + injection strategy ("with the repository mocked and the clock injected", "with DiscountPolicy mocked"). - Pinning the coverage report to an explicit path (coverage.xml for Python, coverage/lcov.info for TS). Also enforce the coverage tooling requirement at assertion time so the prompt's "and the project should be configured so... coverage" is a real deliverable — mirrors the .NET scenario's `coverage.cobertura.xml` file_exists check: - Python: added file_exists for fixtures/python-flask-tasks/coverage.xml (pytest with --cov=tasks_api --cov-report=xml in addopts writes coverage.xml from the directory pytest runs in). - TS: added a `npx vitest run --coverage` run step plus file_exists for fixtures/typescript-vitest-cart/coverage/lcov.info. Both changes are legitimate per eng/skill-validator/src/docs/InvestigatingResults.md "improving the skill vs gaming the eval" — they make the deliverable richer rather than steering toward the skill or relaxing the rubric. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify SKILL.md routing and mirror coverage assertions in eval.vally.yaml Per rubber-duck review of the activation strategy: 1. SKILL.md description was ambiguous about coverage — the previous wording said "DO NOT USE FOR coverage/CRAP analysis" but the polyglot prompts heavily mention coverage tooling. In plugin mode that could confuse the router into deferring to coverage-analysis or crap-score even when the user wants new tests written. Clarify: - In-scope: "configures coverage tooling (coverlet, pytest-cov, @vitest/coverage-v8) as part of test generation" - Out-of-scope: "analyzing existing coverage reports" (not just "coverage") Description re-tightened to 1003 chars (under the 1024 limit). 2. eval.vally.yaml previously asserted only that test files exist; the eval.yaml mirror added coverage XML / lcov.info file_exists assertions in the prior commit. Mirror those into the vally graders so both eval schemas enforce the same coverage deliverable: - Python: assert fixtures/python-flask-tasks/coverage.xml exists - TS: run npx vitest run --coverage --reporter=basic then assert fixtures/typescript-vitest-cart/coverage/lcov.info exists Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Harden polyglot fixtures so baseline cannot perfectly ace them Both polyglot scenarios (Python Flask, TypeScript Vitest) activated the code-testing-agent skill in isolated mode but not in plugin mode. Root cause: the baseline (no skill loaded) was scoring 5/5, leaving the skill no headroom to demonstrate value, so plugin-mode runs went straight to view+write without invoking any skill. Per InvestigatingResults.md section 8, the correct fix is to add genuine complexity that benefits from a planning workflow rather than relaxing rubrics or gaming activation. This commit grows both fixtures into realistic, multi-seam libraries that genuinely reward layered test planning. Python (fixtures/python-flask-tasks/) additions: - models.py: TaskPriority enum, Tag value-object with normalization / validation, due_at on Task, is_overdue(now) helper. - queries.py (new): TaskQuery / TaskPage / apply_query separating filter / sort / pagination from the service. Sort by created_at, due_at, priority, or title with ascending / descending order; filter by status, tag, search substring, and overdue. - repository.py: TaskRepository protocol gains delete(); added normalize_tags() helper that the service uses for tag dedup. - repository_sqlite.py (new): SqliteTaskRepository — a second backend implementing the same Protocol using stdlib sqlite3 (no extra deps), with persisted tags via a task_tags table and ON DELETE CASCADE. - service.py: configurable max_title_length, timezone-aware due_at validation, priority / due_at / tag mutations, reopen() and delete() state transitions, add_tags / remove_tag with normalization. - routes.py: new endpoints — DELETE /tasks/<id>, POST /tasks/<id>/reopen, POST /tasks/<id>/tags, DELETE /tasks/<id>/tags/<name>. GET /tasks now parses status / tag / q / overdue / sort / order / limit / offset and returns a paged response with total / has_more. - app.py: env / config-driven backend selection (TASKS_BACKEND = memory | sqlite, TASKS_DATABASE_URL, TASKS_MAX_TITLE_LENGTH). - README.md: documents the new layout and the expected layered test approach. TypeScript (fixtures/typescript-vitest-cart/) additions: - pricing.ts: FixedAmountDiscountPolicy and CompositeDiscountPolicy with "sum" vs "chain" stacking modes (the latter compounds discounts on the remaining subtotal). - tax.ts (new): TaxCalculator interface; NoTaxCalculator, RegionalTaxCalculator (per-region rate table with default fallback), AsyncTaxRateProvider async seam, AsyncTaxCalculator adapter. - shipping.ts (new): ShippingCalculator interface; FreeShippingCalculator, FlatShippingCalculator (with optional free-over-threshold), WeightBasedShippingCalculator (bracket-based with overflow cost). - inventory.ts (new): PriceFetcher and InventoryChecker async seams, InventoryError with productId / requested / available, refreshPrices() helper. - cart.ts: fixed pricing pipeline subtotal → discount (clamped) → tax (on the discounted subtotal) → shipping; async checkout() that refreshes prices, checks inventory (first denial wins, InventoryError thrown), and supports an AsyncTaxRateProvider for this call. - product.ts: optional weightGrams (used by shipping) and currency. - index.ts: barrel re-exports for all new symbols. - README.md: documents the new layout and the expected layered test approach. eval.yaml + eval.vally.yaml: - Updated both polyglot prompts to name the new modules and seams (queries.apply_query, SqliteTaskRepository, tag endpoints, CompositeDiscountPolicy stacking, AsyncTaxRateProvider, WeightBasedShippingCalculator, PriceFetcher / InventoryChecker, InventoryError). Style remains open-ended (.NET-style) — no bullet list of test cases. - Replaced the small rubric with outcome-based rubric items that grade the new behaviors: pipeline composition (tax on discounted subtotal), CompositeDiscountPolicy stacking-mode differences, weight bracket boundary semantics, async checkout() with mocked collaborators including InventoryError shape, queries.apply_query pagination boundaries, SqliteTaskRepository integration tests against an in-memory connection, multi-status assertions across 200/201/204/400/404/409, and constructor validation on the boundary classes. - Bumped per-command timeouts from 5m to 10m to absorb the larger test suite. No skill description / SKILL.md changes — the skill description is already accurate for these scenarios (1003 / 1024 chars). * Enforce hard 80% coverage floor on polyglot fixtures After the previous harden commit (c78532bf7), the polyglot scenarios still scored a 5.0/5 baseline with no plugin-mode activation. Root cause: even on the larger fixtures the judge ticked every rubric item because the baseline agent wrote tests that *touched* each named module — the rubric grades "covered this concept", not "covered this code path", so a strong baseline naturally clears it. The previous commit hardened the SURFACE of the fixtures. This commit hardens the SUCCESS BAR by baking coverage thresholds into the test runners themselves, so a baseline that skips an entire module fails the existing pytest / vitest run-command assertion outright (assertion exit code != 0 → run fails → judge cannot give 5/5). Python (fixtures/python-flask-tasks/pyproject.toml): - Added `pytest-cov>=5.0,<7.0` to the [test] extra so coverage is pre-installed (the agent no longer has to add it manually). - Set `addopts = "-q --cov=tasks_api --cov-branch --cov-report=xml --cov-report=term-missing --cov-fail-under=80"`. pytest now exits non-zero when total line + branch coverage on tasks_api is below 80%, which makes the existing `python -m pytest` assertion fail. TypeScript (fixtures/typescript-vitest-cart/vitest.config.ts): - Added a `coverage` block that pre-configures provider: 'v8', reporter: ['lcov', 'text'], include: ['src/**/*.ts'], and thresholds of lines / statements / functions ≥ 80, branches ≥ 70. The agent no longer has to wire up the coverage provider. `npx vitest run --coverage` exits non-zero when any threshold is unmet. Prompts (eval.yaml + eval.vally.yaml): - Trimmed the prompts to drop the "configure coverage if not already there" guidance — coverage tooling is now pre-installed and the floor is fixed. The prompt now explicitly tells the agent the threshold is enforced and that incidental coverage will not clear the bar, so the agent's planning workflow has a concrete cross-module target to plan against. READMEs: documented the new behavior so the agent reading the README sees the coverage floor up front. Verified locally: with an empty tests/ directory, pytest exits 1 with "FAIL Required test coverage of 80% not reached" and `vitest run --coverage` exits 1 with "Coverage for lines (0%) does not meet global threshold (80%)". The existing run-command assertions therefore fail for incomplete suites without needing any new assertion / grader. * Address PR #709 review: snapshot deep-copy, due_at sort, tests/ wording - cart.ts: snapshot() now deep-copies the Product on each line in addition to the CartLine, so callers mutating the returned snapshot can no longer reach back into the cart's internal state via line.product. (Copilot 3344261946) - queries.py: _sort_key for 'due_at' returns a (is_none, isoformat) tuple so we never compare across the None / not-None boundary, and never trigger 'cannot compare naive and aware datetimes' TypeErrors when a stray naive datetime sneaks through. (Copilot 3347273389) - README + eval.yaml + eval.vally.yaml: reword the 'tests/ directory is intentionally empty' phrasing to 'no test files yet — only a .gitkeep marker' so the docs are strictly accurate about what's on disk. (Copilot 3344580026 / 3344580055 / 3344580067 / 3344580085 / 3344580104 / 3344580136 / 3344580156) --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Tasks API (Python Flask) — code-testing-agent polyglot eval fixture
A small Flask "tasks" API used as a polyglot eval fixture for the code-testing-agent skill. The agent is asked to write a pytest suite for the service and routes; the eval verifies that pytest passes against the suite the agent produced.
Layout
pyproject.toml # Flask + pytest, src layout
src/tasks_api/
__init__.py
models.py # Task / Tag / TaskStatus / TaskPriority + is_overdue()
queries.py # TaskQuery / TaskPage + apply_query (filter / sort / paginate)
repository.py # TaskRepository protocol + InMemoryTaskRepository + normalize_tags
repository_sqlite.py # SqliteTaskRepository (second backend, stdlib sqlite3)
service.py # TaskService: create / get / query / complete / reopen / delete
# priority / due_at / tag mutations; injected clock
routes.py # Flask blueprint: /tasks CRUD, filtering, tag endpoints
app.py # create_app() — selects backend from TASKS_BACKEND env / config
tests/ # no test files yet (only a .gitkeep marker) — the agent must create the suite here
Running tests locally
Linux / macOS / WSL:
python -m pip install -e ".[test]"
python -m pytest
Windows:
py -m pip install -e ".[test]"
py -m pytest
Coverage (pytest-cov) is enabled by default via pyproject.toml and is
enforced as a hard 80% line + branch floor on the tasks_api package
— pytest exits non-zero (and coverage.xml is not produced) when the
suite does not cover at least 80% of lines and branches.
What the agent should produce
A planned, layered test suite covering the multiple seams in this fixture:
TaskServicetests against a mockedTaskRepository(e.g.unittest.mock.Mock(spec=TaskRepository)) with an injectednowcallable for deterministiccreated_at/completed_at/ overdue checks. Cover the validation matrix: title (type, empty, length), priority, due_at (timezone-awareness), tags (normalization, dedup), state transitions (complete / reopen, already-done, already-pending), and tag add/remove edge cases.queries.apply_querytests against fixedTasklists covering filter combinations (status, tag, search, overdue), sort fields (created_at, due_at, priority, title), ascending vs descending order, and pagination boundaries (limit/offset/has_more).SqliteTaskRepositoryintegration tests against an in-memory connection (SqliteTaskRepository.in_memory()) covering add/get/update/delete/list/next_id and tag persistence.- Flask blueprint tests using
create_app(service=...).test_client()— no real network ports. Exercise the request/response mapping: 200/201/204/400/404/409 paths, query-parameter parsing, JSON shape, and tag endpoint round-trips. - Use the in-memory backend for blueprint tests by default; only use SQLite when the test is specifically about persistence behavior.