The code-testing-generator/implementer agents could treat an unusual or
scaffolded workspace (e.g. a gutted repo with an injected synthetic module)
as corruption and "repair" it with git checkout/restore/reset/clean or rm,
restoring deleted tracked files and testing the wrong code.
- Replace generator Rule 5 ("Clean git first - stash changes") with an
explicit "Treat the workspace as delivered" rule, and add a "Never mutate
version control" rule. Output must be purely additive test files.
- Add a no-revert/no-clean invariant to the implementer's edit boundaries.
- Add a 'workspace integrity' eval to the code-testing-agent suite
(eval.yaml + eval.vally.yaml). The fixture looks gutted: a metricsd project
whose real core/io modules are committed at HEAD but deleted from the
working tree, leaving only a synthetic 'synthstr' decoy. A git restore would
resurrect the deleted sentinel files; graders fail if they reappear and
require passing pytest tests for the module as delivered.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add polyglot evals for code-testing-agent (Python Flask + TypeScript Vitest)
Adds two non-.NET evaluation scenarios to the dotnet-test/code-testing-agent
eval to measure how well the agent generates idiomatic tests in Python and
TypeScript — the languages exercised by the msbench top5-* benchmarks.
New fixtures (no pre-existing tests, so passing pytest/vitest proves the agent
actually generated them):
- fixtures/python-flask-tasks/ — small Flask app with TaskService (injected
clock), TaskRepository Protocol + InMemoryTaskRepository, and a /tasks
blueprint with HTTP 201/200/400/404/409 surfaces. Validated locally with
`python -m pip install -e ".[test]" && python -m pytest`.
- fixtures/typescript-vitest-cart/ — small shopping-cart library with an
injectable DiscountPolicy seam (No/Percentage policies) and a Cart class
with non-trivial merge / clamping semantics. Validated locally with
`npm ci && npx vitest run`. package-lock.json committed so `npm ci` is
reproducible in CI.
New scenarios (added to both eval.yaml and eval.vally.yaml):
- "Generate pytest tests for the Flask tasks API (Python polyglot)" — asserts
that `python3 -m pip install -e '.[test]' && python3 -m pytest` exits 0
and that at least one `tests/**/test_*.py` file was produced. Rubric
rewards using Flask's `test_client()`, mocking `TaskRepository` for
service-level tests, asserting HTTP status + JSON body, and covering the
empty-title / >200-char / not-found / already-done error paths.
- "Generate Vitest tests for the shopping-cart library (TypeScript polyglot)"
— asserts that `npm ci && npx vitest run` exits 0 and that at least one
`tests/**/*.test.ts` file was produced. Rubric rewards mocking
`DiscountPolicy` via `vi.fn()`, covering Cart merge semantics,
`updateQuantity` zero-removes, `totals()` discount-clamping, and the
`PercentageDiscountPolicy` constructor boundary.
Companion to #708 (polyglot examples + sub-agent generification); these
scenarios exercise the per-language guidance that PR adds.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback for #709
- routes.py: guard `create_task` against non-dict JSON payloads (e.g. list)
by returning 400 instead of letting `payload.get(...)` raise AttributeError
and surface as a 500. This keeps the API behavior stable and makes it easier
for generated tests to assert 400 on invalid input shapes.
- tsconfig.json: drop `vitest/globals` from `compilerOptions.types`. The
fixture's vitest.config.ts sets `globals: false`, so allowing TypeScript
to assume globals (describe/it/expect) only enables tests that compile but
fail at runtime. Removing it keeps the fixture aligned with the non-global
Vitest API the eval is exercising.
Both fixtures re-smoke-tested locally: vitest 1/1, pytest 2/2 (including a
new test confirming the non-dict body returns 400).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix skill activation for polyglot scenarios in PR #709
Both polyglot scenarios (pytest + Flask, Vitest + TS) scored 5.0/5
quality but reported `⚠️ NOT ACTIVATED` because the SDK's skill router
did not load `code-testing-agent` into the agent's context:
`activated: false, detectedSkills: []` in skillActivationIsolated /
skillActivationPlugin. The .NET ContosoUniversity scenario activates
fine because its prompt has stronger `project-wide, multi-file ... high
coverage` framing and SKILL.md already covers .NET keywords.
Fixes per `eng/skill-validator/src/docs/InvestigatingResults.md` section
"Skill not activated":
- SKILL.md description: add framework-specific keywords (pytest,
Flask/Django, Vitest, Jest, Mocha, JUnit, Node libraries, API,
package, project-wide, multi-file) so the router has explicit hooks
for polyglot prompts. Trimmed verbose `DO NOT USE FOR` clauses to
stay under the 1,024-char skill spec limit (now 1,019 chars).
- eval.yaml + eval.vally.yaml: rewrite the two polyglot prompts to
mirror the .NET scenario's framing — add `project-wide, multi-file
test generation task across the ... layers` and `achieve high
coverage`, soften the library-prescriptive bullets (Mock(spec=...),
vi.fn()) into capability-level requirements (`mocked or stubbed`,
`mock or hand-written stub`). Rubric items unchanged — they remain
flexible enough to grade either mock style.
Validated locally: `dotnet run --project eng/skill-validator/src --
check --plugin ./plugins/dotnet-test` => ✅ All checks passed
(23 skill(s), 11 agent(s), 1 plugin(s)).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR #709 review + sharpen polyglot skill activation
**Activation fix for polyglot scenarios (Python + TypeScript)**
The previous attempt (e8e6feeb2) added pytest/Vitest framework names
to a long capability list, but both polyglot scenarios still reported
activated: false in skillActivationIsolated/skillActivationPlugin
across two consecutive /evaluate runs.
Rewrote the SKILL.md frontmatter description so trigger vocabulary
moves into the "Use when asked to ..." intent phrase and mirrors
the EXACT phrasing the polyglot prompts use (the .NET scenario
already activates because it says "scaffold a new test project",
which is in the description):
- Action triggers: "generate pytest tests", "generate Vitest tests"
alongside "generate tests" / "write unit tests".
- Scaffolding triggers: "scaffold a new test project / test suite"
(matches "scaffold a comprehensive pytest suite" and "scaffold
a comprehensive Vitest suite").
- Surface nouns matching the polyglot fixtures: "web app",
"REST API", "blueprint", "services, repositories, routes,
and modules".
- Up front: list what the skill scaffolds across all languages
(".NET test projects, pytest test suites, Vitest/Jest test
suites, Go test files, JUnit test suites").
Trimmed the "DO NOT USE FOR" clause to keep the folded description
at 992 chars (under the 1,024 agentskills.io spec limit).
**Review comment fixes**
- fixtures/typescript-vitest-cart/tsconfig.json: drop
types: ["node"] — fixture has no @types/node so tsc
would fail with "Cannot find type definition file for 'node'".
- fixtures/typescript-vitest-cart/src/pricing.ts: rewrite the
DiscountPolicy.computeDiscountCents @returns docstring.
Was "Must not exceed the subtotal" (contradicted by
Cart.totals() which clamps both negative and over-subtotal
values). Now clearly states policies should target
[0, subtotalCents] but Cart.totals() clamps out-of-range
values back into the range, so contract and implementation match.
- fixtures/python-flask-tasks/src/tasks_api/models.py: the
Task.created_at default now uses datetime.now(timezone.utc)
via default_factory, matching TaskService's timezone-aware
_now callable. Avoids mixing naive datetime.utcnow() with
aware datetimes when Task is instantiated directly (e.g., in
generated tests).
- fixtures/python-flask-tasks/src/tasks_api/routes.py: add an
explicit isinstance(title, str) guard in create_task
before delegating to TaskService.create. Previously a JSON
title value that was a number/null/object would fall through
to title.strip() and crash with AttributeError -> 500.
- fixtures/python-flask-tasks/src/tasks_api/service.py:
defense-in-depth — TaskService.create now raises ValueError
on non-string titles (instead of crashing) so non-Flask callers
(and unit tests that exercise the service directly with bad
inputs) get a clean validation error.
Validated locally:
- dotnet run --project eng/skill-validator/src -- check --plugin
./plugins/dotnet-test -> All checks passed (23 skill(s),
11 agent(s), 1 plugin(s)).
- Python fixture: python -m compileall -q src clean.
- TS fixture: npm ci && npx tsc --noEmit clean.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Simplify polyglot prompts to encourage skill activation
The Python pytest and TypeScript Vitest scenarios were marked NOT
ACTIVATED in evaluation runs even after rewriting the SKILL.md
description with polyglot trigger vocabulary. Investigation of
results.json showed the agent went straight to view/create without
ever invoking the `skill` tool.
Root cause: the polyglot prompts contained long bullet lists
pre-specifying exact test cases (Cart.add merge semantics, updateQuantity
zero-removes, PercentageDiscountPolicy boundary tests, TaskService
validation errors, etc.). This gave the model a complete plan and removed
any incentive to load a planning skill.
Compare with the .NET ContosoUniversity scenario (which does activate
the skill): a short open-ended prompt — "write comprehensive unit tests
across the controllers, services, and the data layer. Achieve high code
coverage." plus a real tool-config requirement (coverlet.collector for
Cobertura XML).
Mirror that style in both polyglot prompts:
- Drop the prescriptive bullet lists; keep a short framing intro and a
single open-ended ask.
- Add a coverage tooling requirement (pytest-cov XML for Python,
@vitest/coverage-v8 lcov for TypeScript) to give the skill real
scaffolding value.
- Rubric and assertions unchanged — they still grade the actual quality
of the generated suite.
Mirrors guidance in eng/skill-validator/src/docs/InvestigatingResults.md
section 5 ("Check the scenario itself has sufficient information that
the agent can reason that it needs the skill") and avoids the
"illegitimate fixes" called out there.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Elaborate polyglot prompts and enforce coverage in assertions
The previous prompt rewrite got the TypeScript Vitest scenario
activating in isolated mode (2/3 runs) but the Python scenario still
NEVER activated. Inspecting the session events showed the agent went
straight to view+create on all 3 Python runs without invoking the
`skill` tool.
Theory: the Python prompt is too abstract (just "service layer, the
repository, and the Flask routes") compared to the .NET prompt which
enumerates Students/Courses/Departments/Instructors controllers. Make
the polyglot prompts feel as substantial as the .NET one by:
- Naming concrete symbols (TaskService with injected clock,
InMemoryTaskRepository, /tasks, /tasks/<id>, /tasks/<id>/complete
for Python; Cart.add/remove/updateQuantity/totals plus the two
policy classes for TS) — mirrors the .NET prompt's named
controllers list.
- Pointing at the exact mocking + injection strategy ("with the
repository mocked and the clock injected", "with DiscountPolicy
mocked").
- Pinning the coverage report to an explicit path (coverage.xml for
Python, coverage/lcov.info for TS).
Also enforce the coverage tooling requirement at assertion time so
the prompt's "and the project should be configured so... coverage"
is a real deliverable — mirrors the .NET scenario's
`coverage.cobertura.xml` file_exists check:
- Python: added file_exists for fixtures/python-flask-tasks/coverage.xml
(pytest with --cov=tasks_api --cov-report=xml in addopts writes
coverage.xml from the directory pytest runs in).
- TS: added a `npx vitest run --coverage` run step plus file_exists
for fixtures/typescript-vitest-cart/coverage/lcov.info.
Both changes are legitimate per
eng/skill-validator/src/docs/InvestigatingResults.md "improving the
skill vs gaming the eval" — they make the deliverable richer rather
than steering toward the skill or relaxing the rubric.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Clarify SKILL.md routing and mirror coverage assertions in eval.vally.yaml
Per rubber-duck review of the activation strategy:
1. SKILL.md description was ambiguous about coverage — the previous
wording said "DO NOT USE FOR coverage/CRAP analysis" but the
polyglot prompts heavily mention coverage tooling. In plugin mode
that could confuse the router into deferring to coverage-analysis
or crap-score even when the user wants new tests written. Clarify:
- In-scope: "configures coverage tooling (coverlet, pytest-cov,
@vitest/coverage-v8) as part of test generation"
- Out-of-scope: "analyzing existing coverage reports" (not just
"coverage")
Description re-tightened to 1003 chars (under the 1024 limit).
2. eval.vally.yaml previously asserted only that test files exist; the
eval.yaml mirror added coverage XML / lcov.info file_exists
assertions in the prior commit. Mirror those into the vally graders
so both eval schemas enforce the same coverage deliverable:
- Python: assert fixtures/python-flask-tasks/coverage.xml exists
- TS: run npx vitest run --coverage --reporter=basic then assert
fixtures/typescript-vitest-cart/coverage/lcov.info exists
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Harden polyglot fixtures so baseline cannot perfectly ace them
Both polyglot scenarios (Python Flask, TypeScript Vitest) activated the
code-testing-agent skill in isolated mode but not in plugin mode. Root
cause: the baseline (no skill loaded) was scoring 5/5, leaving the skill
no headroom to demonstrate value, so plugin-mode runs went straight to
view+write without invoking any skill.
Per InvestigatingResults.md section 8, the correct fix is to add genuine
complexity that benefits from a planning workflow rather than relaxing
rubrics or gaming activation. This commit grows both fixtures into
realistic, multi-seam libraries that genuinely reward layered test
planning.
Python (fixtures/python-flask-tasks/) additions:
- models.py: TaskPriority enum, Tag value-object with normalization /
validation, due_at on Task, is_overdue(now) helper.
- queries.py (new): TaskQuery / TaskPage / apply_query separating
filter / sort / pagination from the service. Sort by created_at,
due_at, priority, or title with ascending / descending order; filter
by status, tag, search substring, and overdue.
- repository.py: TaskRepository protocol gains delete(); added
normalize_tags() helper that the service uses for tag dedup.
- repository_sqlite.py (new): SqliteTaskRepository — a second backend
implementing the same Protocol using stdlib sqlite3 (no extra deps),
with persisted tags via a task_tags table and ON DELETE CASCADE.
- service.py: configurable max_title_length, timezone-aware due_at
validation, priority / due_at / tag mutations, reopen() and delete()
state transitions, add_tags / remove_tag with normalization.
- routes.py: new endpoints — DELETE /tasks/<id>, POST /tasks/<id>/reopen,
POST /tasks/<id>/tags, DELETE /tasks/<id>/tags/<name>. GET /tasks now
parses status / tag / q / overdue / sort / order / limit / offset and
returns a paged response with total / has_more.
- app.py: env / config-driven backend selection (TASKS_BACKEND =
memory | sqlite, TASKS_DATABASE_URL, TASKS_MAX_TITLE_LENGTH).
- README.md: documents the new layout and the expected layered test
approach.
TypeScript (fixtures/typescript-vitest-cart/) additions:
- pricing.ts: FixedAmountDiscountPolicy and CompositeDiscountPolicy
with "sum" vs "chain" stacking modes (the latter compounds discounts
on the remaining subtotal).
- tax.ts (new): TaxCalculator interface; NoTaxCalculator,
RegionalTaxCalculator (per-region rate table with default fallback),
AsyncTaxRateProvider async seam, AsyncTaxCalculator adapter.
- shipping.ts (new): ShippingCalculator interface;
FreeShippingCalculator, FlatShippingCalculator (with optional
free-over-threshold), WeightBasedShippingCalculator (bracket-based
with overflow cost).
- inventory.ts (new): PriceFetcher and InventoryChecker async seams,
InventoryError with productId / requested / available, refreshPrices()
helper.
- cart.ts: fixed pricing pipeline subtotal → discount (clamped) → tax
(on the discounted subtotal) → shipping; async checkout() that
refreshes prices, checks inventory (first denial wins,
InventoryError thrown), and supports an AsyncTaxRateProvider for
this call.
- product.ts: optional weightGrams (used by shipping) and currency.
- index.ts: barrel re-exports for all new symbols.
- README.md: documents the new layout and the expected layered test
approach.
eval.yaml + eval.vally.yaml:
- Updated both polyglot prompts to name the new modules and seams
(queries.apply_query, SqliteTaskRepository, tag endpoints,
CompositeDiscountPolicy stacking, AsyncTaxRateProvider,
WeightBasedShippingCalculator, PriceFetcher / InventoryChecker,
InventoryError). Style remains open-ended (.NET-style) — no bullet
list of test cases.
- Replaced the small rubric with outcome-based rubric items that grade
the new behaviors: pipeline composition (tax on discounted
subtotal), CompositeDiscountPolicy stacking-mode differences, weight
bracket boundary semantics, async checkout() with mocked
collaborators including InventoryError shape, queries.apply_query
pagination boundaries, SqliteTaskRepository integration tests
against an in-memory connection, multi-status assertions across
200/201/204/400/404/409, and constructor validation on the boundary
classes.
- Bumped per-command timeouts from 5m to 10m to absorb the larger
test suite.
No skill description / SKILL.md changes — the skill description is
already accurate for these scenarios (1003 / 1024 chars).
* Enforce hard 80% coverage floor on polyglot fixtures
After the previous harden commit (c78532bf7), the polyglot scenarios still
scored a 5.0/5 baseline with no plugin-mode activation. Root cause: even on
the larger fixtures the judge ticked every rubric item because the baseline
agent wrote tests that *touched* each named module — the rubric grades
"covered this concept", not "covered this code path", so a strong baseline
naturally clears it.
The previous commit hardened the SURFACE of the fixtures. This commit
hardens the SUCCESS BAR by baking coverage thresholds into the test
runners themselves, so a baseline that skips an entire module fails the
existing pytest / vitest run-command assertion outright (assertion exit
code != 0 → run fails → judge cannot give 5/5).
Python (fixtures/python-flask-tasks/pyproject.toml):
- Added `pytest-cov>=5.0,<7.0` to the [test] extra so coverage is
pre-installed (the agent no longer has to add it manually).
- Set `addopts = "-q --cov=tasks_api --cov-branch --cov-report=xml
--cov-report=term-missing --cov-fail-under=80"`. pytest now exits
non-zero when total line + branch coverage on tasks_api is below 80%,
which makes the existing `python -m pytest` assertion fail.
TypeScript (fixtures/typescript-vitest-cart/vitest.config.ts):
- Added a `coverage` block that pre-configures provider: 'v8',
reporter: ['lcov', 'text'], include: ['src/**/*.ts'], and thresholds
of lines / statements / functions ≥ 80, branches ≥ 70. The agent no
longer has to wire up the coverage provider. `npx vitest run
--coverage` exits non-zero when any threshold is unmet.
Prompts (eval.yaml + eval.vally.yaml):
- Trimmed the prompts to drop the "configure coverage if not already
there" guidance — coverage tooling is now pre-installed and the floor
is fixed. The prompt now explicitly tells the agent the threshold is
enforced and that incidental coverage will not clear the bar, so the
agent's planning workflow has a concrete cross-module target to plan
against.
READMEs: documented the new behavior so the agent reading the README
sees the coverage floor up front.
Verified locally: with an empty tests/ directory, pytest exits 1 with
"FAIL Required test coverage of 80% not reached" and `vitest run
--coverage` exits 1 with "Coverage for lines (0%) does not meet global
threshold (80%)". The existing run-command assertions therefore fail for
incomplete suites without needing any new assertion / grader.
* Address PR #709 review: snapshot deep-copy, due_at sort, tests/ wording
- cart.ts: snapshot() now deep-copies the Product on each line in addition
to the CartLine, so callers mutating the returned snapshot can no longer
reach back into the cart's internal state via line.product. (Copilot 3344261946)
- queries.py: _sort_key for 'due_at' returns a (is_none, isoformat) tuple
so we never compare across the None / not-None boundary, and never
trigger 'cannot compare naive and aware datetimes' TypeErrors when
a stray naive datetime sneaks through. (Copilot 3347273389)
- README + eval.yaml + eval.vally.yaml: reword the 'tests/ directory is
intentionally empty' phrasing to 'no test files yet — only a .gitkeep
marker' so the docs are strictly accurate about what's on disk.
(Copilot 3344580026 / 3344580055 / 3344580067 / 3344580085 / 3344580104
/ 3344580136 / 3344580156)
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Amaury Levé