mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
f532cf1d94
* Add coverage-touching and self-referential assertion heuristics
- test-anti-patterns: add 'Coverage touching' and 'Self-referential assertion'
as named Critical-severity anti-patterns in the detection catalog
- assertion-quality: add 'Tests with self-referential assertions' metric and
a calibration rule distinguishing tautological from valid round-trip assertions
- Add eval scenarios for both new heuristics in test-anti-patterns (coverage-
touching ProductCatalog facade, self-referential SerializerTests)
- Add eval scenario and fixture for self-referential assertions in
assertion-quality (ConfigTests with identity/round-trip patterns)
* Fix test-anti-patterns activation: rewrite description with USE FOR keyword list
The skill was not activating in any eval scenario because the description
used narrative prose instead of the explicit 'USE FOR:' / 'DO NOT USE FOR:'
keyword list pattern that well-activating sibling skills use (run-tests,
writing-mstest-tests). Restructured the description with trigger phrases
matching common user prompts.
* Fix eval activation, timeouts, and add missing vally scenarios
- test-anti-patterns/eval.vally.yaml: increase timeout 2m→5m, add
coverage-touching and self-referential scenarios with constraints
- assertion-quality/eval.vally.yaml: add self-referential scenario
with environment files, broaden assertion pattern
- test-anti-patterns/eval.yaml: add setup.files to all 6 scenarios
to create workspace files that trigger skill discovery/activation
- assertion-quality/eval.yaml: increase timeouts 120→300 for
self-referential and decline-request scenarios, broaden
self-referential assertion pattern
- Create fixture files for mixed-severity, well-written,
coverage-touching, and self-referential test-anti-patterns scenarios
* Address review comments: fix test names and use source references
- Rename ThrowsArgumentException to ThrowsArgumentOutOfRangeException in
PaymentProcessorTests.cs fixture and inlined code in eval.yaml and
eval.vally.yaml to match the actual exception type being asserted.
- Replace inline csproj content blocks with source references to existing
fixture files for UserRepo, Payments, and CatalogService scenarios in
eval.yaml to avoid drift between fixtures and eval setup.
* Address review comments: pin MSTest versions, use source refs, fix naming
- Pin MSTest to 3.8.2 in all fixture .csproj files, add standard
properties (ImplicitUsings, Nullable, IsPackable)
- Replace inline csproj content blocks with source references to
fixture files to avoid content drift
- Create missing .csproj fixtures for flaky-coupled, duplicate-magic,
and assertion-quality self-referential scenarios
- Fix test method names: ThrowsArgumentException ->
ThrowsArgumentOutOfRangeException to match the actual assertion
* Fix activation, eval timeouts, and per-skill description limits
Address remaining issues from PR #635 eval scoreboard
(run 25677669933) and skill-validator failures:
1. test-anti-patterns activation (was NOT ACTIVATED in BOTH plugin and
isolated mode for ALL 6 scenarios — model answered everything from
inline code without ever needing the skill):
- eval.yaml + eval.vally.yaml: drop the giant inline `csharp` blocks
from every prompt. Each prompt now names the workspace files
explicitly (`Audit `Foo.Tests/FooTests.cs` for .NET test
anti-patterns ...`) and asks for the skill's signature deliverable
(severity-ranked list with concrete code-level fixes). The model
must read the workspace, which gives it a chance to consult the
skill — and the skill's structured-findings format is the actual
value-add over inline analysis.
- All 6 scenarios consistently get reject_tools (read-only review)
and timeout 180s.
- eval.vally.yaml: also added missing environment.files for each
scenario (was prompt-only before, no fixture on disk).
2. assertion-quality `Flag assertion-free tests` plugin-mode activation
(was bouncing to other skills because the prompt was generic `check
whether these tests verify anything useful`):
- eval.yaml + eval.vally.yaml: rewrite the prompt to name the file
(`SmokeTests/ApiEndpointSmokeTests.cs`) and explicitly request
`assertion quality and depth`, `assertion-free tests`, and
`trivial assertions like Assert.IsNotNull` — direct trigger
phrases for assertion-quality.
- assertion-quality SKILL.md: extend description with explicit USE
FOR keywords for `assertion-free tests`, `self-referential`, and
`tautological` so plugin-mode activation isn't won by sibling
skills like test-anti-patterns.
3. test-anti-patterns description size — was 1,301 chars (over the
1,024 per-skill limit) AND pushed dotnet-test plugin aggregate to
15,312 (over the 15,000 limit), failing skill-check on PR #635.
Trimmed to 779 chars; aggregate now at 14,983.
4. Vally job-level timeout (60m → 180m) was already raised on main
in commit 0f31228b5 (#641); inheriting that via merge of upstream/main.
Verified: skill-validator check --plugin passes (22 skills, 11 agents,
1 plugin); markdownlint clean; all 4 eval YAMLs parse OK.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address round-2 review comments: add missing production stubs, reject_tools
Address the 3 new Copilot review comments on PR #635
(IDs 3240547649, 3240547701, 3240547725):
1. coverage-touching fixture: `ProductCatalogTests.cs` references
`new InMemoryStore()` but the fixture only declared the
`IProductStore` interface. Audit agents could surface `this won't
compile` as the top finding instead of the intended coverage-touching
pattern. Added a minimal `InMemoryStore : IProductStore`
implementation in the same file as `ProductCatalog.cs`.
2. self-referential fixture: `SerializerTests.cs` referenced 8 production
types (Serializer, UserDto, Money, Config, NameNormalizer, Mapper,
UserEntity, Validator) with no production code on disk. Added a new
`Serialization/Serialization.cs` file with minimal stubs for all 8
types so the audit can stay focused on assertion semantics. Updated
the `Detect self-referential...` scenario in eval.yaml +
eval.vally.yaml to copy the new file into the workspace.
3. assertion-quality `Identify self-referential assertions...` scenario:
added `reject_tools: [bash, edit, create]` and explicit `Do not
edit the files` wording to match the read-only audit pattern used by
other scenarios in the file. Applied to both eval.yaml and
eval.vally.yaml (where it lives under `constraints.reject_tools`).
Verified: skill-validator check passes, markdownlint clean, all 4
eval YAMLs parse OK.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Strengthen activation: name anti-patterns explicitly, dedupe self-referential boundary
PR #635 eval scoreboard run [25859929883](https://github.com/dotnet/skills/actions/runs/25859929883):
- 3 test-anti-patterns scenarios still NOT ACTIVATED in plugin mode (mixed
severity, flakiness, well-written) — variance was high (CV 0.90-1.59),
but only 1-3 of 9 plugin runs actually loaded the skill, vs 3/3 for
scenarios that named specific patterns (coverage-touching, self-referential).
- assertion-quality 'Identify self-referential' NOT ACTIVATED in plugin —
detectedSkills was empty (model picked neither AQ nor TAP despite both
matching the prompt's 'self-referential or tautological' phrasing).
Two-pronged fix:
1. Reduce overlap between assertion-quality and test-anti-patterns —
remove 'self-referential or tautological assertions' from
test-anti-patterns USE FOR. Both skills could plausibly own this, but
assertion-quality is the more focused owner; in plugin mode neither
was winning, so giving assertion-quality the unambiguous trigger
improves activation. test-anti-patterns still owns the broader 'tests
pass but don't verify anything', and the test-anti-patterns 'Detect
self-referential' eval scenario activates fine on the explicit
'audit ... for .NET test anti-patterns' phrasing.
2. Strengthen the 5 problem prompts with skill-specific terminology —
the working pattern from coverage-touching/self-referential prompts
(3/3 and 2/3 plugin activation) was naming the specific anti-pattern
category. Apply the same pattern to the failing scenarios:
- test-anti-patterns/Detect mixed severity: enumerate the catalog
(missing/no assertions, swallowed exceptions, always-true /
self-comparing, weak negative, broad exception types, unused
fields) and ask for findings tied to named anti-patterns.
- test-anti-patterns/Detect flakiness: name the pattern categories
explicitly (Thread.Sleep / DateTime.Now flakiness, ordering
dependency, shared static state, reflection coupling, god tests).
- test-anti-patterns/Recognize well-written: enumerate the catalog
and ask for severity-ranked output 'Critical / Warning / Info /
no issues found' tied to named anti-patterns.
- assertion-quality/Identify self-referential: ask for an
'assertion-quality analysis' / 'assertion-quality metrics' /
'assertion-quality review' (3 mentions of the skill name) and
name the metric (how many tests are self-referential).
Same prompt updates applied to both eval.yaml and eval.vally.yaml.
skill-validator check passes (22 skills, 11 agents, 1 plugin),
aggregate description 14,938 / 15,000.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
146 lines
4.3 KiB
C#
146 lines
4.3 KiB
C#
using Microsoft.VisualStudio.TestTools.UnitTesting;
|
|
|
|
namespace ConfigService.Tests;
|
|
|
|
[TestClass]
|
|
public class ConfigTests
|
|
{
|
|
[TestMethod]
|
|
public void GetTimeout_Default_ReturnsDefault()
|
|
{
|
|
var config = new AppConfig();
|
|
config.Set("Timeout", "30");
|
|
var result = config.Get("Timeout");
|
|
Assert.AreEqual("30", result);
|
|
}
|
|
|
|
[TestMethod]
|
|
public void GetRetries_Default_ReturnsDefault()
|
|
{
|
|
var config = new AppConfig();
|
|
config.Set("Retries", "3");
|
|
var result = config.Get("Retries");
|
|
Assert.AreEqual("3", result);
|
|
}
|
|
|
|
[TestMethod]
|
|
public void SetAndGet_CustomValue_RoundTrips()
|
|
{
|
|
var config = new AppConfig();
|
|
var input = "my-custom-value";
|
|
config.Set("Key", input);
|
|
var output = config.Get("Key");
|
|
Assert.AreEqual(input, output);
|
|
}
|
|
|
|
[TestMethod]
|
|
public void Serialize_Deserialize_ProducesOriginal()
|
|
{
|
|
var config = new AppConfig();
|
|
config.Set("Host", "localhost");
|
|
config.Set("Port", "5432");
|
|
var json = config.Serialize();
|
|
var restored = AppConfig.Deserialize(json);
|
|
Assert.AreEqual(config.Get("Host"), restored.Get("Host"));
|
|
Assert.AreEqual(config.Get("Port"), restored.Get("Port"));
|
|
}
|
|
|
|
[TestMethod]
|
|
public void FormatConnectionString_RoundTrip()
|
|
{
|
|
var config = new AppConfig();
|
|
config.Set("Host", "localhost");
|
|
config.Set("Port", "5432");
|
|
config.Set("Database", "mydb");
|
|
var connStr = config.FormatConnectionString();
|
|
var parsed = AppConfig.ParseConnectionString(connStr);
|
|
Assert.AreEqual(config.Get("Host"), parsed.Get("Host"));
|
|
Assert.AreEqual(config.Get("Port"), parsed.Get("Port"));
|
|
Assert.AreEqual(config.Get("Database"), parsed.Get("Database"));
|
|
}
|
|
|
|
[TestMethod]
|
|
public void ValidateKey_Valid_ReturnsSameKey()
|
|
{
|
|
var key = "ValidKey123";
|
|
var result = AppConfig.ValidateKey(key);
|
|
Assert.AreEqual(key, result);
|
|
}
|
|
|
|
[TestMethod]
|
|
public void Clone_MatchesOriginal()
|
|
{
|
|
var config = new AppConfig();
|
|
config.Set("A", "1");
|
|
config.Set("B", "2");
|
|
var clone = config.Clone();
|
|
Assert.AreEqual(config.Get("A"), clone.Get("A"));
|
|
Assert.AreEqual(config.Get("B"), clone.Get("B"));
|
|
}
|
|
|
|
[TestMethod]
|
|
public void ToDisplayString_ReturnsString()
|
|
{
|
|
var config = new AppConfig();
|
|
config.Set("Env", "Production");
|
|
var display = config.ToDisplayString();
|
|
Assert.IsNotNull(display);
|
|
}
|
|
}
|
|
|
|
// --- Production code (in same file for test simplicity) ---
|
|
|
|
public class AppConfig
|
|
{
|
|
private readonly Dictionary<string, string> _values = new();
|
|
|
|
public void Set(string key, string value) => _values[ValidateKey(key)] = value;
|
|
public string? Get(string key) => _values.GetValueOrDefault(key);
|
|
|
|
public string Serialize() =>
|
|
System.Text.Json.JsonSerializer.Serialize(_values);
|
|
|
|
public static AppConfig Deserialize(string json)
|
|
{
|
|
var config = new AppConfig();
|
|
var dict = System.Text.Json.JsonSerializer.Deserialize<Dictionary<string, string>>(json)!;
|
|
foreach (var kv in dict)
|
|
config.Set(kv.Key, kv.Value);
|
|
return config;
|
|
}
|
|
|
|
public string FormatConnectionString() =>
|
|
$"Host={Get("Host")};Port={Get("Port")};Database={Get("Database")}";
|
|
|
|
public static AppConfig ParseConnectionString(string connStr)
|
|
{
|
|
var config = new AppConfig();
|
|
foreach (var part in connStr.Split(';'))
|
|
{
|
|
var kv = part.Split('=', 2);
|
|
if (kv.Length == 2) config.Set(kv[0], kv[1]);
|
|
}
|
|
return config;
|
|
}
|
|
|
|
public static string ValidateKey(string key)
|
|
{
|
|
if (string.IsNullOrWhiteSpace(key))
|
|
throw new ArgumentException("Key cannot be empty", nameof(key));
|
|
if (key.Contains(' '))
|
|
throw new ArgumentException("Key cannot contain spaces", nameof(key));
|
|
return key;
|
|
}
|
|
|
|
public AppConfig Clone()
|
|
{
|
|
var clone = new AppConfig();
|
|
foreach (var kv in _values)
|
|
clone.Set(kv.Key, kv.Value);
|
|
return clone;
|
|
}
|
|
|
|
public string ToDisplayString() =>
|
|
string.Join(", ", _values.Select(kv => $"{kv.Key}={kv.Value}"));
|
|
}
|