Files
Amaury Levé f532cf1d94 Add coverage-touching and self-referential assertion heuristics (#635)
* Add coverage-touching and self-referential assertion heuristics

- test-anti-patterns: add 'Coverage touching' and 'Self-referential assertion'
  as named Critical-severity anti-patterns in the detection catalog
- assertion-quality: add 'Tests with self-referential assertions' metric and
  a calibration rule distinguishing tautological from valid round-trip assertions
- Add eval scenarios for both new heuristics in test-anti-patterns (coverage-
  touching ProductCatalog facade, self-referential SerializerTests)
- Add eval scenario and fixture for self-referential assertions in
  assertion-quality (ConfigTests with identity/round-trip patterns)

* Fix test-anti-patterns activation: rewrite description with USE FOR keyword list

The skill was not activating in any eval scenario because the description
used narrative prose instead of the explicit 'USE FOR:' / 'DO NOT USE FOR:'
keyword list pattern that well-activating sibling skills use (run-tests,
writing-mstest-tests). Restructured the description with trigger phrases
matching common user prompts.

* Fix eval activation, timeouts, and add missing vally scenarios

- test-anti-patterns/eval.vally.yaml: increase timeout 2m→5m, add
  coverage-touching and self-referential scenarios with constraints
- assertion-quality/eval.vally.yaml: add self-referential scenario
  with environment files, broaden assertion pattern
- test-anti-patterns/eval.yaml: add setup.files to all 6 scenarios
  to create workspace files that trigger skill discovery/activation
- assertion-quality/eval.yaml: increase timeouts 120→300 for
  self-referential and decline-request scenarios, broaden
  self-referential assertion pattern
- Create fixture files for mixed-severity, well-written,
  coverage-touching, and self-referential test-anti-patterns scenarios

* Address review comments: fix test names and use source references

- Rename ThrowsArgumentException to ThrowsArgumentOutOfRangeException in
  PaymentProcessorTests.cs fixture and inlined code in eval.yaml and
  eval.vally.yaml to match the actual exception type being asserted.
- Replace inline csproj content blocks with source references to existing
  fixture files for UserRepo, Payments, and CatalogService scenarios in
  eval.yaml to avoid drift between fixtures and eval setup.

* Address review comments: pin MSTest versions, use source refs, fix naming

- Pin MSTest to 3.8.2 in all fixture .csproj files, add standard
  properties (ImplicitUsings, Nullable, IsPackable)
- Replace inline csproj content blocks with source references to
  fixture files to avoid content drift
- Create missing .csproj fixtures for flaky-coupled, duplicate-magic,
  and assertion-quality self-referential scenarios
- Fix test method names: ThrowsArgumentException ->
  ThrowsArgumentOutOfRangeException to match the actual assertion

* Fix activation, eval timeouts, and per-skill description limits

Address remaining issues from PR #635 eval scoreboard
(run 25677669933) and skill-validator failures:

1. test-anti-patterns activation (was NOT ACTIVATED in BOTH plugin and
   isolated mode for ALL 6 scenarios — model answered everything from
   inline code without ever needing the skill):

   - eval.yaml + eval.vally.yaml: drop the giant inline `csharp` blocks
     from every prompt. Each prompt now names the workspace files
     explicitly (`Audit `Foo.Tests/FooTests.cs` for .NET test
     anti-patterns ...`) and asks for the skill's signature deliverable
     (severity-ranked list with concrete code-level fixes). The model
     must read the workspace, which gives it a chance to consult the
     skill — and the skill's structured-findings format is the actual
     value-add over inline analysis.
   - All 6 scenarios consistently get reject_tools (read-only review)
     and timeout 180s.
   - eval.vally.yaml: also added missing environment.files for each
     scenario (was prompt-only before, no fixture on disk).

2. assertion-quality `Flag assertion-free tests` plugin-mode activation
   (was bouncing to other skills because the prompt was generic `check
   whether these tests verify anything useful`):

   - eval.yaml + eval.vally.yaml: rewrite the prompt to name the file
     (`SmokeTests/ApiEndpointSmokeTests.cs`) and explicitly request
     `assertion quality and depth`, `assertion-free tests`, and
     `trivial assertions like Assert.IsNotNull` — direct trigger
     phrases for assertion-quality.
   - assertion-quality SKILL.md: extend description with explicit USE
     FOR keywords for `assertion-free tests`, `self-referential`, and
     `tautological` so plugin-mode activation isn't won by sibling
     skills like test-anti-patterns.

3. test-anti-patterns description size — was 1,301 chars (over the
   1,024 per-skill limit) AND pushed dotnet-test plugin aggregate to
   15,312 (over the 15,000 limit), failing skill-check on PR #635.
   Trimmed to 779 chars; aggregate now at 14,983.

4. Vally job-level timeout (60m → 180m) was already raised on main
   in commit 0f31228b5 (#641); inheriting that via merge of upstream/main.

Verified: skill-validator check --plugin passes (22 skills, 11 agents,
1 plugin); markdownlint clean; all 4 eval YAMLs parse OK.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address round-2 review comments: add missing production stubs, reject_tools

Address the 3 new Copilot review comments on PR #635
(IDs 3240547649, 3240547701, 3240547725):

1. coverage-touching fixture: `ProductCatalogTests.cs` references
   `new InMemoryStore()` but the fixture only declared the
   `IProductStore` interface. Audit agents could surface `this won't
   compile` as the top finding instead of the intended coverage-touching
   pattern. Added a minimal `InMemoryStore : IProductStore`
   implementation in the same file as `ProductCatalog.cs`.

2. self-referential fixture: `SerializerTests.cs` referenced 8 production
   types (Serializer, UserDto, Money, Config, NameNormalizer, Mapper,
   UserEntity, Validator) with no production code on disk. Added a new
   `Serialization/Serialization.cs` file with minimal stubs for all 8
   types so the audit can stay focused on assertion semantics. Updated
   the `Detect self-referential...` scenario in eval.yaml +
   eval.vally.yaml to copy the new file into the workspace.

3. assertion-quality `Identify self-referential assertions...` scenario:
   added `reject_tools: [bash, edit, create]` and explicit `Do not
   edit the files` wording to match the read-only audit pattern used by
   other scenarios in the file. Applied to both eval.yaml and
   eval.vally.yaml (where it lives under `constraints.reject_tools`).

Verified: skill-validator check passes, markdownlint clean, all 4
eval YAMLs parse OK.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Strengthen activation: name anti-patterns explicitly, dedupe self-referential boundary

PR #635 eval scoreboard run [25859929883](https://github.com/dotnet/skills/actions/runs/25859929883):
- 3 test-anti-patterns scenarios still NOT ACTIVATED in plugin mode (mixed
  severity, flakiness, well-written) — variance was high (CV 0.90-1.59),
  but only 1-3 of 9 plugin runs actually loaded the skill, vs 3/3 for
  scenarios that named specific patterns (coverage-touching, self-referential).
- assertion-quality 'Identify self-referential' NOT ACTIVATED in plugin —
  detectedSkills was empty (model picked neither AQ nor TAP despite both
  matching the prompt's 'self-referential or tautological' phrasing).

Two-pronged fix:

1. Reduce overlap between assertion-quality and test-anti-patterns —
   remove 'self-referential or tautological assertions' from
   test-anti-patterns USE FOR. Both skills could plausibly own this, but
   assertion-quality is the more focused owner; in plugin mode neither
   was winning, so giving assertion-quality the unambiguous trigger
   improves activation. test-anti-patterns still owns the broader 'tests
   pass but don't verify anything', and the test-anti-patterns 'Detect
   self-referential' eval scenario activates fine on the explicit
   'audit ... for .NET test anti-patterns' phrasing.

2. Strengthen the 5 problem prompts with skill-specific terminology —
   the working pattern from coverage-touching/self-referential prompts
   (3/3 and 2/3 plugin activation) was naming the specific anti-pattern
   category. Apply the same pattern to the failing scenarios:
   - test-anti-patterns/Detect mixed severity: enumerate the catalog
     (missing/no assertions, swallowed exceptions, always-true /
     self-comparing, weak negative, broad exception types, unused
     fields) and ask for findings tied to named anti-patterns.
   - test-anti-patterns/Detect flakiness: name the pattern categories
     explicitly (Thread.Sleep / DateTime.Now flakiness, ordering
     dependency, shared static state, reflection coupling, god tests).
   - test-anti-patterns/Recognize well-written: enumerate the catalog
     and ask for severity-ranked output 'Critical / Warning / Info /
     no issues found' tied to named anti-patterns.
   - assertion-quality/Identify self-referential: ask for an
     'assertion-quality analysis' / 'assertion-quality metrics' /
     'assertion-quality review' (3 mentions of the skill name) and
     name the metric (how many tests are self-referential).

Same prompt updates applied to both eval.yaml and eval.vally.yaml.
skill-validator check passes (22 skills, 11 agents, 1 plugin),
aggregate description 14,938 / 15,000.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-14 17:43:35 +02:00

146 lines
4.3 KiB
C#

using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace ConfigService.Tests;
[TestClass]
public class ConfigTests
{
[TestMethod]
public void GetTimeout_Default_ReturnsDefault()
{
var config = new AppConfig();
config.Set("Timeout", "30");
var result = config.Get("Timeout");
Assert.AreEqual("30", result);
}
[TestMethod]
public void GetRetries_Default_ReturnsDefault()
{
var config = new AppConfig();
config.Set("Retries", "3");
var result = config.Get("Retries");
Assert.AreEqual("3", result);
}
[TestMethod]
public void SetAndGet_CustomValue_RoundTrips()
{
var config = new AppConfig();
var input = "my-custom-value";
config.Set("Key", input);
var output = config.Get("Key");
Assert.AreEqual(input, output);
}
[TestMethod]
public void Serialize_Deserialize_ProducesOriginal()
{
var config = new AppConfig();
config.Set("Host", "localhost");
config.Set("Port", "5432");
var json = config.Serialize();
var restored = AppConfig.Deserialize(json);
Assert.AreEqual(config.Get("Host"), restored.Get("Host"));
Assert.AreEqual(config.Get("Port"), restored.Get("Port"));
}
[TestMethod]
public void FormatConnectionString_RoundTrip()
{
var config = new AppConfig();
config.Set("Host", "localhost");
config.Set("Port", "5432");
config.Set("Database", "mydb");
var connStr = config.FormatConnectionString();
var parsed = AppConfig.ParseConnectionString(connStr);
Assert.AreEqual(config.Get("Host"), parsed.Get("Host"));
Assert.AreEqual(config.Get("Port"), parsed.Get("Port"));
Assert.AreEqual(config.Get("Database"), parsed.Get("Database"));
}
[TestMethod]
public void ValidateKey_Valid_ReturnsSameKey()
{
var key = "ValidKey123";
var result = AppConfig.ValidateKey(key);
Assert.AreEqual(key, result);
}
[TestMethod]
public void Clone_MatchesOriginal()
{
var config = new AppConfig();
config.Set("A", "1");
config.Set("B", "2");
var clone = config.Clone();
Assert.AreEqual(config.Get("A"), clone.Get("A"));
Assert.AreEqual(config.Get("B"), clone.Get("B"));
}
[TestMethod]
public void ToDisplayString_ReturnsString()
{
var config = new AppConfig();
config.Set("Env", "Production");
var display = config.ToDisplayString();
Assert.IsNotNull(display);
}
}
// --- Production code (in same file for test simplicity) ---
public class AppConfig
{
private readonly Dictionary<string, string> _values = new();
public void Set(string key, string value) => _values[ValidateKey(key)] = value;
public string? Get(string key) => _values.GetValueOrDefault(key);
public string Serialize() =>
System.Text.Json.JsonSerializer.Serialize(_values);
public static AppConfig Deserialize(string json)
{
var config = new AppConfig();
var dict = System.Text.Json.JsonSerializer.Deserialize<Dictionary<string, string>>(json)!;
foreach (var kv in dict)
config.Set(kv.Key, kv.Value);
return config;
}
public string FormatConnectionString() =>
$"Host={Get("Host")};Port={Get("Port")};Database={Get("Database")}";
public static AppConfig ParseConnectionString(string connStr)
{
var config = new AppConfig();
foreach (var part in connStr.Split(';'))
{
var kv = part.Split('=', 2);
if (kv.Length == 2) config.Set(kv[0], kv[1]);
}
return config;
}
public static string ValidateKey(string key)
{
if (string.IsNullOrWhiteSpace(key))
throw new ArgumentException("Key cannot be empty", nameof(key));
if (key.Contains(' '))
throw new ArgumentException("Key cannot contain spaces", nameof(key));
return key;
}
public AppConfig Clone()
{
var clone = new AppConfig();
foreach (var kv in _values)
clone.Set(kv.Key, kv.Value);
return clone;
}
public string ToDisplayString() =>
string.Join(", ", _values.Select(kv => $"{kv.Key}={kv.Value}"));
}