mirror of
https://github.com/NVIDIA/skills.git
synced 2026-09-14 15:41:50 +08:00
Moshe Abramovitch
99772d9192
SkillEvaluator's report format changed in July 2026. It stopped emitting `NVSkills-Eval profile` and `Pass threshold`, both of which the aggregator reads, so those columns in benchmarks.json have been emptying one skill at a time as each is re-evaluated: 0% null through June, 57% in July, 100% for every skill evaluated in August. Nothing failed. The regeneration succeeded, the schema stayed valid, --check passed, and the data quietly degraded for six weeks before anyone noticed. Neither field is recoverable from a v3 report. `profile` is gone entirely. The only "50%" left is a glossary sentence that is byte-identical across skills with 4, 5 and 18 tasks, so it is template prose rather than a measurement — scraping it would stamp an unverified pass threshold on every skill in the file whose purpose is recording provenance. Both stay null until SkillEvaluator emits them as real fields; a NO_FABRICATION note records why. What v3 does add is stronger provenance the parser was discarding: - evaluator_version which build produced the score - dataset_digest sha256 of the exact task set, unique per skill - validation_status pipeline outcome for the run Capturing these follows the existing multi-pattern idiom already used for `tasks`, so v1/v2 reports are unaffected and simply leave them null. The generic fix is null_rate_regressions(), which compares a regeneration against the committed benchmarks.json and refuses to write when any field lost values, counting only skills present in both so new additions cannot false-fire. --allow-null-regressions lands a deliberate retirement without requiring a code edit under pressure. Verified against the real catalog: it does not fire today, it does fire when the July drift is replayed over 20 reports, and it flags the exact fields lost in PR #466. Adds the first tests for .github/scripts (16 cases, two real BENCHMARK.md fixtures, one per format). Confirmed no regression: every pre-existing field is byte-identical across all 341 skills. Also adds the SPDX header aggregate_benchmarks.py was missing; it was the only .py in .github/scripts/ without one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>