SkillEvaluator's report format changed in July 2026. It stopped emitting
`NVSkills-Eval profile` and `Pass threshold`, both of which the aggregator
reads, so those columns in benchmarks.json have been emptying one skill at a
time as each is re-evaluated: 0% null through June, 57% in July, 100% for
every skill evaluated in August. Nothing failed. The regeneration succeeded,
the schema stayed valid, --check passed, and the data quietly degraded for
six weeks before anyone noticed.
Neither field is recoverable from a v3 report. `profile` is gone entirely.
The only "50%" left is a glossary sentence that is byte-identical across
skills with 4, 5 and 18 tasks, so it is template prose rather than a
measurement — scraping it would stamp an unverified pass threshold on every
skill in the file whose purpose is recording provenance. Both stay null until
SkillEvaluator emits them as real fields; a NO_FABRICATION note records why.
What v3 does add is stronger provenance the parser was discarding:
- evaluator_version which build produced the score
- dataset_digest sha256 of the exact task set, unique per skill
- validation_status pipeline outcome for the run
Capturing these follows the existing multi-pattern idiom already used for
`tasks`, so v1/v2 reports are unaffected and simply leave them null.
The generic fix is null_rate_regressions(), which compares a regeneration
against the committed benchmarks.json and refuses to write when any field
lost values, counting only skills present in both so new additions cannot
false-fire. --allow-null-regressions lands a deliberate retirement without
requiring a code edit under pressure. Verified against the real catalog: it
does not fire today, it does fire when the July drift is replayed over 20
reports, and it flags the exact fields lost in PR #466.
Adds the first tests for .github/scripts (16 cases, two real BENCHMARK.md
fixtures, one per format). Confirmed no regression: every pre-existing field
is byte-identical across all 341 skills.
Also adds the SPDX header aggregate_benchmarks.py was missing; it was the
only .py in .github/scripts/ without one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
The --no-ai case asserted on main()'s bare exit code. --check returns 1
for output drift as well as for skill warnings, so any PR that edits
skill content failed this test for reasons unrelated to enrichment.
PR #445 is the first to hit it: 70 changed files including SKILL.md at
62+/147-, which legitimately drifts the checked-in metadata.
Neutralise diff_text for the duration of the run so the exit code is
attributable to the warning gate alone. Drift is already covered by
case 4.
A first attempt simply excused drift with 'rc == 0 or drifted'. That was
worse than the original bug: removing the --no-ai gate entirely still
passed, because drift was present. Mutation-checked properly this time.
Verified against PR #445's content: reproduces the failure without this
change, passes with it, and still fails when the --no-ai gate is removed
-- on both #445's tree and a clean one.
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
The previous commit made any unbuilt skill non-zero in both modes. That
is wrong for --no-ai: PR CI runs without an inference key by design, so
every newly synced skill arrives there unenriched and would red-line all
open PRs until the regenerate job ran. Caught by CI on this branch --
three freshly synced hsb-ip skills failed the check step.
Gate the non-zero exit on AI being available. Warnings still print in
both modes; a skill that cannot be enriched WITH AI available still
fails. Adds test cases pinning both halves so neither can regress.
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
The recovery and reporting paths only execute when a skill cannot be
built, so a healthy catalog never exercises them and a regression stays
invisible until it drops a skill from the published catalog.
- add test_generate_skill_metadata.py: induces enrichment failures against
a stub inference API, no network, nothing written to the repo. Covers
amendment-failure-keeps-entry, amendment-success, unbuildable-new-skill
omitted-and-warned, and unbuildable-published-skill recovered
byte-identically. Fixtures are selected from real metadata.json entries
at runtime so a rename or retirement does not break them.
- wire it into the PR check job, ahead of the byte-stability check.
- the issue-body script still read warnings.omitted_skills, which the key
rename left dangling; it would have silently reported zero affected
skills on every partial run. Read stale_skills, and correct the copy:
recovered skills are stale, not excluded.
Mutation-checked: disabling recovery fails 3 assertions, routing
amendment failures back into skill_warnings fails 2.
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
Moving the warning block above the --check early return dropped its
return 1, so a run that omitted a skill would have exited 0. Set the
code once and return it from both the --check and write paths: outputs
are still written so the catalog does not lose a skill, but CI cannot
read a partial success as a pass.
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
Addresses review feedback on #408.
- workflow: set -o pipefail. Piping the generator into tee reported tee's
exit status, so a failing generator was read as rc=0 and the failure
path never ran. This is what let the original silent drop through CI.
- generator: when a skill cannot be rebuilt, reuse its last-good entry
from the checked-in metadata.json (schema-validated first) instead of
dropping it. Only entries with no valid prior are omitted, and those
are the only ones that fail the run.
- generator: an amendment failure on an already-complete entry is no
longer reported as an omission. The entry ships unchanged with its
existing values, so it is recorded as a retained notice, not a warning.
- generator: print retained notices and stale-skill warnings before the
--check early return, so PR CI surfaces them too.
- rename the warnings JSON key omitted_skills -> stale_skills to match
what the entries now mean.
Verified with an induced-failure harness driving a stub inference API:
11/11 assertions pass (entry kept + values preserved on amendment
failure; byte-identical recovery from baseline; genuinely-new unbuildable
skill still omitted and still fails the run; published skill unbuildable
-> catalog written, not delisted, failure surfaced).
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
A skill classified materially_changed is sent back to the AI client to see
whether its existing metadata values should be amended. Every required
field already holds a valid value at that point, so the amendment is
optional — but an EnrichmentError returned None, dropping the skill from
metadata.json and skills.sh.json entirely.
The generator does report this, printing a PARTIAL SUCCESS block naming the
skill. What it does not do is stop: it exits 0, writes the reduced file, and
the regeneration PR opens as normal. The output is schema-valid and
internally consistent, so --check passes and the round-trip validation is
handed the missing path as skipped_paths.
This happened on 2026-08-03 to cuopt-multi-objective-exploration, legitimately
materially_changed after the 15:04 sync. The enrichment call returned HTTP 403
three minutes after the identical call had succeeded; 403 was not in the
retryable set, so it was terminal. Run 30842167516 emitted 323 skills, run
30842382996 emitted 322 and opened a PR deleting the entry.
Three changes:
- On enrichment failure in the amendment path, keep the existing metadata and
record a warning instead of returning None.
- Fail the run if a discovered skill produces no entry, when that skill was in
the previous metadata.json or when AI enrichment was available. Skills
dropped via metadata-exclusions.yaml never reach this point. Because the
workflow opens its validation issue on a non-zero exit, this also gives the
case an issue without new wiring.
- Retry 403 alongside 429 and 5xx. The gateway returns 403 for transient edge
rejections as well as real permission failures; retrying costs ~7s on a
genuine failure and saves an enrichment on a transient one.
Verified against the real catalog with a stubbed API, main vs this branch:
403 then 200 before: terminal after 1 call
after: recovered on call 2, 5 fields returned
persistent 403 before: 1 call, no "attempts" in message
after: 4 calls, terminal, message cites attempts
amend-path failure before: entry dropped
after: entry kept, 5 metadata fields intact
forced drop, write before: "updated (322 skills)", exit 0
after: VALIDATION FAILED naming the skill, exit 1
Normal run unchanged: metadata.json unchanged (323 skills), both drift gates
exit 0.
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
Two corrections, neither of which had been reflected in the generated
files:
metadata.json picks up the cuopt-multi-objective-exploration description
reworded in the 8/3 re-sign.
benchmarks.json is regenerated against 937d0e3, which made "±0 points"
parse as 0.0 rather than None. Dimensions with no change had been
dropped from the average-uplift denominator, so every skill carrying a
v2 report was reporting an inflated figure. Six skills move, all
downward and all more accurate:
nemo-automodel-distributed-training 40.88 -> 32.7
nemo-relay-plugin-observability 38.0 -> 34.2
nemo-relay-get-started 36.89 -> 33.2
nemo-relay-debug-runtime-integration 30.0 -> 24.0
cuopt-multi-objective-exploration 29.3 -> 28.2
portfolio-optimization 19.0 -> 17.1
Skill count and row count are unchanged at 323 and 3000.
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
v2 evaluation reports write an unchanged dimension as "±0 points". The
uplift capture in CELL/CELL_V2 accepted only + and -, so those cells fell
through as None. average_uplift() skips None entries, so the row dropped
out of the denominator rather than counting as zero — inflating the
published average for any skill that scored the same with and without the
skill applied.
Accept ± in both cell patterns and route the captures through
parse_uplift(), which normalises ±0 to 0.0.
Re-running the aggregator over all 323 skills corrects six averages, all
downward, and clears every null uplift row (7 -> 0 of 3000):
nemo-automodel-distributed-training 40.88 -> 32.7
nemo-relay-plugin-observability 38.0 -> 34.2
nemo-relay-get-started 36.89 -> 33.2
nemo-relay-debug-runtime-integration 30.0 -> 24.0
cuopt-multi-objective-exploration 29.3 -> 28.2
portfolio-optimization 19.0 -> 17.1
No other skill's figures change. benchmarks.json is left to the generator
workflow to regenerate.
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
SkillEvaluator 0.9.x emits a redesigned BENCHMARK.md. The aggregator was
written against the older layout, so every re-signed skill silently lost
its evaluation data in benchmarks.json: has_results went false, with null
tasks, no agents and no uplift, even though the underlying report carried
richer results than before.
Both layouts are now accepted, since the catalog will hold a mix until
every skill has been re-signed:
- Evaluation Summary (v1) and Evaluation Metadata (v2) field lists
- "Agents Used" bullets (v1) and the inline "- Agents:" line (v2)
- "## Results" and "## Results at a Glance" tables, keyed on either
Dimension or Measure
- v1 cells ("100% (+70%)") and v2 cells ("45% -> 98% (+53 points)"),
recording the skill-assisted score in both so the series stays
comparable
- agent names normalised, so v2's "Claude Code (`provider/model`)" and
v1's `claude-code` aggregate as one agent
Also fixes a verdict misread. In v2 the real verdict is a callout above
the report body, and the old pattern instead matched the methodology
bullet "- Overall verdict: PASS only when every configured dimension
passes...", which appears in every v2 report. Any failing v2 skill would
have been recorded as PASS. The v1 pattern is now anchored to
end-of-line, and the v2 callout is matched explicitly.
Effect on the current catalog: 5 skills regain results (skills without
results 27 -> 22) and result rows go 2955 -> 3000. Only the 7 skills
already carrying v2 reports change; v1 records are untouched.
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Rename components.d/cufolio.yml → components.d/portfolio-optimization.yml
(filename must match the name slug for auto-generated README version cell)
- Regenerate skills.sh.json and metadata.json: removes cufolio entry
(was blocking the CI check gate since committed file diffed from generated output)
Signed-off-by: Ramakrishna Prabhu <ramakrishnap@nvidia.com>
Picks up the SKILL.md description reworded in the 7/28 resync. Generated
with `generate-skill-metadata.py --no-ai`; 322 skills unchanged, 1
materially changed.
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The enrichment client capped completions at max_tokens=512. Reasoning
models spend that budget on hidden reasoning tokens before emitting any
visible content, so a skill whose classification needed longer reasoning
came back with empty content (finish_reason=length) on every run and was
silently excluded from metadata.json / skills.sh.json under the
partial-success policy (seen deterministically on
nemo-relay-debug-runtime-integration since 2026-07-16).
- Raise max_tokens to 4096; the JSON output is tiny, the headroom is
for reasoning.
- Move response parsing inside the retry loop so an empty or garbled
body is retried with backoff instead of permanently dropping the
skill from that run's output.
- Include finish_reason and a content excerpt in the terminal error so
the next such failure is diagnosable from the workflow log alone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
Catalog-only discovery/router skill staged via direct PR (no components.d
upstream); registered in catalog-exceptions.yml so the hourly sync won't
prune it. Includes the NVSkills validation signature (skill.oms.sig) and
the current benchmarks.json / metadata.json rollups.
History linearized into a single commit to replace the branch's merge
commits, which blocked the repo's rebase-merge (only allowed method) with
"Base branch was modified". Tree is identical to the prior branch tip.
Co-authored-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
Signed-off-by: Jason Dudash <jdudash@nvidia.com>
Per review: manual-components.yml already declares the five manually
staged Physical AI skills with their component name (and feeds README
regeneration). Listing them again in catalog-exceptions.yml created a
second source of truth. Now:
- The benchmarks aggregator and orphan pruner read manual-components.yml
directly (catalog_dirs are expected, component name attributed), so
the Physical AI team keeps one edit surface.
- catalog-exceptions.yml carries only truly uncataloged dirs
(cuopt-developer), with its header documenting the three-tier
registry: components.d, manual-components, exceptions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
Six skills exist in the catalog without a components.d registration by
design (contributor-facing cuopt-developer; five manually curated
Physical AI Hub skills). Their component field in benchmarks.json was
null, rendering as 'Unmapped' on the adoption dashboard's component
view.
Let catalog-exceptions.yml entries declare a display component and use
it as a fallback in the benchmarks aggregator (components.d remains
authoritative where a registration exists). All 229 skills in the feed
now carry a component; output verified byte-stable.
Reported by Michael Raheem (dashboard component view), 2026-07-09.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
The sync workflow, README regenerator, and metadata generator all glob
components.d/*.yml as the component registry. catalog-exceptions.yml
(added in #331) has no name/repo, so the sync manufactured a 'null'
component and reported 'null (null): clone failed' on every run —
first observed in sync PR #327.
Move the exceptions file to the repository root where no component
glob can sweep it, update the prune script path (and drop its now
unneeded skip-case), and update the docs. Verified locally: the
aggregated config has 33 components and zero null entries, and the
pruner still resolves exceptions correctly from the new path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
Deregistering a skill from components.d did not remove its catalog copy
— the sync only writes declared paths and never prunes. Three teams
(Dynamo, TAO, cuOpt) independently assumed deregistration removes the
dir, leaving stale skills live on the catalog and downstream surfaces.
- .github/scripts/prune-orphans.sh: after the sync loop, delete any
top-level skills/ dir with no components.d registration and no entry
in the exceptions file. Safety rails: pruning is skipped for the run
if any component yml fails to parse (a parse error would make that
component's skills look unregistered), and if more than PRUNE_CAP=5
dirs would prune at once nothing is deleted — the list is surfaced
for human triage instead.
- components.d/catalog-exceptions.yml: explicit allowlist for dirs that
intentionally exist without a registration, seeded with the
contributor-facing cuopt-developer and the five manually curated
Physical AI Hub skills (which testing confirmed would otherwise be
treated as orphans).
- sync-skills.yml: run the prune after compliance enforcement; register
pruning as a change so a prune-only sync still opens a PR; list
pruned dirs (and any over-cap skip) in the sync PR body.
- components.d/README.md: document that deregistration now removes the
catalog copy on the next sync.
Tested locally against the live repo state: correctly identified all 11
current orphans, refused at the default cap, pruned exactly the 6
deregistered dirs with cap raised (exceptions honored), idempotent
after prune, and safe on parse errors, empty declared sets, and a
missing exceptions file.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
Review feedback from mraheem-arch:
- Replace nested per-skill results with a flat results[] array (one row
per skill x dimension x agent, denormalized with catalog_dir and
component) so BI tools can load it as a table and join on catalog_dir.
skills[] remains as a flat per-skill summary with a has_results flag.
- Parse agent columns from the Results table header instead of assuming
they match Agents Used order, so reordered or partial columns attach
scores to the right agent.
- Bump schema_version to 2.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
- aggregate_benchmarks.py: add --check mode (exit 1 when benchmarks.json
does not match skills/*/BENCHMARK.md); drop the embedded commit SHA so
the checked-in file is byte-stable and drift-checkable.
- generate-skill-metadata.yml: validate benchmarks.json in PR mode,
regenerate it in the post-sync/dispatch job, and include it in the
change detection and the automated regeneration PR.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
Aggregate every skill's BENCHMARK.md evaluation report into a single
machine-readable benchmarks.json at the repo root, so downstream tooling
can consume per-skill scores and uplift-vs-baseline data without parsing
markdown.
- .github/scripts/aggregate_benchmarks.py walks skills/*/BENCHMARK.md,
extracts the evaluation summary, agents, and per-dimension results
(score + uplift per agent), maps each skill to its component via
components.d/, and computes a per-skill average uplift.
- benchmarks.json covers all 235 skills; 212 have Tier-3 dimension
results, 23 are listed under skills_without_results.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
Two generator bugs compounded to mislabel 9 TAO skills on downstream
surfaces that group by product.primary (e.g. build.nvidia.com):
- The components.d lookup map was keyed by the source-repo path from
each yml entry, but build_skill_entry looks up the catalog path.
Products whose source layout nests skills under subdirs (TAO:
skills/models/<name>/) never matched, so all their skills fell
through to AI enrichment, which guessed the product label from
content ('Cosmos', 'Brev', 'TAO', 'TAO Toolkit API', 'DGX Cloud').
Key the map by skills/<catalog_dir> instead.
- The component-derived product was only applied when the field was
missing, so wrong carried values persisted forever. The registered
component name (when it is a valid taxonomy value) is now
authoritative over the carried baseline.
Also remove skills/tao-run-on-lepton/: deleted from the source repo
and from components.d/tao-toolkit.yml, but the sync never prunes
catalog dirs whose registration was removed, leaving a stale entry
live on downstream surfaces.
Regenerated metadata.json (234 skills; 8 product.primary corrections,
all TAO) and skills.sh.json with --no-ai; outputs verified byte-stable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
Post-consolidation cleanup, confirmed with the cuOpt team:
- Remove four skills/ dirs that were deregistered from components.d by
the cuOpt consolidation but never deleted (the sync copies declared
paths and does not prune): cuopt-numerical-optimization-api-python,
cuopt-routing-formulation, cuopt-server-common, cuopt-skill-evolution.
(cuopt-developer stays — intentionally uncataloged, contributor-facing.)
- README.md + docs/advanced-install.mdx: the install examples used the
now-removed -api-python skill as the canonical name; swap all
occurrences to the consolidated cuopt-numerical-optimization-api.
- README.md: add a "Keep Skills Up to Date" section covering
npx skills update / list / check (all verified against the current
CLI), so users refreshing installs pick up renames/consolidations.
- Regenerate metadata.json (231 skills) and skills.sh.json; outputs
verified byte-stable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
Add a Cursor track to the plugin build pipeline so each catalog plugin in
plugins.d/ also emits a .cursor-plugin/plugin.json manifest and is
registered in a top-level .cursor-plugin/marketplace.json, mirroring the
existing Claude (.claude-plugin/) and Codex (.codex-plugin/) tracks.
- build-plugins.py: add render_cursor_plugin_json() and
upsert_cursor_marketplace(); wire both into build_catalog_plugin() and
main(); include the cursor marketplace in the drift check.
- Cursor manifest omits the skills field so Cursor folder discovery scans
skills/ (verified loading via cursor-agent --plugin-dir); projects the
author down to {name}; strips the leading ./ from logo for
raw.githubusercontent resolution.
- Add .cursor-plugin/marketplace.json (owner per Cursor schema); document
marketplace_enabled.cursor in _defaults.yml and nvidia-skills.yml.
- validate-plugins.yml: watch .cursor-plugin/marketplace.json and document
the Cursor validation story.
- Update plugins.d/README.md and the build-plugins.py docstring.
Signed-off-by: Jason Dudash <jdudash@nvidia.com>
Correct the file headers and LICENSE preamble to match the OSS CC-BY-4.0/Apache-2.0
dual-license template (nvidia.atlassian.net OSS page 3151960247):
- File headers now use 'SPDX-License-Identifier: CC-BY-4.0 AND Apache-2.0' (was Apache-2.0)
with SPDX-FileCopyrightText form, across the 7 catalog-owned files.
- LICENSE opens with the prescribed two-line dual-license preamble.
Signed-off-by: mosheabr <257371078+mosheabr@users.noreply.github.com>
Addresses OSS review (Bernd Weber):
- LICENSE: replace CC-BY-4.0 deed with full legal text (all 8 sections); fix copyright entity to NVIDIA CORPORATION & AFFILIATES.
- CONTRIBUTING.md: reproduce full DCO v1.1 text (a-d) in Signing Your Work.
- Add SPDX-License-Identifier headers to 5 catalog-owned scripts (.github/scripts/).
- Fix entity (NVIDIA Corporation -> NVIDIA CORPORATION & AFFILIATES.) in verify_content_integrity.py and generate-skill-metadata.py.
Note: the remaining 14 flagged files live under skills/ (synced from product source repos) and must be fixed at source; routed to owning teams separately.
Signed-off-by: mosheabr <257371078+mosheabr@users.noreply.github.com>
Moshe Abramovitch