Cursor supports bundling multiple skills in a single plugin, so the
Cursor marketplace now publishes one redis-development plugin - the
same directory the Claude Code plugin uses - instead of eight
per-skill plugins. Requested by the Cursor marketplace team for
user discoverability.
A `with_skill` run passes `skills/<skill-name>/` to the model under test via
addDirs and tells it to use the skill at that path. `evals.json` carries
`expected_output` and the grader's `expectations`, so while it sat at
`skills/<skill-name>/evals/<suite>/evals.json` the run handed the model the
answer sheet for the question it was being asked, along with the committed
baseline scores. Whether any past run read it cannot be established: only the
final text, grading, and timing are persisted, not the tool calls.
Everything under `skills/` is also what the marketplaces publish, so nesting
suites and baselines there shipped them to every user installing via
`npx skills add`.
Move the suites to `evals/<skill-name>/<suite-name>/` and point the runner,
aggregator, baseline promoter, and baseline validator at the new root. All 56
files move unchanged.
Two workarounds go away with the nesting: skill-validator no longer needs
`--allow-dirs=evals` to accept a skill directory, and the plugin sync no longer
excludes `evals` (only `.cursor-plugin` remains, which Cursor reads from
`skills/`). The published plugin is byte-identical, so this does not trigger a
republish.
The Claude Code plugin directory pins our entry to the `plugins/redis-development`
subdirectory, and its nightly bot advances that pin only when the subtree itself
changes. Every skill lived only in top-level `skills/`, so each nightly run
logged "subtree unchanged; suppressing no-op bump" and nothing since 3d6f255
reached users. The same subdirectory fetch is also why the installed plugin
reports no skills or agents: it materializes only `plugins/redis-development`,
where the eight skill entries were symlinks pointing outside the fetched tree.
Vendor real copies instead of pointing the symlinks the other way. Claude Code
and Cursor both drop symlinks that escape a plugin root on install, per
neondatabase/agent-skills, which hit this and reached the same conclusion; real
files also survive a Windows checkout, where git writes symlinks as plain text
unless core.symlinks is on. `skills/` stays the source of truth and keeps the
paths the agentskills CLI and the Cursor marketplace read.
Vendor only what an agent loads at runtime. `evals/` and `.cursor-plugin/` stay
out, so eval suites and baselines are not duplicated and an eval-only change
leaves the published plugin untouched.
Keep the copies honest without hand work: the pre-commit hook regenerates and
stages them, and `validate:plugin-skills` runs inside `validate:plugins` in both
the hook and CI. It fails on drift, on a file left behind by a deleted skill, on
a symlink reappearing on either side, and on a `SKILL.md` that is not a regular
file. The hook refuses when `skills/` has unstaged changes, so the committed
source and its published copies always agree.
* fix(validate): fail structure validation when --all finds no skills
`--all` walking to zero skill directories printed "No skill directories found."
and exited 0, so a broken walk reads as a clean pass. The check then reports
success having validated nothing, which is worse than failing: CI goes green
over an empty result.
`--changed` legitimately resolves to nothing when a PR touches no skill, so it
keeps the exit-0 path.
* ci: run validation on every pull request, not only those targeting main
Restricting the pull_request trigger to branches: [main] means a PR stacked on
another branch gets no checks at all, which reads as passing rather than as
unverified.
A matrix without repetitions runs at the runner's default of 1 and promotes
normally, but the run-count check skipped when the field was absent, silently
exempting that suite from partial-run detection forever. The check now
resolves repetitions ?? 1 like the runner, and a present-but-invalid value is
an error instead of a skip, since the runner never validates it and garbage
would corrupt the expected counts on both sides.
Two more from Bugbot on #37.
Model names alone cannot tell a full run from a killed one that touched every
model. The aggregate records each configuration's run count per model, which a
complete run pins to repetitions x eval count, so checkCoverage now verifies
that. A partial workspace aggregated and promoted by hand previously produced
a baseline that passed validation with scores from a subset of the runs.
A baseline aggregate without a models array skipped coverage checking
silently. Nothing else ever reads the committed aggregate, so absence must
fail here; the live matrix keeps its skip because the runner validates it
loudly before any spend.
Two more from Bugbot on #37.
checkMatrix compares the baseline's copied matrix against the suite's, which
are the same file, so it cannot tell whether the run behind the baseline
actually covered those models. A run killed partway and then promoted leaves a
baseline whose scores come from a subset of the matrix. checkCoverage now
compares the models the aggregate scored against the models the matrix lists,
in both directions. Three runs were killed mid-flight while producing these
baselines, so this was reachable rather than theoretical.
listDirectories treated every readdir failure as an empty directory, so a
permissions or I/O error on skills/ was indistinguishable from a repo with no
skills and exited 0 having validated nothing. Only ENOENT now counts as empty;
anything else is reported, and discovery errors reach the failure report
instead of taking the empty-repo exit.
Five more from Bugbot on #37.
The validator now mirrors the runner's own suite validation: evals.json's
skill_name and eval_suite are required rather than only checked when present,
and a non-array evals field reads as a validation error instead of throwing a
TypeError out of .map and aborting the run.
It also rejects duplicate eval ids and duplicate matrix models or
configurations. The runner has no duplicate handling and derives output paths
from the eval id and model, so repeats silently overwrite each other's results
rather than failing.
Retries no longer discard billed cost. A failed attempt is frequently still
charged, but only the successful result was recorded, so any run that retried
understated its spend. Failures now carry their CommandResult, and the
recovered cost is added to the recorded total alongside an attempts count and
a discarded_retry_usd field.
The runner compares model-matrix.json's eval_suite to evals.json's and
resolves the skill directory from evals.json's skill_name, so a one-sided
edit passed validation while `npm run eval` refused the suite. A skill_name
naming another real skill is worse than an error: the runner would resolve
that skill's directory and produce results for the wrong skill.
checkIdentity now compares evals.json's skill_name and eval_suite, and
model-matrix.json's eval_suite, against the directory, alongside the baseline
fields it already checked. The directory is the ground truth, since that is
what suite discovery walks.
Two more from Bugbot on #37.
Discovering no suites exited 0, so a wrong working directory or removed
suite files reported success for having validated nothing. It now fails when
skills exist but no suite does, while a genuinely empty skills/ still passes.
Nothing compared the baseline's own skill and suite names against the
directory holding it, so another suite's promoted aggregate could sit in a
baselines/ directory and pass whenever the matrix and eval names aligned.
Three divergences from the runner, all found by Bugbot on #37:
The promote hint hardcoded --iteration iteration-1, but eval:baseline
resolves the iteration from the matrix default_iteration. Following the hint
after an iteration-2 run would re-promote the older results while the matrix
files still looked current, which is the stale baseline this check exists to
catch. The hint now omits the flag.
Suite discovery accepted any directory holding evals.json, while the runner
requires model-matrix.json alongside it, so a half-scaffolded directory was
asked for a baseline it could not yet produce.
Eval keys were built from raw names, but the aggregate trims them and falls
back to eval-<id> when blank, so a padded name would fail validation straight
after a valid promote.
The extra script and --skip passthrough existed to keep link checking out of
the pre-commit hook, which the hook can do by simply not running
skill-structure validation. Link checks only cost 3s, so speed was never the
reason; the reason is that a link failing on a flaky network fails an
otherwise good commit, and false positives train people into --no-verify.
The hook now runs the plugin-manifest and eval-baseline checks, which are
fast and deterministic. CI still runs the full validate, links included.
`npm run validate` was red on a fresh clone. It shells out to
skill-validator, a Go binary that is not an npm dependency and was
documented nowhere, and a missing binary was a hard failure, so the
pre-commit hook blocked every commit. It now prints the install command
and skips locally, while still failing in CI where a silent skip would let
unvalidated skills merge. CI reads the version from package.json so the
documented command cannot drift from the installed one.
Link checking made committing depend on network access: the validator
makes a live request per external link, and under --enforce an unreachable
docs host became 26 errors. Adds a --skip passthrough and a
validate:offline script for the pre-commit hook; CI still checks links.
Adds validate-eval-baselines, which requires every eval suite to carry a
baseline that still matches its evals and model matrix. Nothing previously
caught drift, so a report kept comparing against a stale baseline instead
of failing, which is how redis-core's 4.x baseline survived. Reordering the
models list is not a change and does not trip it.
* chore(validate): allow evals/ directory in spec-format skills
* feat(redis-core): add spec-compliant skill for data modeling
* feat(redis-core): move eval suite from redis-development
* chore(eval): show generation cost only in skill-vs-baseline comparison
* docs: note agentskills.io migration in root AGENTS.md
Address two issues flagged in PR review:
1. Validate pluginRoot and source with isSafeRelativePath before using
them in path.resolve. Without this, absolute paths or .. traversals
in marketplace.json would pass validation silently.
2. Error when the per-plugin .cursor-plugin/plugin.json is missing
instead of silently skipping it. A missing manifest would prevent
the plugin from loading, so the validator should catch it.
Made-with: Cursor
The pre-commit validator previously required a root plugin.json and
would fail when only marketplace.json was present.
Refactor the validator to check for marketplace.json first, falling
back to root plugin.json for single-plugin repos. For marketplace
repos, validate the marketplace manifest (name, owner, plugins array),
resolve each plugin directory using pluginRoot + source, and validate
per-plugin manifests with paths relative to the plugin directory.
Made-with: Cursor
Vasko Atanasov