npm run eval was failing with a 404 not_found_error for anyone not
overriding EVAL_MODEL/EVAL_JUDGE_MODEL, since the hardcoded default model
ID is no longer served by the API. Default to claude-sonnet-5 instead.
* Add eval runner script and update contributing guide
Adds scripts/run-evals.js — runs evals for any skill with and without the
SKILL.md as system context, grades each expectation via Claude, and reports
pass rates and delta.
Usage:
ANTHROPIC_API_KEY=... npm run eval <skill-name>
Also updates CONTRIBUTING.md to:
- Replace the skill-creator eval command (not yet implemented) with the
actual npm run eval command
- Clarify the difference between knowledge evals and tool-execution evals,
and how to interpret results from each
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix Prettier formatting in run-evals.js
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove skills-ref dependency, replace with enhanced self-owned validator
- Add validation for name format, description length, SKILL.md line count,
evals/evals.json structure, and bidirectional references/ checks
- Remove all agentskills.io and external spec references from docs
- Frame references/ progressive disclosure as our own convention
- Replace skill-creator CLI references with npm run eval
- Add missing evals for mapbox-search-patterns
- Update skill structure with references/, assets/, scripts/ directories
and progressive loading model per https://agentskills.io/specification
- Add references/ convention: when to split, rules for file organization
- Update SKILL.md format with full field constraints and optional fields
- Strengthen content guidelines with Mapbox-specific focus
- Replace validate-skills.js with official skills-ref validator
- Add skills-ref as devDependency
The setup-hooks.sh bash script is redundant since we have the cross-platform
Node.js version (setup-hooks.js) that works on Windows, macOS, and Linux.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Replaced bash script with Node.js version using native fs/path modules
- Updated package.json to use 'node scripts/setup-hooks.js'
- Works on Windows (PowerShell/cmd), macOS, and Linux
- Handles chmod gracefully on Windows where it's not needed
Set up automatic git hooks that run all CI validation checks before pushing code. This prevents common CI failures by catching issues early in the development workflow.
Changes:
- Add .githooks/pre-push that runs format, spell, lint, and validation checks
- Add scripts/setup-hooks.sh to install hooks into .git/hooks/
- Update package.json with setup and postinstall scripts for automatic installation
- Document git hooks setup in CONTRIBUTING.md with usage instructions
The pre-push hook runs the same checks as CI:
- Prettier formatting validation
- cspell spell checking
- markdownlint markdown linting
- Skills structure validation
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Add prettier as dev dependency
- Create .prettierrc config with markdown-specific settings
- Add .prettierignore to exclude node_modules and generated files
- Add npm scripts:
- format: Format all files with prettier
- format:check: Check formatting without changes
- Update npm check script to include format:check
- Add format check step to GitHub Actions CI
- Format all existing files with prettier
Prettier config:
- Preserve line wrapping in markdown (proseWrap: preserve)
- 120 char print width for markdown
- Consistent formatting for JSON and JS files
This ensures all markdown, JSON, and JS files maintain consistent
formatting across the repository.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Matthew Podwysocki