Normalize every SKILL.md description to double-quoted YAML so the corpus,
templates, and examples have one visible convention. Harden frontmatter
parsing against partial fence matches and non-string name/description values,
and extend tests to cover those regressions.
Fix stale or misleading developer-facing docs: remove phantom book-sft plugin
install instructions, document directory-copy installs, add .agents/skills
manual install guidance, fix the digital-brain setup script target, and mark
historical 13-skill analysis as historical. Cut the prepared release metadata
to 2.3.1 across manifests and root SKILL.md.
Tighten production gates by sharing dev dependencies, making orchestrators use
strict repo validation, aligning skill-health required sections with repo
validation, and linting documented plugin install commands against the actual
marketplace manifest.
Co-authored-by: Muratcan Koylan <muratcankoylan@users.noreply.github.com>
* fix: YAML-safe skill frontmatter for cross-platform discovery
Eleven skills used unquoted descriptions containing colons, which strict
YAML parsers reject during skill discovery on Cursor, Claude Code, and
Codex. Normalize all skill frontmatter to valid YAML, add a shared
skill_frontmatter parser with PyYAML validation, fix install docs to use
directory layout, and add .agents/skills symlink plus Open Plugins
skills path declaration.
Co-authored-by: Muratcan Koylan <muratcankoylan@users.noreply.github.com>
* test: add frontmatter parser unit tests and harden examples
Add researcher/scripts/tests/test_skill_frontmatter.py (19 tests) covering
parser edge cases, a strict-YAML regression guard for the unquoted-colon
bug, format/parse round-trips, and corpus + example integration checks.
Wire unit tests into CI ahead of the strict repo gate. Quote the
description fields in the two example skills that had the same
unquoted-colon YAML hazard. Simplify CRLF handling in split_frontmatter
and document CONTRIBUTING test commands.
Co-authored-by: Muratcan Koylan <muratcankoylan@users.noreply.github.com>
* chore: harden production plugin packaging validation
Remove the committed .agents/skills symlink because symlinks are fragile
on Windows checkouts and in plugin packaging. Keep Cursor and Codex
support through Open Plugins discovery and documented directory-copy
manual installs.
Extend validate_repo.py so strict validation checks that the Open
Plugins manifest and Claude marketplace manifest name the same bundled
plugin, and that Open Plugins skills discovery resolves to the same 15
published skills as the repository.
Co-authored-by: Muratcan Koylan <muratcankoylan@users.noreply.github.com>
* ci: certify cross-platform skill compatibility
Add validate_platform_compat.py to check the production surfaces defined by
Agent Skills, Cursor, Claude Code, Codex, and Open Plugins documentation.
The gate validates published skills, manifest discovery parity, and
platform-local directory-copy installs for .cursor/skills, .claude/skills,
.codex/skills, and .agents/skills.
Install skills-ref in CI and require the upstream agentskills reference
validator in the compatibility gate. Tighten manifest path checks so both
Claude marketplace and Open Plugins skill paths must start with './', and
update operator docs to run the new gate before claiming production
readiness.
Co-authored-by: Muratcan Koylan <muratcankoylan@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Muratcan Koylan <muratcankoylan@users.noreply.github.com>
Apply the auto-research quality standard across every published skill and update the mechanisms, claims, fixtures, benchmarks, and release docs so the corpus improves as a coherent system rather than isolated prose edits.
Co-authored-by: Cursor <cursoragent@cursor.com>
The v2.3.0 description rewrites (commit f917ce8) changed the one-line
frontmatter `description` for three skills. The router benchmark
validated those changes. The mistake was assuming the work was done:
the SKILL.md body for each skill still claimed the broader pre-rewrite
scope in its `When to Activate` and `Practical Guidance` sections.
The router benchmark could not catch this because router prompts run
with `settingSources: []` and only see the frontmatter description. The
gap only matters at activation time (when the agent actually loads the
SKILL.md body), and only Stage 3 effectiveness benchmarks would measure
it directly.
Aligned all three bodies with their new descriptions:
context-fundamentals (2.0.0 -> 2.1.0):
- `When to Activate` rewritten to list conceptual triggers and an
explicit do-not-activate routing map.
- Removed `Practical Guidance` sections `File-System-Based Access`
(owned by filesystem-context) and `Context Budgeting` (owned by
context-optimization).
- Replaced with conceptual application advice and a reading-order
recommendation for new contributors.
- `Integration` rewritten as a routing map across all 14 sibling
skills with explicit ownership statements.
tool-design (2.0.0 -> 2.1.0):
- `When to Activate` anchors on the unit of work (single tool or tool
catalog) and lists adjacent decisions owned by project-development,
multi-agent-patterns, context-optimization.
- `Integration` rewritten with explicit routing reasons per sibling.
project-development (1.1.0 -> 1.2.0):
- `When to Activate` anchors on project shape and pipeline; lists
adjacent decisions owned by tool-design, context-optimization,
multi-agent-patterns, harness-engineering.
- `Integration` rewritten with explicit routing reasons per sibling.
Honest documentation updates:
- CHANGELOG.md v2.3.0 section gains a "Skill body alignment" subsection
explaining the two-step process (description fix then body alignment).
- researcher/insights/auto-research-experiment.md adds finding #15: a
skill is a multi-surface artifact; description rewrites without body
audits are incomplete; the router benchmark cannot catch the gap.
- AGENTS.md captures the body-audit-same-day rule as a learned default
for future agents working on this corpus.
Gates green:
- validate_repo --strict: 0 errors, 0 warnings, 15 skills
- run_benchmarks: 3 checks, 0 failures, 7 scenarios
- check_activation_cases: 14 cases, 0 failures
- skill_health corpus: 0.811 (was 0.814; small drop from removing one
operational code example, acceptable trade for cleaner scope)
Co-authored-by: Cursor <cursoragent@cursor.com>
Promotes the release from v2.2.0 (scaffolding only) to v2.3.0 (measured
end-to-end results across four frontier models with delta-vs-baseline
attribution). Version bumped in marketplace.json, plugin.json, and root
SKILL.md.
Updates:
- CHANGELOG.md: new v2.3.0 entry with full per-model and per-skill effect
sizes, methodology notes, and honest scope caveats. v2.2.0 section
preserved as the scaffold release.
- researcher/benchmarks/PLAN.md: status table reflects Stage 0/1/2
shipped; Stage 3/4 still scaffolded. New "Shipped Stage 2 results"
subsection links to baseline and post-fix published reports.
- researcher/insights/auto-research-experiment.md: two new findings.
#14 documents the description-benchmark loop closing in ~2 hours with
+23.4pp on context-fundamentals. #15 documents concurrency + resume +
per-run logging as non-negotiable runner features. Reusable-patterns
list grows from 10 to 12.
- researcher/insights/how-we-built-this.md: new "Stage 2: The Description
-Benchmark Loop Closes" chapter. By-the-numbers updated: 14 activation
cases, 50 router prompts, 1200 measured agent calls across two sweeps,
4 models evaluated.
- README.md: researcher OS section gains a measured-results subsection
with per-skill effect-size table and per-model leaderboard.
- AGENTS.md: workspace memory captures the description-benchmark loop as
a learned default, the three runner non-negotiables (concurrency,
resume, per-run logging), and the API-key exposure rotation policy.
All gates green: validate_repo --strict 0 errors, run_benchmarks 0
failures, check_activation_cases 14 cases 0 failures, lints clean.
Co-authored-by: Cursor <cursoragent@cursor.com>
Real numbers from the first end-to-end router benchmark run against the
Cursor SDK. All four frontier models (composer-2, claude-opus-4-7,
gpt-5.5, gemini-3.1-pro) evaluated against the 50 ground-truth prompts
at 3 replications each. Process exited at 566 of 600 runs; per-model
coverage stayed balanced at ~141 runs each (statistically more than
sufficient).
Headline findings (full report in
researcher/benchmarks/router/results-published/2026-05-15.md):
- All four models cluster within 0.3 percentage points on top-1
accuracy (0.886 to 0.888). 95% CIs overlap completely. Model choice
is not the deciding factor at the router stage.
- One skill carries almost all of the routing failure:
context-fundamentals (12/47 correct). The rest split across
context-degradation, project-development, context-optimization,
evaluation. Rewriting this description is the highest-leverage
follow-up action.
- tool-design vs project-development is a real boundary problem
(~25% symmetric leakage). Mild but consistent.
- evaluation vs advanced-evaluation is essentially solved (~6%
leakage); the v2.2.0 boundary hardening worked.
- Format compliance is 99.8% across all models. Strict-JSON routing
prompts are reliable.
- Latency varies ~3x: Claude 3.4s, GPT-5.5 3.8s, Composer-2 4.0s,
Gemini 3.1 Pro 9.1s median.
- Negative-control prompts (where no skill should fit) behave
correctly: predictions split across multiple skills with low
per-skill rate.
Also lands:
- researcher/scripts/render_router_report.py: deterministic report
generator with per-model leaderboard, bootstrap 95% CIs, per-skill
confusion matrix, hardest-prompt drill-down, and a reproduction
command.
- researcher/benchmarks/router/results-published/README.md: explains
the committed-summary vs gitignored-raw split.
- runRouter.ts: summary.json now includes run metadata (timestamp,
repo_sha, fixture_sha, seed, models, reps) so future runs render
reports without manual metadata injection.
- AGENTS.md: workspace memory refresh from this turn's transcript.
Next action committed in the report's executive summary: rewrite the
context-fundamentals activation description and rerun to measure the
delta. Expected improvement is roughly +5-7 percentage points across
all four models.
Co-authored-by: Cursor <cursoragent@cursor.com>
Ships a file-based research-to-skill operating system on top of the v2.1
skill corpus. The corpus now compounds across runs: mechanisms have a
gated promotion path, claims have explicit provenance, runs have an
enforceable state machine, skill activation is regression-tested, and the
harness is benchmarked against adversarial scenarios. A continuous loop
runs the whole thing unattended from launchd, retrieving sources over
stdlib HTTP and parking anything that needs human review.
Highlights
- Mechanism registry + accepted/rejected ledgers (5 seeded mechanisms)
- Claim provenance for 6 volatile claims across 6 skills
- Corpus index mapping skills to scenarios, mechanisms, and claims
- Activation regression fixtures (8 cases, all passing)
- Adversarial benchmark harness (7 scenarios) with append-only history
- Run state machine: initialized -> retrieved -> evaluated -> proposed ->
novelty_checked -> validated -> pr_ready -> closed
- Continuous loop (loop_discover, loop_step, loop_daily, loop_status)
with atomic JSONL writes and fcntl locks
- launchd service definitions and operator runbook
- CI workflow runs validate_repo --strict, run_benchmarks, and
check_activation_cases on every PR
- AGENTS.md, CHANGELOG.md, researcher/runs/README.md,
researcher/insights/auto-research-experiment.md
Skill changes
- All 14 published skills migrated from keyword triggers to
activation-scenario descriptions
- Hardened evaluation/advanced-evaluation and project-development/
harness-engineering boundaries
Repository policy
- Active research runs (researcher/runs/*/) are runtime state and
gitignored. Seed run kept as a worked example, closed reference-only.
- Runtime queue, logs, snapshots, status, and loop event files are
gitignored.
Verification
- validate_repo --strict: 0 errors, 0 warnings, 14 skills
- run_benchmarks: 3 checks, 0 failures, 7 scenarios
- check_activation_cases: 8 cases, 0 failures
- Python compile: clean (12 researcher scripts)
- Lint: clean
- Live loop: 13 iterations executed across init/retrieve/park/reap paths
No paid LLM judges, no automated source discovery beyond the manual seed,
and no log rotation. See researcher/insights/auto-research-experiment.md
for the full lessons-learned writeup.
Co-authored-by: Cursor <cursoragent@cursor.com>
muratcankoylan