mirror of
https://github.com/CopilotKit/CopilotKit.git
synced 2026-09-14 16:26:20 +08:00
Ben Taylor
450e35be8c
## What
Tests the six remaining skills on the axis that is cheap to check and
actually broken: **are the facts they state still true?**
The existing guard covers the two entry points, which point at
documentation, by resolving every docs path they name. The Inspector and
Intelligence skills point at the *repository* — an Nx target, a
dev-server port, a lab scenario id, a landing-page source file, a
Callout snippet — and nothing checked any of it.
## It was already wrong
`465abb0239` (OSS-948) deleted `open-inspector-step-angular.mdx`, and
said so explicitly:
> the Angular Open Inspector snippet — which asserted "Angular does not
mount Inspector by default" — is deleted in favour of the shared one
every other web frontend already uses
`@copilotkit/angular@0.4.0` auto-mounts a pinned
`@copilotkit/web-inspector`, so there is no install step left to link.
**That commit never touched `pane-map.md`.** So the map kept pointing at
a deleted file, and `inspector-docs` Procedure 3 kept telling agents
"Angular uses the Angular step snippet, which links the Angular
Inspector install page first."
A change to exactly the thing the skill tracks left the skill's own
source of truth stale, silently, for two weeks. Both are corrected here.
## The four assertions
Each has an unambiguous ground truth:
| assertion | ground truth |
| --- | --- |
| every `open-inspector-*.mdx` the pane map names exists |
`snippets/shared/inspector/` |
| every `docs/…mdx` page the pane map names exists | the content tree |
| every `showcase/shell-docs/src/…` file `intelligence-docs` says to
edit exists | the repo |
| the Nx target, port, scenario ids and query keys `inspector-workbench`
names resolve | `project.json`, `threads-state-lab.ts` |
## What is deliberately not asserted
**Which panes the Inspector ships.** Panes are not enumerated as data
anywhere in `packages/web-inspector`, whose entry point is a single
~14,700-line module, so matching a pane label against source proves
nothing in either direction.
A first draft of this suite tried it and **passed while the map really
was wrong**: `pane-map.md` lists "Pop-out window" under Unshipped, but
`src/lib/pop-out.ts` is imported by the package entry, carries live
`popOut` state in `index.ts`, and has its own test suite. My slug match
looked for `pop-out-window`, found nothing, and went green. That is
worse than not testing it, because a green run reads as confirmation.
So that check is gone, the reasoning is in a comment in the suite, and
**the Pop-out row is left as-is for the Inspector owner to rule on**
rather than guessed at. Making that direction testable needs a pane
registry in the package, not a cleverer regex here.
## Testing
```
$ vitest run scripts/__tests__/public-skill-drift.test.ts scripts/__tests__/sync-plugin-skills.test.ts
Test Files 2 passed (2)
Tests 20 passed (20)
```
`pnpm check:plugin-skills` → `plugin skill mirror in sync`. `pnpm
check:intelligence-env-names` → `Intelligence env var names and hosts
are canonical`. `oxfmt --check` clean (it reported two files needing
format; ran `--write`, re-checked clean, re-ran tests).
**Mutation-checked, since a check that has never failed is not a
check.** Each mutation produces exactly one failing test, and the
restored tree is clean:
| mutation | failures |
| --- | ---: |
| baseline | 0 |
| `docs/meta.json` → `docs/NOPE.json` in `intelligence-docs` | 1 |
| port `5177` → `9999` in `inspector-workbench` | 1 |
| `scenario=pro-enabled-existing` → `no-such-scenario` | 1 |
| nx target `dev:standalone` → `dev:gone` | 1 |
| `docs/inspector.mdx` → `docs/no-such-page.mdx` in the pane map | 1 |
| restored | 0 |
The Angular finding is the suite's own first catch: before the fix it
failed with exactly `[ 'open-inspector-step-angular.mdx' ]` and no false
positives. An earlier iteration also flagged `inspector.mdx` and
`react-native.mdx` — page references from a different table — which is
why the snippet pattern is anchored on the `open-inspector-` prefix.
This runs in `plugin-skills-check.yml`, which already triggers on
`skills/**`.
## Scope
This is the cheap half of testing a skill — whether it is still true.
Whether a skill measurably *helps* an agent is the other half, and that
needs the lift eval in #5689, which is parked.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Documentation**
- Updated Angular Inspector quickstart guidance to use the shared web
Inspector steps.
- Clarified that Angular automatically mounts the pinned web Inspector,
so no separate installation step is required.
- Updated the Inspector pane map to reflect the shared guidance and
revised installation notes.
- **Tests**
- Added checks to keep procedure documentation aligned with current
Inspector, Intelligence, and Workbench configuration details.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->