6 Commits

Author	SHA1	Message	Date
Samuel Moelius	09dfbd9153	Drop the "When to Use" / "When NOT to Use" requirement (#216) ... * Drop the "When to Use" / "When NOT to Use" requirement * Fix inconsistency in AGENTS.md * Eliminate `REQUIRED_SKILL_SECTIONS` check in validate_plugin_metadata.py	2026-07-30 11:12:07 -04:00
Dan Guido	ca08fc8a91	Commit plugin lockfiles; unblock Dependabot (#213) ... * Commit plugin lockfiles so Dependabot can do something useful The uv ecosystem config added in #206 pointed at four directories that declare PEP 621 ranges and carry no lockfile. With nothing to pin, Dependabot's only available action is raising the lower bound of an already-open range — which changes nothing about what installs and only drops support for older versions. It opened five such PRs within a minute of #206 merging (#208-#212), all no-ops: the existing ranges already resolved to exactly the versions being proposed as new floors. The one directory that did have a lockfile, constant-time-analysis, produced no PR at all, because there was genuinely nothing to update. That is the whole diagnosis. Lockfiles committed for the other four. .gitignore ignored uv.lock globally, which is why they were missing; constant-time-analysis's was tracked only because it predates the rule. Now scoped to the root file (ephemeral — there is no root pyproject.toml) with plugin lockfiles explicitly allowed, matching the pattern already used for .mcp.json. Also fixes two bugs #206 introduced: - The version-increment check failed all five Dependabot PRs, and Dependabot can neither bump a plugin version nor label its own PR, so every future dependency PR would have been permanently red. Exempted by actor. - The 'no-version-bump' label was documented in AGENTS.md and wired into validate.yml but never created, so the escape hatch did not exist. Created. * Re-run CI with the no-version-bump label applied The version-increment check fired on this PR: adding uv.lock under plugins/<name>/ counts as touching those plugins. Correct behaviour — the lockfiles pin exactly what the existing ranges already resolve to, so nothing changes for anyone installing these plugins, which is what the label is for. First real use of the escape hatch created in this same PR. * Fix the three findings from this PR's review A local uv setting leaked into all four new lockfiles. /etc/uv/uv.toml on ToB machine images sets exclude-newer = "1 week", so every lock carried an [options] block with exclude-newer-span = "P1W" and pinned versions resolved a week stale — diverging from constant-time-analysis/uv.lock, which predates this PR and has no such block. Regenerated with UV_NO_CONFIG=1. That cooldown is the org's supply-chain posture and it belongs in dependabot.yml's 'cooldown: default-days: 7', where it already is; baking it into committed lockfiles was my environment leaking, not a decision. "EVERY directory here must carry a committed uv.lock" was enforced by a comment, which is precisely the anti-pattern AGENTS.md tells people to avoid. Now a validator check: it parses the uv ecosystem block out of dependabot.yml and asserts a uv.lock beside each listed directory. Scoped to that block rather than grepping for '- /plugins/...' so a future ecosystem's paths are not swept in, and it errors if the block exists but no directories parse out — otherwise the checker could inspect zero items and report clean, which is the exact failure it exists to prevent. Three self-test fixtures, and verified by deleting a real lockfile and confirming CI would go red. The Dependabot exemption keyed on github.actor, which on a synchronize event is whoever pushed. A human adding one commit to a Dependabot branch would re-arm the version check and turn the PR red — making the follow-up bump mandatory exactly where the comment says it is discretionary. Keyed on PR authorship now.	2026-07-29 20:09:36 -04:00
Dan Guido	8ea3b6a700	Move the contribution checklist into machinery (#206) ... * Add validator self-test, structural checks, and make check The repo documented ~53 rules in AGENTS.md and machine-enforced 6 of them. This closes the gap for the ones a machine can decide, and adds the guard that keeps the checkers honest. New error-level checks (all currently pass, so none of this blocks anyone today): agent files must use `tools:` while skills use `allowed-tools:` (the loader silently ignores the wrong key, so the restriction just does not apply); subagent_type must be namespaced or the dispatch fails at runtime; plugin dir names kebab-case and <=64 chars; plugin README present, listed rather than stat'd so `Readme.md` fails on Linux CI the way it should; semver format; the forbidden runtime sidecars AGENTS.md already banned but nothing checked; and version-increment against the base branch, which is the gap that let debfb29 ship an allowed-tools fix across 25 plugins that no installed user received. New warning-level checks, reported but not blocking: the two required SKILL.md sections, the 500-line limit, and unresolved relative references. 55 warnings across 40 plugins today, concentrated in testing-handbook-skills and building-secure-contracts. The point of the exercise is the three anti-vacuity guards. A checker that has silently stopped matching reports a clean repo forever, and that failure mode has shipped repeatedly: `--self-test` builds fixtures and asserts every checker rejects a known-bad one; a full scan that resolves zero references exits 1 rather than declaring everything clean; and SELF_TEST_MINIMUM fails the self-test if it runs fewer than 20 assertions, because the self-test is itself a checker. It currently runs 26. The reference extractor skips fenced and inline code so that skill-authoring docs citing example paths do not generate warnings nobody reads. Makefile mirrors CI as one `make check`. Its RUFF_VERSION must match the ruff-pre-commit rev, and the self-test asserts that — verified by breaking it. * Fix CI checks that could pass or fail without inspecting anything Four of these are the same bug in different places: a check whose empty case is indistinguishable from success. - python-tests ran `python3 <file>` per file. A test file with no `if __name__ == "__main__"` block exits 0 having run nothing, which reads as a pass. All 10 current files happen to comply; nothing enforced it. Now pytest per directory, run via `python3 -m` from inside each one to preserve the sibling imports these suites rely on, with --import-mode=importlib because c-review and rust-review both ship scripts/test_split.py and the default import mode collides on the basename. 278 tests now run where the count was previously unknown. - bats used --no-run-if-empty, so a broken glob was a silent pass. This repo ships bats suites; finding none is a failure. - The SKILL.md frontmatter walk printed "All 0 SKILL.md files have valid frontmatter" if discovery broke. Now fails on zero. - zeroize-audit's shell regression suites matched no CI glob and had never run. Also: the validator self-test runs first, before validation. The hardcoded-path grep now covers .sh, .bats, .yml and .toml — test fixtures and install scripts are where absolute paths hide. The personal-email exclusion is anchored; an unanchored '.git' also dropped any line containing '/github'. The two npm CLI installs are pinned rather than @latest, which zizmor flagged and which made CI able to break with no commit. check_codex_loadability.py: tempdir cleanup raced the codex app-server and threw `OSError: [Errno 39] Directory not empty: '.git'`, failing the job after every loadability check had passed (seen on PR #148 today). A teardown race must not be reported as a validation failure. * Wire validators into pre-commit; fix two dead Dependabot entries pre-commit: the three .github/scripts validators were CI-only, so the first signal a contributor got was a red check after pushing — AGENTS.md asked people to remember to run them by hand. Now they run locally, with the validator self-test scoped to fire only when the validator itself changes. Adds actionlint, zizmor, check-toml, check-merge-conflict, and detect-private-key. detect-private-key earns its place: there is no secret scanning here at all, which is how a live API key sat in an untracked config file in a working tree this morning without anything noticing. Markdown linting is deliberately absent, with the measurement recorded in the config so the next person does not have to redo it: markdownlint reports ~12,400 violations across this repo (7,282 MD013 line-length alone). It would land either permanently red or with so many rules disabled that it checks nothing. Dependabot had two defects that made it quieter than it looked. The `pip` entry at / had no manifest to resolve — there is no root pyproject.toml and uv.lock is gitignored — so it reported nothing, indefinitely. And two script directories with real dependencies were uncovered: trailmark's slicing-code-context (trailmark>=0.5,<0.6) and yara-authoring's rule scripts (yara-x>=0.10.0). Switched to the uv ecosystem per the house standard, grouped minor/patch so majors still get their own CI run. * Add automated PR review, inert until a key is configured Two tiers: fast (effort low, every push, sticky comment) and deep (effort xhigh, on a deep-review label). Per-job permissions with an empty workflow-level grant; concurrency keyed per tier so a routine push cannot cancel an in-flight deep review — GitHub will not re-fire `labeled` for a label already present, so that would leave a PR sitting labeled with no review, looking reviewed. The prompt is the substance. It forbids pre-filtering: current models follow "only report high-severity issues" literally, investigating fully and then declining to report what they judge below the bar, which reads as a capability regression but is a prompt bug. It asks for everything ranked P1-P4 with a concrete failure scenario each, and filters downstream. It also names the five defect classes that actually reach main in a repo of markdown that instructs a model, rather than asking for generic code review. There is no ANTHROPIC_API_KEY secret on this repo, so every review step is gated on the secret being present. Until someone adds it these workflows check out the code and do nothing — they do not fail. A review workflow that goes red on every PR for want of a credential teaches people to ignore red checks. Fork review is a separate file with a separate decision attached. It needs pull_request_target, because under `pull_request` a fork PR gets a read-only token and the job could not post its comment at all. The usual exploit path is closed by a maintainer-only label gate, a checkout pinned to the SHA as of the labeling event (so a later force-push does not change what is reviewed), and a tool allowlist with no general Bash, so fork code is read and never executed. The residual prompt-injection risk is documented in the file header. Deleting that one file costs nothing but fork review. * Replace the PR checklist with make check Deletes the 20-item PR checklist. Current model guidance is explicit that verification scaffolding of that shape degrades output rather than improving it, and that removing it costs no capability — so the response is to move each check into machinery, not to restate it louder. Every item that could be mechanized landed in the validator first, with a fixture, before this prose was allowed to go. What replaces the checklist is two lists: what the validator enforces so you do not have to, and what no tool can check. The second list is the one that matters — whether the description actually triggers, whether the version bump is the right size, whether the README row is in the right section (the validator only checks the plugin appears somewhere in that file). Also corrects the frontmatter example, which showed `allowed-tools` as a YAML list. Every SKILL.md in the repo uses the space-delimited form; debfb29 converted them deliberately. The doc was the outlier, and it caused a false finding during review of #192. Agent files genuinely do use a YAML list, under the `tools` key — both forms are now shown side by side with the reason the distinction matters. Two new sections. "Scripts a plugin ships" leads with the rule this whole change is built around: a checker that inspects zero items must fail, not pass, with the three worked examples. "Working effectively in this repo" covers effort sweeps, a subagent cap (current models over-delegate, which is a reversal from the previous generation), not adding verification scaffolding to prompts, and not telling a reviewer to pre-filter — that last one reads as a capability regression and is a prompt bug. Scopes the claude-code-guide line to questions that cannot be answered by reading this repo, and adds .opencode/ to the banned sidecar list now that the validator enforces it. * Fix three bugs CI caught in the guardrails themselves All three were mine, and the first is the one that mattered. 1. The version-increment check ran against every plugin, not just the ones the branch touched — so it demanded a version bump from all 42 plugins on a PR that changed no plugin at all. Now scoped to plugins with file changes between the base ref and HEAD. Added three self-test assertions covering it (touched plugin errors, untouched plugin does not), and verified by reverting the fix and confirming the self-test goes red. The check had no coverage before, which is exactly why it shipped broken. 2. The new pre-commit hooks invoked `uv run`, which is not installed in the Pre-commit CI job. The validator declares no dependencies, so plain python3 is correct and one less thing to install. 3. The review workflows called `claude` without installing it. Added a pinned install step, matching how validate.yml installs the same CLI. Also corrects a factual claim in both review workflow headers. I wrote that they were inert for want of an ANTHROPIC_API_KEY, based on `gh secret list` returning empty — but that only lists repository secrets. An organization-level key is visible here, so the fast tier is live on merge. The headers now say so. The deep and fork tiers still cannot fire until their labels are created. * Fix the findings from this PR's own automated review The review posted on #206 found eight issues. Seven were real. Working through them in severity order: P1 — deleted claude-review-fork.yml. It checked out the fork tree and then ran this repo's review script *from that tree*, so the script itself, and any CLAUDE.md or .claude/hooks sitting beside it, was fork-authored and executed with the org API key in the environment. My header claimed "fork code is read, never executed" and that was simply wrong; the SHA pin and the tool allowlist do not touch that path. Doing it safely means checking out base into the workspace and the fork commit into a subdirectory, which is deliberate work rather than a footnote to this change. Fork PRs get no automated review for now, and claude-review.yml explains why. P2 — the prompts instructed `gh pr comment --body-file <file>` while the tool allowlist has no Write and no general Bash, so there was nowhere to put a file. The reviewer hit this and fell back to `--body-file -`; since a give-up does not fail the step, the failure mode was a green check with no review. Now uses a heredoc on stdin. The deep tier also told the model to run the scripts a PR adds, which its allowlist cannot do; it now says so explicitly. While fixing that: both tier heredocs were unquoted so `${PR_NUMBER}` would interpolate, which means backticks in the prompt body were command substitution. Quoted them and moved substitution out to parameter expansion afterwards. shellcheck caught this. P2 — `no-version-bump` was a phantom feature: named in the validator's error message and in AGENTS.md, read by nothing. Implemented via --allow-no-bump, wired to the label through validate.yml. P2 — AGENTS.md claimed both loadability checks run in `make check`. They do not and cannot; they need two CLIs installed. It now says what runs locally, what is CI-only, and how to run the loadability checks by hand. P3 — the new shell-suites discovery reintroduced the exact bug this PR exists to remove: it printed "No shell regression suites found" and exited 0, and used `**` without globstar so it only ever matched one directory level. Now find-based and fails on empty. It is a Makefile target but not part of `make check`, because zeroize-audit's suite pipes to `python3 -` which the modern-python shim rejects — filed as #207. P3 — a failed `git diff` returned an empty changed-plugin set, silently disarming the version check for every plugin. Now raises. P3 — the `/home/user/` exclusion was a content filter applied across all of plugins/, so a skill legitimately documenting that path would be dropped from the results. Scoped to the shim bats fixtures that need it. Not fixed, deliberately: the reviewer noted `_check_ruff_parity` covers one pinned version. True, and it is the only version both files pin. * Fix the second review pass: the review job could not detect its own silence The marquee one: claude_review.sh ended with `claude --print` and nothing else, so if the model finished without calling `gh pr comment` — a denied tool, a hit timeout, or it simply summarising instead of posting — the script exited 0 and the job went green with no review attached. That is defect class 1 from the prompt this very script ships, in the script that ships it. It now timestamps before the run and fails if no comment was created or updated since. _check_ruff_parity returned None (= pass) when the Makefile or pre-commit config was missing, while still counting toward SELF_TEST_MINIMUM. A vacuous pass inside the anti-vacuity harness. Now returns an error string. The scoping claim in AGENTS.md and the Makefile was backwards. I wrote that CI scopes the validator to changed plugins while local scans everything, making local a strict superset. Both scan everything; only the version-increment check is scoped, and it is the one check CI runs that local cannot — so local is not a superset at all. Corrected in both places. README said `make check` "runs everything CI runs" and the Makefile echoed "this is what CI will run". Neither is true: it omits both loadability checks, the version-increment check, and every pre-commit hook except ruff/shellcheck/shfmt. Both now say what they actually cover. AGENTS.md already had this right. Stale count in a pre-commit comment ("these three" preceding two hooks). Accepted without change, with reasoning: the reference resolver matches any file in the plugin, so a link to a file that exists under a different skill resolves when it should not — pinning a base directory produces a flood of false positives, and the looser check is still worth having. The subagent-dispatch check only knows its own plugin's agents, so a bare cross-plugin reference is missed; catching that needs a repo-wide agent map, which is a larger change than belongs here. * Fix the third review pass, including two more silent-pass checks The hardcoded-path step had the defect this PR exists to remove. It was written as `if grep A | grep -v B | grep -v C`, so the exit status came from the last `grep -v`. A first-stage failure — plugins/ renamed, a grep built without -P — produced no output, exited 2, and the step printed "No hardcoded user paths found" while inspecting nothing. Now counts the files it scanned, fails if that is zero, and checks grep's own status via PIPESTATUS rather than the pipe's. The self-test fixture named "empty scan returns non-zero" was passing for the wrong reason: it deleted the plugin's files but left the directory, so scan_plugins_directory still returned {"demo"} and the non-zero came from the missing-README error. The guard it claimed to cover was never exercised. It now rmtree's the directory and asserts the fixture actually emptied plugins/ before relying on it. Same class of bug as everything else here, one level up. Concurrency: every `labeled` event that was not `deep-review` resolved to the `fast` key with cancel-in-progress. Adding `dependencies` mid-review cancelled the running review and then skipped both jobs, leaving a cancelled check, no review, and no event that would re-trigger one until the next push. Each label now gets its own key. The version-increment check read the old version at `base.sha` while changed_plugins diffed `base...HEAD`. Those disagree once anything lands on main after a branch forks, so a PR could fail for not out-bumping a sibling it never saw. Both halves now use the merge base, and the error message says to rebase. Smaller: the sidecar ban only matched `.agents/plugins/marketplace.json` while AGENTS.md bans `.agents/` — widened, with fixtures for it and for `plugins/*/.codex-plugin`, neither of which the harness covered. `make shell` globbed only plugins/, so the one new shell script in this PR went unchecked locally. The deep job's fork exclusion is now explicit rather than relying on secrets being unavailable to fork events. changed_plugins' docstring said working tree; it compares commits.	2026-07-29 19:52:28 -04:00
William Tan	f09e5c729a	Remove legacy codex compatiblity scripts/shims. (#173) ... * Remove legacy codex compatiblity scripts/shims. Codex supports claude plugins so this shouldn't be necessary. Add a script to test the plugin loadablility in both claude and codex * fix: resolve code review findings for PR #173 Review findings addressed (4 reviewers: pr-review-toolkit agents, Codex gpt-5.3-codex, direct diff review): P2 fixed: - Bump versions for the 5 substantively changed plugins in both plugin.json and marketplace.json (gh-cli 1.5.0 new skill, claude-in-chrome-troubleshooting 1.1.0 skill rename, modern-python 1.5.1 / skill-improver 1.0.3 hooks change, zeroize-audit 0.1.1 MCP config relocation) so clients pick up the changes - README Codex install: replace unpasteable /plugins slash-command block with verified CLI syntax (codex plugin marketplace add) - check_claude_loadability: parse_json_output now fails fast with command context on empty CLI output instead of returning None - check_codex_loadability: surface skipped RPC error messages in timeout failures instead of a bare TimeoutError P3 fixed: - Both checkers: error out when marketplace.json lists no plugins instead of passing vacuously Dismissed: - @latest CLI installs in validate.yml: deliberate; the check validates against the clients users actually run - select.select portability: CI-only script on ubuntu-latest - Divergent mcpServers validation between checkers: intentional; the Codex checker enforces the repo's .mcp.json convention Verified: ruff, prek, validate_plugin_metadata.py, and both loadability checks pass end-to-end (39 plugins, 74 skills, 2 MCP servers load in Claude Code and Codex) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Dan Guido <dan@trailofbits.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-05 12:18:09 -04:00
Artem Dinaburg	aab484c5b7	Add Codex-native skill installation support (#123) ... * Add Codex-native skill installation support * Document Codex install commands in README * Enforce Codex skill mappings in CI * Fix review issues in Codex skill support - Fix ruff line-length violations in validate_codex_skills.py - Rewrite gh-cli SKILL.md description to third-person voice - Fix misleading error messages: describe actual symlink fix instead of referencing the user-local installer script - Add early check for missing .codex/skills/ directory - Distinguish dangling symlinks from mismatched symlinks - Add defensive ValueError handling in rel() - Add PLUGINS_DIR existence guard - Add install count and zero-install warning to installer - Add SOURCE_DIR existence check to installer Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Dan Guido <dan@trailofbits.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-03-16 14:29:40 -04:00
Paweł Płatek	f4b2a7218f	better ci validations, fix marketplace error (#85) ... * better ci validations, fix marketplace error * fix lints * fix lint script * fix lint script2 * fix ruff errors * fix abs path regex * fix shellcheck lint * fix: resolve code review findings for PR #85 - Remove dead CHANGED_FILES code path (unreachable in CI) - Add version and description consistency checks between plugin.json and marketplace.json - Eliminate duplicate plugin.json reads via parse_plugin_json() - Fix bats test discovery for filenames with spaces (print0/xargs -0) - Sync marketplace.json with plugin.json for 5 pre-existing mismatches (second-opinion version, ask-questions/burpsuite/fix-review/insecure-defaults descriptions) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: gh-cli bats tests fail when gh is in /usr/bin The _no_gh test helpers used PATH=/usr/bin:/bin to exclude gh, but on Ubuntu CI runners gh is installed at /usr/bin/gh. Fix by creating a temp directory with symlinks to only jq and bash. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: improve plugin descriptions for better skill triggering - ask-questions-if-underspecified: restore trigger context ("asking questions") while keeping invocation constraint - burpsuite-project-parser: drop filler "directly from the command line", use outcome-oriented "for security analysis" - insecure-defaults: restore specific scenarios (hardcoded credentials, fallback secrets, weak auth defaults) for better matching Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Dan Guido <dan@trailofbits.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>	2026-02-12 10:28:33 -05:00