mirror of
https://github.com/zernie/vigiles.git
synced 2026-09-14 20:53:57 +08:00
zernie
32fb830814
* fix(harness): parse every content block, not just `text` `flattenContent` read `b.text` and returned "" for anything else. Eight of the sixteen `ContentBlockParam` variants carry their payload in `content`, so half the union was invisible to every instrument built on `extractRequest` — `requestContains`, refs-nudge, injectable-events-delivery. What that cost. Claude Code <= 2.1.227 delivered a PostToolUse hook's additionalContext as its own `text` block; from 2.1.228 (a PATCH) it arrives appended to the `tool_result` block's `content` inside a <system-reminder>. Both deliver. Our probe went blind, three tests reported a false negative, and that reading was written up as #231 and nearly filed upstream as a regression in somebody else's product. Proven by changing only the parser on one machine at claude 2.1.267: blind = "landed=false", typed = "landed=true". Both harnesses now green on 2.1.267 with no change to `injectableEvents`. The guard is a typed exhaustive switch over the union from @anthropic-ai/sdk (MIT, types only, devDependency — erased at compile time, never in a consumer's tree). Mutation-proven: deleting the `tool_result` case fails tsc with "Type 'ToolResultBlockParam' is not assignable to type 'never'". An unknown block serialises rather than vanishing — this is a measurement instrument and its proven failure mode is the false negative. mock-model.test.ts asserted the blind spot as INTENDED ("no `text` -> ''"), which is why no run could find it; that assertion is now inverted and a newer-API block type is asserted as over-included. Refs #231. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(harness): parse every content block, not just `text` — the probe was blind, not the product `flattenContent` read `.text` and returned "" for anything else. Eight of the sixteen ContentBlockParam variants carry their payload in `content`, so half the union was invisible to every instrument built on extractRequest — requestContains, refs-nudge, injectable-events-delivery. What that cost: Claude Code <= 2.1.227 delivered a PostToolUse hook's additionalContext as its own text block; from 2.1.228 (a PATCH) it arrives inside the tool_result block's content. Both delivered. Our probe went blind, every delivery test reported a false negative, that reading was written up as #231 and came within one word of being filed upstream as somebody else's regression. Proven both ways on one machine at claude 2.1.267, changing only the parser: blind = landed=false, typed = landed=true. Mutation: deleting the tool_result case fails tsc with "Type 'ToolResultBlockParam' is not assignable to 'never'". The fix is a construction, not a patch: ContentBlockParam from @anthropic-ai/sdk (MIT, types only, devDependency — erased at compile time, never in a consumer's tree) + an exhaustive switch + a never binding. A seventeenth block type now breaks the BUILD instead of quietly emptying a measurement. An unrecognised block serialises rather than vanishing: for a measurement instrument the false negative is the failure mode that costs, so over-inclusion is the deliberate asymmetry. Also corrected: the mock-model test that asserted the blind spot as INTENDED, the requestContains doc comment that claimed to search "every message", and the false prose in the delivery harness. The Codex parser has the same idiom and a wider hole (32 input-item variants, one read) — named in code, roadmapped P1, not fixed here: closing it changes what `prompt` means. npm run check 18/18 RC=0 · test:harness 14 passed RC=0 · both delivery harnesses green on 2.1.267. * test: exercise every branch of the exhaustive block switch CI job `test` was red on the 100% coverage gate — 3836 tests passed, nothing failed. mock-model.ts sat at 95.65% lines / 95.83% funcs with 328, 340-341 and 353 uncovered: the `thinking`, `document` and no-text (`image`, `redacted_thinking`, `container_upload`) cases of the switch this PR introduced. The missing function was the arrow inside the `document` case, which no run reached. That gap is the PR's own subject, one level up. A case no run exercises is a case whose behaviour nobody has checked, and this function's defect was exactly that: a family of blocks flattening to "" with a test asserting it as intended. Writing the cases out satisfies tsc; running them is what checks them. Also asserted rather than merely commented: `tool_use` carrying a needle in its input contributes nothing, so a match inside a tool ARGUMENT can never be read as "the model was told this". Measured on the file alone: lines 95.65 -> 100, funcs 95.83 -> 100. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(harness): walk object-valued content payloads, not just string and array Review on this PR found the fix incomplete, and incomplete in the same shape as the defect it replaces. `flattenBlock` grouped eight variants as "the ones that carry `content`" and handed each to `flattenContent`, which accepts a string or an array. Measured against the SDK types, SIX of those eight carry an OBJECT there — web_fetch, web_search, code_execution, bash_code_execution, text_editor_code_execution and tool_search results — so they still flattened to "". The grouping was made on the field's NAME while the defect lives in its TYPE: the same assumption about shape as the `.text`-only read, one level up. `document` was wrong too — its text is under `source`, not beside it. No single extra field closes it: the payload's text sits at a different key in every shape (`stdout`/`stderr` on a bash result, a nested `content` document on a fetch result, `data` on a plain-text source). Picking one re-commits the assumption. `flattenUnknown` walks every string leaf and commits to none. One exclusion, and it is the only over-inclusion that costs more than the false negative it avoids: a `type: "base64"` source is bytes, and emitting megabytes of encoding would bury every real match. `type` and `media_type` are dropped as discriminators, not payload. Also closes the coverage gate the previous commit addressed, now over the walk as well: an object payload, a nested fetched document, a plain-text source, a skipped base64 source, and a numeric leaf (not text, contributes nothing — which is how this test caught a wrong expectation of mine rather than the reverse). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test: cover the absent-content arm of a tool_result `ToolResultBlockParam.content` is optional, and the block with it absent had never been through a run: statements on mock-model.ts sat at 99.09 with the `""` arm of line 338 unexercised. Not percentage-chasing. An absent `content` and an unrecognised shape both flatten to "" for entirely different reasons — one is a block that carries nothing, the other is a block this parser does not model — and nothing distinguished them until now. Also formatted: prettier flagged the previous commit's test block, and `npm run lint` does not look at formatting at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(harness): decide each block by what its fields MEAN, not what they are named Second review round on the same file, so this stops patching per finding and sweeps the union instead. Codex flagged `search_result`: its `title` and `source` sit BESIDE `content` and were dropped. Enumerating all 16 variants found a second one it did not name — `text` carries `citations[].cited_text` and `document_title`, quoted source text the model was shown. Two rounds of findings in one file mean the patch-per-finding is not converging, and it was not: both rounds are the same mistake, which is reading a block through the field NAME that happened to be salient. So the switch is now split by what the fields MEAN. Blocks whose strings are readable text are WALKED (`text`, `search_result`, `document`) and no field is singled out. Blocks read narrowly now NAME the field they drop and why, so the claim can be checked instead of trusted: `thinking.signature` is an attestation blob; `tool_use_id` is a correlation id; `redacted_thinking.data` is encrypted; `container_upload.file_id` is an identifier; `tool_use`/`server_tool_use` are the model's own call, excluded on purpose. The point of naming them: a third gap now reads as a missing row in a list rather than an unexamined assumption. Sweep recorded, since it is the evidence and not an opinion — of the 16 variants, exactly two were dropping readable text, and both are fixed here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
154 lines
4.5 KiB
JSON
154 lines
4.5 KiB
JSON
{
|
|
"name": "vigiles",
|
|
"version": "0.0.0-semantically-released",
|
|
"description": "Audit, test and measure the harness your AI agent runs on — grade your CLAUDE.md / AGENTS.md, skills, subagents and hooks, run them against a scripted model, and measure whether they actually fire.",
|
|
"keywords": [
|
|
"claude-code",
|
|
"codex",
|
|
"agents",
|
|
"agentic",
|
|
"ai",
|
|
"llm",
|
|
"harness",
|
|
"hooks",
|
|
"skills",
|
|
"claude",
|
|
"agents-md",
|
|
"eval",
|
|
"testing",
|
|
"linter"
|
|
],
|
|
"homepage": "https://github.com/zernie/vigiles#readme",
|
|
"repository": {
|
|
"type": "git",
|
|
"url": "git+https://github.com/zernie/vigiles.git"
|
|
},
|
|
"bugs": {
|
|
"url": "https://github.com/zernie/vigiles/issues"
|
|
},
|
|
"license": "MIT",
|
|
"workspaces": [
|
|
"packages/*",
|
|
"report",
|
|
"site"
|
|
],
|
|
"bin": {
|
|
"vigiles": "dist/cli.js"
|
|
},
|
|
"main": "./dist/test.js",
|
|
"types": "./dist/test.d.ts",
|
|
"exports": {
|
|
".": "./dist/test.js",
|
|
"./spec": "./dist/core/spec.js",
|
|
"./eval": "./dist/eval-surface.js",
|
|
"./linting": "./dist/linting.js",
|
|
"./hook": "./dist/hook.js",
|
|
"./claude-code": "./dist/claude-code.js",
|
|
"./codex": "./dist/codex.js",
|
|
"./adapter": "./dist/adapter.js",
|
|
"./vitest": {
|
|
"types": "./dist/vitest.d.mts",
|
|
"default": "./dist/vitest.mjs"
|
|
},
|
|
"./jest": "./dist/jest.js"
|
|
},
|
|
"files": [
|
|
"dist/**/*.js",
|
|
"dist/**/*.mjs",
|
|
"dist/**/*.d.ts",
|
|
"dist/**/*.d.mts",
|
|
"dist/audit-report.template.html",
|
|
"!dist/**/*.test.js",
|
|
"!dist/**/*.test.d.ts",
|
|
"action.yml",
|
|
"plugin.json",
|
|
".claude-plugin",
|
|
"hooks",
|
|
"skills",
|
|
"!skills/**/*.spec.ts",
|
|
"README.md",
|
|
"LICENSE"
|
|
],
|
|
"scripts": {
|
|
"build": "tsc && node scripts/build-report.mjs",
|
|
"build:core": "tsc",
|
|
"test": "npm run build && vitest run",
|
|
"coverage": "npm run build && vitest run --coverage",
|
|
"lint": "eslint src/",
|
|
"fmt": "prettier --write .",
|
|
"fmt:check": "prettier --check .",
|
|
"test:unit": "npm run build && vitest run --project unit",
|
|
"test:integration": "npm run build && vitest run --project integration",
|
|
"test:e2e": "npm run build && vitest run --project e2e",
|
|
"test:cli-e2e": "bash test/e2e/run.sh",
|
|
"test:harness": "npm run build && node dist/cli.js test",
|
|
"test:eval": "npm run build && node dist/cli.js eval --all",
|
|
"test:vitest": "npm run build && vitest run --project runners",
|
|
"test:jest": "npm run build && jest",
|
|
"test:types": "npm run build && tsc --noEmit -p test/types/tsconfig.json",
|
|
"api:report": "npm run build && node scripts/api-extractor.mjs --local",
|
|
"api:check": "npm run build && node scripts/api-extractor.mjs",
|
|
"check": "node scripts/check.mjs",
|
|
"docs:check": "npm run build && node scripts/check-doc-imports.mjs . docs README.md",
|
|
"exports:check": "npm run build && node scripts/check-export-prefixes.mjs .",
|
|
"internal:check": "npm run api:check && node scripts/check-internal-tag.mjs",
|
|
"docs:api": "typedoc"
|
|
},
|
|
"devDependencies": {
|
|
"@anthropic-ai/sdk": "^0.124.0",
|
|
"@eslint/js": "^10.0.1",
|
|
"@jackchuka/mdschema": "^0.12.8",
|
|
"@microsoft/api-extractor": "^7.58.9",
|
|
"@semantic-release/commit-analyzer": "^13.0.1",
|
|
"@types/js-yaml": "^4.0.9",
|
|
"@types/markdown-it": "^14.1.2",
|
|
"@types/minimatch": "^5.1.2",
|
|
"@types/node": "^20.19.39",
|
|
"@types/picomatch": "^4.0.2",
|
|
"@typescript-eslint/eslint-plugin": "^8.58.0",
|
|
"@typescript-eslint/parser": "^8.58.0",
|
|
"@vitest/coverage-v8": "^4.1.8",
|
|
"conventional-changelog-conventionalcommits": "^10.3.0",
|
|
"eslint": "^10.1.0",
|
|
"eslint-import-resolver-typescript": "^4.4.5",
|
|
"eslint-plugin-boundaries": "^6.0.2",
|
|
"eslint-plugin-sonarjs": "^4.0.2",
|
|
"globals": "^17.4.0",
|
|
"jest": "^30.4.2",
|
|
"picomatch": "^4.0.4",
|
|
"prettier": "^3.8.1",
|
|
"tsx": "^4.21.0",
|
|
"typedoc": "^0.28.19",
|
|
"typescript": "^5.9.3",
|
|
"vitest": "^4.1.8"
|
|
},
|
|
"peerDependencies": {
|
|
"jest": ">=28",
|
|
"vitest": ">=1"
|
|
},
|
|
"peerDependenciesMeta": {
|
|
"jest": {
|
|
"optional": true
|
|
},
|
|
"vitest": {
|
|
"optional": true
|
|
}
|
|
},
|
|
"dependencies": {
|
|
"@ast-grep/lang-python": "^0.0.6",
|
|
"@ast-grep/lang-ruby": "^0.0.7",
|
|
"@ast-grep/lang-rust": "^0.0.7",
|
|
"@ast-grep/napi": "^0.43.0",
|
|
"@iarna/toml": "^2.2.5",
|
|
"ci-info": "^4.4.0",
|
|
"cosmiconfig": "^9.0.1",
|
|
"glob": "^13.0.6",
|
|
"js-yaml": "^4.1.0",
|
|
"markdown-it": "^14.3.0",
|
|
"minimatch": "^10.0.1",
|
|
"mvdan-sh": "^0.10.1",
|
|
"ts-essentials": "^10.2.1",
|
|
"typescript": "^5.9.3"
|
|
}
|
|
}
|