mirror of
https://github.com/trailofbits/skills.git
synced 2026-09-14 14:28:48 +08:00
Akshay K
304c81a8ce
* fix(codeql): make the skill's guards real, add tests, trim the prose Every verification step piped its command to a formatter and used the pipeline's exit status, which without pipefail belongs to the formatter, not the command. Nine sites. The sharpest was the arm64e detection: `EXIT_CODE=$?` after `| tee` compared against 137, a value it could never hold, so the check underpinning Essential Principle #5 and three Rationalizations could not fire. The build workflow now runs check_db_quality.py as its Step 4 exit condition rather than describing metrics it never compared to a threshold, and suite generation aborts on zero resolved queries. Two blocks that were invalid bash — an else branch containing only comments — are fixed. Log helpers move to scripts/build_log.sh, sourced by the workflow and by the three reference docs that use run_logged. They were previously defined in one markdown file and used from three others, so those blocks failed standalone. As the skill's first .sh file it is also the first thing here that `make shell` lints. Sourcing it now checks the log is writable: under pipefail an unwritable log made tee's failure the pipeline's, so run_logged returned 1 for a build that succeeded and the method ladder walked to --build-mode=none blaming CodeQL. Suite generation moves to scripts/generate_suite.sh, which takes the mode as its argument. Both modes had been copy-pasted into two reference docs sharing some 25 lines of identical scaffolding — guards, the third-party pack loop, the excludes, the verification call — none of it lintable where it sat. The tests now run the script instead of extracting bash from markdown, and one of them fails if either doc inlines a generation block again. check_db_quality.py counts project files under the source root recorded in codeql-database.yml instead of against a hardcoded prefix list that only knew where macOS keeps its toolchain. src.zip stores each file at its absolute path minus the leading separator, so the recorded root is a prefix of the project's entries and of nothing else — verified against a database built by CodeQL 2.25.6. It also resolves the summary-versus- severity preference per extractor: a single `extractor-failures: 0` used to suppress the fallback for every other language in the database. run-analysis.md had its own database discovery loop, which SKILL.md states the workflows do not restate. It also lacked SKILL.md's `codeql resolve database` filter, so a marker file left by a failed build could be selected as a database. It now uses the canonical block. Twenty-odd snippets across language-details, threat-models, and performance-tuning created and analysed a literal `codeql.db` in the working directory — the exact shortcut SKILL.md lists as a Rationalization to Reject, and one its Success Criteria forbid. They use "$DB_NAME" under $OUTPUT_DIR. Adds six hermetic test suites needing no CodeQL install, plus the first tests that execute build_log.sh rather than reading it. They cover the shell in every markdown block, both suite generators, the quality thresholds, the .qls templates, and the exit-status claim the build ladder rests on. Corrects the run-all coverage figures against cpp-queries 1.8.0: the pack holds 515 alert queries, not 510, and run-all leaves 307 of them unrun, not 302 — 13 of those are Security/CWE queries, which the prose described as harmless refactoring metrics. Trims the skill's markdown from 2867 lines to 2748, and SKILL.md from 269 to 257. Out: the When NOT to Use section that #216 dropped repo-wide, five Essential Principles that restated their own Rationalization, two literal prompt mock-ups, three Reference Index rows duplicating the workflow table above them, three identical qlpack.yml blocks, and code fences in performance-tuning and threat-models that each carried a single flag. Replaces three approval prompts with one confirmation gate, and drops allowed-tools entries for tools that no longer exist. Bumps static-analysis to 1.3.0. * fix(codeql): address review findings on shell-block scope and suite claims Each markdown block runs in a fresh shell, so scalars and sourced functions cross it no better than arrays do. build-database.md now says to re-source build_log.sh and re-set DB_NAME in every block using run_logged: without it run_logged exits 127, the method ladder reads that as a failed build method, and it walks to --build-mode=none having never invoked CodeQL. run-analysis.md Step 4 re-establishes DB_NAME, RAW_DIR and SUITE_FILE for the same reason -- under set -u it aborted with "unbound variable" before analysis started. create-data-extensions.md was the third caller of database discovery and still used a bare find for codeql-database.yml, which selects a marker left by a build killed mid-run; the queries then return nothing and Step 3 reports coverage as adequate. It now uses find_databases.sh like the other two. quality-assessment.md sources build_log.sh in the Collect Metrics block, so a failed quality gate is recorded rather than silently dropped by a command-not-found, and the raised-threshold override logs inside the if -- it previously wrote "raised to 15%" even when the re-run still failed. generate_suite.sh no longer describes the run-all suite as every security, experimental and quality query: it imports two suites totalling 219 of the pack's 515 alert queries, as run-all-suite.md documents. Changed in the doc template too, which test_generation_scripts.py pins to the script's output. The two remaining review findings were already fixed on this branch and needed no change: quality-assessment.md assigns ERROR_RATIO from the script's JSON before reading it, and extractor_error_count() resolves the summary-versus- severity preference per extractor rather than once for the tree. * style(codeql): trim comments in the shared shell scripts build_log.sh carried more comment than code. The consequence of an unwritable log -- the ladder walking to --build-mode=none after a build that succeeded -- was stated twice within ten lines; it is stated once now. find_databases.sh and generate_suite.sh get the same treatment. Comments only: the diff contains no code lines. 435 tests pass unchanged. * feat(codeql): ship /static-analysis:codeql-build as a dynamic workflow The build is the part of this skill with real judgement and no user in it: try a method, read the failure, apply a fix, retry, escalate. That loop now runs unattended as workflows/codeql-build.js, beside the four workflows already on main. Three phases. Detect resolves the output directory and profiles the language, build system and macOS arm64e state. Build walks the ladder. Assess runs check_db_quality.py and applies the improvements from quality-assessment.md before re-running it. The ladder is deterministic and lives in the script; diagnose-fix-retry for a single rung lives in that rung's agent, where the build output it has to read already is. The agent is told not to escalate itself -- the caller owns the order, so a rung that fails is a result rather than a licence to try something else. Go and Swift never reach Method 4, which they reject outright; an arm64e Mac starts at 2m rather than spending two rungs to reach the same SIGKILL; an interpreted language does one extraction and no ladder at all. Nothing is asked. Every method failing returns no-method-succeeded, and a database that built but sits below the quality threshold returns built-below-threshold with its metrics. Whether the remaining extractor errors are confined to code nobody needs analysed is the caller's call, so the assess phase is told not to raise --max-error-ratio to make its own gate pass. A build command exiting 0 is not a database: every rung is confirmed with codeql resolve database before it counts, because finalize after a failed trace-command leaves one that resolves and holds nothing. tests/codeql_build_harness.js compiles the workflow with stubbed agents and asserts the ladder and the guards; --self-test mutates it six ways and requires every mutation to turn a scenario red. run_codeql_build_tests.sh wraps both so CI's existing shell-suite discovery runs them, with no workflow file changes. SKILL.md documents it as the unattended alternative and keeps the manual path. Database selection, analysis planning and data extensions stay in the session. * refactor(codeql): stop the docs recomputing what check_db_quality.py reports Collect Metrics parsed baseline-info.json with an inline python3 -c into BASELINE_LOC, then read .baseline_loc out of the checker's JSON into DB_LOC in the same block -- the same number, computed two ways, logged twice under two labels. The block's own comment said "Everything downstream reads $QUALITY_JSON rather than recomputing" while the recount sat sixteen lines above it. The inline parse and the print-baseline call before it are gone; the checker is the only thing that counts now, and the Quality Criteria table cites it rather than a command the doc no longer runs. Log Assessment read five variables out of Collect Metrics' shell. It runs in its own, so they expanded to empty and the log recorded "Baseline LoC:" with no number -- the same shape as the dangling ERROR_RATIO the first review found. It re-sources the helpers and re-reads the metrics, and no longer prints an expected-file count it cannot see. test_shell_blocks.py's embedded-python guard required three matches and there are two now, which is the guard working: removing the last sample of a construct must fail rather than silently leave the extractor untested. The floor moves to two, with the reason recorded. * test(codeql): drop test_suite_resolution.py, which never ran in CI Its six tests needed the CodeQL CLI and codeql/cpp-queries; CI installs neither, so every one of them reported as a skip on every run. Confirmed by reproducing CI's environment locally -- with codeql off PATH the directory gives 428 passed, 135 skipped, matching the job log exactly. Removing it leaves 428 passed, 129 skipped, and the remaining skips are per-block parametrisations of test_shell_blocks.py whose test functions do run for other blocks. What goes with it: the only check that resolved the suite templates against a real CodeQL rather than a fake one. run-all-suite.md's coverage claims -- that run-all is not the whole pack, that important-only reaches queries run-all does not -- are now prose nobody verifies, so the doc carries the command to re-derive them instead of pointing at a test file that is gone. test_generation_scripts.py's docstring no longer claims a sibling covers real-CLI resolution. * fix(codeql): Rust supports --build-mode=none; say so The language table left Rust as "check your CLI — not listed either way" and the Overview's three categories did not cover it at all, so a reader had no route for a Rust project and codeql-build.js resolved the ambiguity silently by putting Method 4 on its ladder. Settled against CodeQL 2.25.6 rather than the help text: `codeql database create --language=rust --build-mode=none` exits 0 and writes a database with `finalised: true`. Go, for contrast, fails immediately with "Go does not support the none build mode". So `--help` omits Rust the same way it omits C/C++, which the Overview already warned about; Rust joins that category and the ladder in codeql-build.js was right. * refactor(codeql): one prose copy of the quality-gate exit codes, not two check_db_quality.py's exit contract was written out four times: the script's own docstring, codeql-build.js's assess prompt, build-database.md Steps 4-5, and quality-assessment.md's Enforce the Thresholds. The first two earn it -- one is the source of truth, the other is an agent prompt that cannot read a docstring. The two prose copies are one too many. build-database.md keeps the part a reader needs at that moment (exit 1 is not a judgement call, exit 3 is) and defers the table to quality-assessment.md, which it already links and which is where someone looks for gate detail. The raised-threshold rationale there loses two lines it did not need. * refactor(codeql): assess phase reads the exit-code table instead of copying it The Assess prompt spelled out all four exit codes while its sibling phases point at a file -- Select says to read rulesets.md rather than choose from memory, and each build rung is sent to build-fixes.md. It now reads "Enforce the Thresholds" in quality-assessment.md the same way, keeping inline only the two facts that decide what it does: exit 1 is not overridable, exit 3 is a heuristic. That leaves one prose description of the contract instead of two, and a change to the script's exits reaches the agent without a second edit. Also aligns the ladder comment with the Rust finding: --help omits C/C++ and Rust, not just C/C++. * fix(codeql): pass --format=json, the flag check_db_quality.py defines The workflow's Assess phase ran `check_db_quality.py --json`. The script takes --format {text,json}, so argparse exited 2 before reading the database, and 2 is the one exit code ASSESS_SCHEMA does not describe. test_script_flags.py checks the class: it reads each script's accepted flags from its own --help and verifies every invocation across the plugin's .md and .js. The bug shipped because test_shell_blocks.py scans the skill tree and codeql-build.js sits outside it. * test(codeql): scan every block in one test instead of parametrizing over all of them Seven tests parametrized over all 65 bash blocks, which is 455 cases for seven assertions, and three of them skipped the blocks that did not qualify. That was 129 skips, and it hid a check that matched no block at all: the unpreserved-pipeline assertion had never run against the skill. Each now scans in one pass and lists every offending file:line, so a run reports all offenders rather than the first. 557 cases down to 118, none skipped. The array collector's empty case is an assertion rather than a skip. * refactor(codeql): point the workflow at build-database.md instead of restating it codeql-build.js carried its own copy of the build procedure: the arm64e detection block verbatim, the build-system command table, every method's invocation, and the output-directory logic. Both copies were live, which is how the workflow came to pass check_db_quality.py a --json flag while the doc had --format=json. Each rung now names its section of build-database.md, the way Method 2m already pointed at macos-arm64e-workaround.md. quality-assessment.md called the checker three times and re-derived its numbers with jq, unzip and grep. check_db_quality.py now reports archive_files and finalised from the two files it already reads, so one call covers the whole assessment. The fresh-shell rule was stated in five places; SKILL.md holds it once and the workflows link to it with the consequence specific to their site. test_section_pointers.py checks what this trade depends on: every "Section" in file.md pointer and every #anchor link must land on a real heading. The repo validator resolves paths, not section names, so a renamed heading would leave the pointers aimed at nothing with every check still green. Prose 1128 -> 1084 lines, codeql-build.js 390 -> 340. * fix(codeql): address PR review findings on skill paths and database selection codeql-build.js hardcoded plugins/static-analysis/skills/codeql, which only resolves in a checkout of this repo. Installed, the first build block sourced a build_log.sh that was not there and exited 127, so every rung of the ladder reported a build failure for a project that would have built. The Detect phase now resolves the directory at runtime from $CLAUDE_PLUGIN_ROOT, $CODEX_PLUGIN_ROOT, or a find over ~/.claude and ~/.codex, accepting a candidate only when scripts/build_log.sh exists. skillDir is a required schema field validated as absolute, so an unresolved path stops the run before the ladder starts. SKILL.md built FOUND_DBS in one bash fence and looped over it in the next. Each fence is a separate shell, so the metadata loop iterated zero times and the selection prompt had no language or creation time to show. The two are now one block. run-analysis.md Step 1 branched only on the zero-database case and fell through to FOUND_DBS[0] for any other count, analysing whichever database find returned first without telling the user there was a choice. It now uses the elif/else shape from create-data-extensions.md and exits with an error when DB_NAME is still unset. Two new checks cover the class rather than the instance. test_shell_blocks.py fails when a block reads an array it does not build. test_section_pointers.py fails on a repo-relative plugins/ path in the workflow, and resolves ${SKILL_DIR} pointers against the skill root so the workflow's file references stay checked: nine of them now, up from two. The rest of the review: run-all-suite.md no longer claims total coverage, which its own measurement section refutes; the analyze block expands optional flags as ${ARR[@]+"${ARR[@]}"}, since bash 3.2 treats an empty array under set -u as unbound; find_databases.sh exits 2 when codeql is absent instead of printing nothing, which auto-detection reads as "no databases, rebuild"; make -j$(nproc) falls back to sysctl -n hw.ncpu on macOS; every . build_log.sh site is || exit 1, as none of those blocks set -e; the Reference Index lists find_databases.sh and generate_suite.sh. * fix(codeql): source build_log.sh in every block that uses its helpers Each fenced block is its own Bash call, so a helper defined in an earlier one is undefined and exits 127. The build ladder reads that as a failed method and walks to the next one, reporting failure for a build that was never attempted. test_shell_blocks.py now fails any block that uses run_logged, log_step, log_cmd, log_result or LOG_FILE without sourcing build_log.sh, with fixtures pinning the detector in both directions. * fix(codeql): gate each step of the Method 3 multi-step build build_log.sh does not set -e, so the four run_logged calls ran regardless of each other: a failed trace-command still reached finalize, and the resulting database resolves while holding nothing, which the ladder reads as success. The steps now chain through if/elif, as 2m-a already does. test_shell_blocks.py fails an ungated `codeql database finalize`. * fix(codeql): stop database discovery reporting none when it found some find_databases.sh resolves each root to an absolute path, so the old -not -path '*/.*' exclusion also matched dotted ancestors: a checkout under ~/.cache or ~/.local had every database filtered out, the script printed nothing and exited 0, and the caller rebuilt from scratch. Prune dotted directories by name instead, with -mindepth 1 so a root that is itself dotted still searches. All three callers read the script through a process substitution, whose exit status is unobservable. Exit 2 (no codeql on this shell's PATH, a fresh shell per block) arrived as an empty list and was reported as "No CodeQL database found" for a project with several. Read it with command substitution and check the status. Tests cover the dotted-ancestor case, the dot-directory-below-root case the fix must not widen into, and a block scanner rejecting a process substitution around the script. * chore(static-analysis): bump version to 1.3.1 #231 bumped the plugin to 1.3.0 on main after this branch had already done the same, so merging main left HEAD and the merge base equal and the version-increment check failed. --------- Co-authored-by: kz-tob <kara.zaffarano@trailofbits.com>
399 lines
19 KiB
JavaScript
399 lines
19 KiB
JavaScript
// Ships as /static-analysis:codeql-build. Plugin workflows are namespaced by the plugin's
|
|
// `name`, so the prefix is always `static-analysis:`; meta.name supplies the rest.
|
|
export const meta = {
|
|
name: 'codeql-build',
|
|
description:
|
|
'Build a CodeQL database: detect the language, walk the build-method ladder applying fixes between rungs, and enforce the quality gate',
|
|
whenToUse:
|
|
'When a CodeQL database needs building and the build may not succeed first try — the ladder escalates autobuild → custom command → multi-step → no-build, diagnosing and retrying at each rung. Pass args as a JSON OBJECT: {"target": "/abs/path", "out": "/abs/path", "lang": "cpp"}. target defaults to cwd; lang is detected when omitted. Returns a status rather than asking anything, so the caller decides what to do when every method fails or the database is below the quality threshold. Database selection and analysis planning stay in the codeql SKILL.md.',
|
|
phases: [
|
|
{ title: 'Detect', detail: 'language, build system, macOS arm64e, exclusion config' },
|
|
{ title: 'Build', detail: 'walk the method ladder, diagnosing and retrying at each rung' },
|
|
{ title: 'Assess', detail: 'quality gate, apply improvements, re-run' },
|
|
],
|
|
}
|
|
|
|
// The skill directory is resolved at runtime by the Detect phase, not written down here. A
|
|
// workflow script cannot read $CLAUDE_PLUGIN_ROOT — there is no environment access — and the
|
|
// literal 'plugins/static-analysis/skills/codeql' this used to hold only resolves inside a
|
|
// checkout of the marketplace repo. For an installed plugin, under ~/.claude/plugins/…, every
|
|
// build block would source a build_log.sh that is not there, and the 127 that follows reads as
|
|
// a failed build: four rungs fail for a project that would have built. The probe below is the
|
|
// same shape c-review's SKILL.md uses — try the env var, fall back to a find, and validate the
|
|
// candidate against a marker file rather than trusting it.
|
|
const SKILL_DIR_PROBE = [
|
|
'1. Resolve SKILL_DIR: the codeql skill directory, the one holding SKILL.md, workflows/ and',
|
|
' scripts/. Every step below and both phases after this one read files under it. Try in',
|
|
' order, first hit wins, and accept a candidate only when $SKILL_DIR/scripts/build_log.sh',
|
|
' is a file:',
|
|
' a. "$CLAUDE_PLUGIN_ROOT/skills/codeql"',
|
|
' b. "$CODEX_PLUGIN_ROOT/skills/codeql"',
|
|
" c. find ~/.claude ~/.codex . -path '*/static-analysis/skills/codeql/scripts/build_log.sh' -print -quit 2>/dev/null,",
|
|
' then strip the trailing /scripts/build_log.sh. The home directories are searched',
|
|
' before . so an installed copy wins over one vendored into the project being scanned.',
|
|
' Report it as an absolute path. If none resolves, stop with ok=false naming what you',
|
|
' searched. Do not guess: a wrong SKILL_DIR fails every rung of the ladder on a missing',
|
|
' file, which reads as a project that cannot be built.',
|
|
].join('\n')
|
|
|
|
// args = { target, out, lang }. Prose is parsed too, since a caller passing a string would
|
|
// otherwise kill the run on the first line with nothing to show for it.
|
|
const parseArgs = (raw) => {
|
|
if (!raw) return {}
|
|
if (typeof raw === 'object') return raw
|
|
if (typeof raw !== 'string') return {}
|
|
const text = raw.trim()
|
|
if (text.startsWith('{')) {
|
|
try {
|
|
return JSON.parse(text)
|
|
} catch {
|
|
// Fall through to key: value parsing rather than dying on a malformed brace.
|
|
}
|
|
}
|
|
const KEYS = ['target', 'out', 'lang']
|
|
const out = {}
|
|
let key = null
|
|
for (const part of text.split(/;\s*|\n/)) {
|
|
const m = part.match(/^\s*(\w+)\s*:\s*([\s\S]*)$/)
|
|
if (m && KEYS.includes(m[1].toLowerCase())) {
|
|
key = m[1].toLowerCase()
|
|
out[key] = m[2].trim()
|
|
} else if (key && part.trim()) {
|
|
out[key] = `${out[key]} ${part.trim()}`.trim()
|
|
}
|
|
}
|
|
return out
|
|
}
|
|
|
|
const input = parseArgs(args)
|
|
|
|
// What matters is which build modes a language accepts, not whether it is interpreted. Go is
|
|
// compiled and has no `none` mode; C# and Java are compiled and do. CodeQL's own `--help` list
|
|
// omits C/C++ and Rust, both of which do support `none` (checked on 2.25.6).
|
|
const NO_BUILD = new Set(['python', 'javascript', 'typescript', 'ruby'])
|
|
const NO_NONE_FALLBACK = new Set(['go', 'swift'])
|
|
|
|
// The ladder for a compiled language. Method 2m replaces 1 and 2 on an affected Mac: the system
|
|
// toolchain is arm64e, CodeQL's libtrace.dylib is not, and macOS SIGKILLs the process (137), so
|
|
// trying them first spends two rungs to arrive at the same failure.
|
|
function ladderFor(lang, isMacosArm64e) {
|
|
if (NO_BUILD.has(lang)) return ['interpreted']
|
|
const rungs = isMacosArm64e ? ['2m'] : ['1', '2']
|
|
rungs.push('3')
|
|
// Go and Swift reject --build-mode=none outright, so Method 4 is not a fallback for them —
|
|
// it is a wasted cycle ending in the same "could not create a database".
|
|
if (!NO_NONE_FALLBACK.has(lang)) rungs.push('4')
|
|
return rungs
|
|
}
|
|
|
|
// Each rung names its section of build-database.md rather than restating the commands.
|
|
// Both copies were live before, and they had already drifted: the workflow was passing
|
|
// check_db_quality.py a --json flag the script does not define while the doc had it right.
|
|
//
|
|
// `detail` takes the resolved skill directory rather than closing over a constant, so the
|
|
// paths handed to an agent are the ones that exist on the machine the run is on. Each writes
|
|
// its target path out in full: `${SKILL_DIR}/workflows/build-database.md` is a pointer
|
|
// test_section_pointers.py can resolve and check, and a shared BUILD_DB constant was not.
|
|
const METHODS = {
|
|
interpreted: {
|
|
label: 'interpreted extraction',
|
|
detail: (SKILL_DIR) =>
|
|
`Run the command under "For Interpreted Languages" in ${SKILL_DIR}/workflows/build-database.md: one \`codeql database create\`, with --codescanning-config pointing at the exclusion config written in the Detect phase.`,
|
|
},
|
|
1: {
|
|
label: 'Method 1: autobuild',
|
|
detail: (SKILL_DIR) => `Follow "Method 1: Autobuild" in ${SKILL_DIR}/workflows/build-database.md.`,
|
|
},
|
|
2: {
|
|
label: 'Method 2: custom command',
|
|
detail: (SKILL_DIR) =>
|
|
`Follow "Method 2: Custom Command" in ${SKILL_DIR}/workflows/build-database.md, using BUILD_CMD from the Detect phase. Mind the quoting note there.`,
|
|
},
|
|
'2m': {
|
|
label: 'Method 2m: macOS arm64 toolchain',
|
|
detail: (SKILL_DIR) =>
|
|
[
|
|
`Follow the sub-method sequence in ${SKILL_DIR}/references/macos-arm64e-workaround.md:`,
|
|
'2m-a Homebrew compiler with multi-step tracing, then 2m-b Rosetta x86_64, then 2m-c the',
|
|
'system compiler. Stop at 2m-c: 2m-d is "ask the user", and this run reports back instead.',
|
|
].join('\n'),
|
|
},
|
|
3: {
|
|
label: 'Method 3: multi-step build',
|
|
detail: (SKILL_DIR) =>
|
|
`Follow "Method 3: Multi-step Build" in ${SKILL_DIR}/workflows/build-database.md. Every step must succeed before the next runs.`,
|
|
},
|
|
4: {
|
|
label: 'Method 4: no-build fallback',
|
|
detail: (SKILL_DIR) =>
|
|
`Follow "Method 4: No-Build Fallback" in ${SKILL_DIR}/workflows/build-database.md. A database built this way often fails the quality gate, and that is the honest result rather than a problem to work around.`,
|
|
},
|
|
}
|
|
|
|
const DETECT_SCHEMA = {
|
|
type: 'object',
|
|
required: ['ok', 'skillDir', 'lang', 'target', 'outputDir', 'dbPath', 'isMacosArm64e'],
|
|
additionalProperties: false,
|
|
properties: {
|
|
ok: {
|
|
type: 'boolean',
|
|
description: 'false when the skill directory did not resolve, CodeQL is missing, or no language could be detected',
|
|
},
|
|
skillDir: {
|
|
type: 'string',
|
|
description:
|
|
'absolute path of the codeql skill directory, validated by $SKILL_DIR/scripts/build_log.sh existing. Every later prompt is built from it, so a guess fails the whole ladder',
|
|
},
|
|
lang: { type: 'string', description: 'CodeQL language identifier, lowercase: cpp, java, python, go, …' },
|
|
target: { type: 'string', description: 'absolute path of the source root' },
|
|
outputDir: { type: 'string', description: 'absolute path of the created output directory' },
|
|
dbPath: { type: 'string', description: 'absolute path the database will be built at' },
|
|
isMacosArm64e: {
|
|
type: 'boolean',
|
|
description:
|
|
'true only on an arm64 Mac whose libtrace.dylib lacks arm64e while /usr/bin/make has it — the combination macOS SIGKILLs with exit 137',
|
|
},
|
|
buildSystem: { type: 'string', description: 'make, cmake, gradle, maven, cargo, dotnet, or "" when none was found' },
|
|
buildCommand: { type: 'string', description: 'the command Method 2 should pass to --command, or "" when unknown' },
|
|
error: { type: 'string', description: 'why ok is false, else ""' },
|
|
},
|
|
}
|
|
|
|
const BUILD_SCHEMA = {
|
|
type: 'object',
|
|
required: ['ok', 'resolved', 'attempts', 'error'],
|
|
additionalProperties: false,
|
|
properties: {
|
|
ok: {
|
|
type: 'boolean',
|
|
description:
|
|
'true only when `codeql resolve database` succeeds afterwards, for the reason given in the prompt. The build command exiting 0 is not enough.',
|
|
},
|
|
resolved: { type: 'boolean', description: 'the literal result of running codeql resolve database' },
|
|
attempts: { type: 'integer', description: 'how many times this method was run, including retries after a fix' },
|
|
fixesApplied: {
|
|
type: 'array',
|
|
description: 'fixes from build-fixes.md that were applied before a retry',
|
|
items: { type: 'string' },
|
|
},
|
|
error: { type: 'string', description: 'the failure, with the last lines of stderr, when ok is false; else ""' },
|
|
},
|
|
}
|
|
|
|
const ASSESS_SCHEMA = {
|
|
type: 'object',
|
|
required: ['exitCode', 'passed'],
|
|
additionalProperties: false,
|
|
properties: {
|
|
exitCode: {
|
|
type: 'integer',
|
|
description:
|
|
'final exit of check_db_quality.py. 0 pass; 1 nothing to analyse, not overridable; 3 error ratio above threshold; 4 diagnostics format changed',
|
|
},
|
|
passed: { type: 'boolean', description: 'true only when the final exit was 0' },
|
|
baselineLoc: { type: 'integer', description: 'from the checker JSON, or -1 if unavailable' },
|
|
projectFiles: { type: 'integer', description: 'from the checker JSON, or -1 if unavailable' },
|
|
errorRatio: { type: 'number', description: 'from the checker JSON, or -1 if unavailable' },
|
|
improvements: {
|
|
type: 'array',
|
|
description: 'quality improvements applied between runs of the checker',
|
|
items: { type: 'string' },
|
|
},
|
|
},
|
|
}
|
|
|
|
phase('Detect')
|
|
const detected = await agent(
|
|
[
|
|
'Prepare a CodeQL build: locate the skill, resolve where it writes, confirm the tool, and',
|
|
'profile the source.',
|
|
'',
|
|
`Source root: ${(input.target || '').trim() || 'the current working directory'}`,
|
|
(input.out || '').trim() ? `Output directory: ${input.out.trim()}` : 'Output directory: choose it as described below.',
|
|
(input.lang || '').trim() ? `Language: ${input.lang.trim()} — use it, do not re-detect.` : 'Language: detect it.',
|
|
'',
|
|
SKILL_DIR_PROBE,
|
|
'',
|
|
'2. Resolve the source root to an absolute path. Fail with ok=false if it is not a directory.',
|
|
'3. Confirm `codeql --version` succeeds. If not, stop with ok=false — there is nothing to',
|
|
' profile if the database cannot be built.',
|
|
(input.out || '').trim()
|
|
? '4. Use the output directory given above. mkdir -p it and resolve it absolutely.'
|
|
: '4. Pick the output directory with the block under "Output Directory" in $SKILL_DIR/SKILL.md, leaving USER_SPECIFIED_DIR unset so it auto-increments. mkdir -p and resolve it absolutely.',
|
|
' Set dbPath to <outputDir>/codeql.db.',
|
|
'',
|
|
'5. Detect the language by counting source files, and report the CodeQL identifier for it',
|
|
' (cpp, csharp, go, java, javascript, python, ruby, swift). CodeQL builds one language',
|
|
' per database. $SKILL_DIR/references/language-details.md has the mapping.',
|
|
'',
|
|
// Section name and path stay on one line in each of these: test_section_pointers.py reads
|
|
// `"Section" in <path>.md` out of the source, and a line break between the two hides the
|
|
// pointer from the check that the section still exists.
|
|
'6. macOS arm64e check. Run the block under "Step 2a: macOS arm64e Detection" in $SKILL_DIR/workflows/build-database.md',
|
|
' exactly as written and report its IS_MACOS_ARM64E. This is the only thing deciding',
|
|
' whether the ladder starts at Method 1 or jumps to 2m, so do not approximate it.',
|
|
'',
|
|
'7. For a compiled language, identify the build system and the command Method 2 should run,',
|
|
' from the detection table under "Method 2: Custom Command" in $SKILL_DIR/workflows/build-database.md.',
|
|
' Also look for build.sh, compile.sh and the README.',
|
|
'',
|
|
'8. For python, javascript, typescript or ruby only, write the exclusion config to',
|
|
' <outputDir>/codeql-config.yml. Compiled languages do not support it — everything traced',
|
|
' is extracted — so do not write one for them.',
|
|
].join('\n'),
|
|
{ label: 'detect', schema: DETECT_SCHEMA },
|
|
)
|
|
|
|
if (!detected || !detected.ok) {
|
|
const why = (detected && detected.error) || 'the detect phase returned nothing'
|
|
return { status: 'detect-failed', error: why, dbPath: '', method: '', attempts: [] }
|
|
}
|
|
const lang = String(detected.lang || '').trim().toLowerCase()
|
|
if (!lang) return { status: 'detect-failed', error: 'no language was detected', dbPath: '', method: '', attempts: [] }
|
|
// skillDir is in this list for the same reason as the rest: a relative path here resolves
|
|
// only when the run happens to start in the right directory, which is the failure the probe
|
|
// exists to prevent, and it would surface as four unexplained build failures rather than as
|
|
// a missing skill.
|
|
for (const [name, value] of [
|
|
['skillDir', detected.skillDir],
|
|
['target', detected.target],
|
|
['outputDir', detected.outputDir],
|
|
['dbPath', detected.dbPath],
|
|
]) {
|
|
if (!value || !String(value).startsWith('/')) {
|
|
return { status: 'detect-failed', error: `detect returned a non-absolute ${name}: ${JSON.stringify(value)}`, dbPath: '', method: '', attempts: [] }
|
|
}
|
|
}
|
|
const SKILL_DIR = String(detected.skillDir)
|
|
|
|
const ladder = ladderFor(lang, Boolean(detected.isMacosArm64e))
|
|
log(
|
|
`${lang} at ${detected.target}, database ${detected.dbPath}` +
|
|
`${detected.isMacosArm64e ? ', macOS arm64e — starting at Method 2m' : ''}` +
|
|
`${detected.buildSystem ? `, build system ${detected.buildSystem}` : ''}`,
|
|
)
|
|
log(`ladder: ${ladder.map((r) => METHODS[r].label).join(' → ')}`)
|
|
|
|
phase('Build')
|
|
// The ladder is deterministic and lives here; the diagnose-fix-retry loop for one rung lives
|
|
// inside that rung's agent, where the build output it has to read already is.
|
|
const attempts = []
|
|
let built = null
|
|
for (const rung of ladder) {
|
|
const method = METHODS[rung]
|
|
const result = await agent(
|
|
[
|
|
`Build the CodeQL database with ${method.label}.`,
|
|
'',
|
|
` OUTPUT_DIR=${detected.outputDir}`,
|
|
` DB_NAME=${detected.dbPath}`,
|
|
` CODEQL_LANG=${lang}`,
|
|
detected.buildCommand ? ` BUILD_CMD=${detected.buildCommand}` : '',
|
|
'',
|
|
'Each Bash call is a fresh shell, so start every block by re-establishing those and',
|
|
`re-sourcing the helpers: . "${SKILL_DIR}/scripts/build_log.sh" || exit 1. Without them`,
|
|
'run_logged is "command not found" and exits 127, which reads as a failed build rather',
|
|
'than a missing function — and the `|| exit 1` matters for the same reason, since the',
|
|
'helpers are not defined when an unwritable log makes the source fail.',
|
|
'',
|
|
method.detail(SKILL_DIR),
|
|
'',
|
|
'If it fails, diagnose before giving up. Read the build output, then consult',
|
|
`${SKILL_DIR}/references/build-fixes.md — clean existing state, clean the build cache,`,
|
|
'install missing dependencies, handle private registries — apply the ones the output',
|
|
'actually points to, and retry this method once. Do not move to a different build method:',
|
|
'the caller owns the ladder and will escalate.',
|
|
'',
|
|
'Then verify: `codeql resolve database -- "$DB_NAME"`. Report ok=true only when that',
|
|
'succeeds. A build command exiting 0 is not sufficient on its own — finalize after a',
|
|
'failed trace-command leaves a database that resolves and holds nothing, so if you ran',
|
|
'Method 3 confirm each step succeeded too.',
|
|
'',
|
|
'Report the fixes you applied and the last lines of stderr on failure. A method that',
|
|
'failed is a result to return, not a problem to work around.',
|
|
]
|
|
.filter(Boolean)
|
|
.join('\n'),
|
|
{ label: `build:${rung}`, schema: BUILD_SCHEMA },
|
|
)
|
|
|
|
const ok = Boolean(result && result.ok && result.resolved)
|
|
attempts.push({
|
|
method: method.label,
|
|
ok,
|
|
attempts: (result && result.attempts) || 0,
|
|
fixesApplied: (result && result.fixesApplied) || [],
|
|
error: ok ? '' : (result && result.error) || 'the build agent returned nothing',
|
|
})
|
|
log(`${method.label}: ${ok ? 'database resolves' : 'failed'}`)
|
|
if (ok) {
|
|
built = rung
|
|
break
|
|
}
|
|
}
|
|
|
|
if (!built) {
|
|
// Every rung failed. Returned rather than asked: the caller decides whether to change the
|
|
// build, install a toolchain, or stop.
|
|
return {
|
|
status: 'no-method-succeeded',
|
|
lang,
|
|
target: detected.target,
|
|
outputDir: detected.outputDir,
|
|
dbPath: '',
|
|
method: '',
|
|
attempts,
|
|
error: `every build method failed for ${lang}`,
|
|
}
|
|
}
|
|
|
|
phase('Assess')
|
|
const assessed = await agent(
|
|
[
|
|
'Run the quality gate on the database that was just built, and improve it if it fails.',
|
|
'',
|
|
` DB_NAME=${detected.dbPath}`,
|
|
` OUTPUT_DIR=${detected.outputDir}`,
|
|
'',
|
|
` uv run ${SKILL_DIR}/scripts/check_db_quality.py "$DB_NAME" --format=json`,
|
|
'',
|
|
`Read "Enforce the Thresholds" in ${SKILL_DIR}/references/quality-assessment.md for what`,
|
|
'each exit code calls for, rather than working it out from the number. The two that decide',
|
|
'what you do here: exit 1 is nothing to analyse and is not overridable, and exit 3 is a',
|
|
'heuristic that legitimate partial extraction can exceed.',
|
|
'',
|
|
'On exit 3 only, try the improvements further down that same file — adjust the source root,',
|
|
'force a clean rebuild if the build was cached, install type stubs, set extractor options —',
|
|
'then re-run the checker once and report the final exit code, with every improvement applied.',
|
|
'',
|
|
'Do not raise --max-error-ratio to make the gate pass. Whether the remaining errors are',
|
|
'confined to code that does not need analysing is the caller\'s call, not yours; report the',
|
|
'ratio and let them decide.',
|
|
'',
|
|
'Read baseline_loc, project_files and error_ratio from the checker\'s JSON. Use -1 for any',
|
|
'the JSON did not carry rather than guessing a number.',
|
|
].join('\n'),
|
|
{ label: 'assess', schema: ASSESS_SCHEMA },
|
|
)
|
|
|
|
const passed = Boolean(assessed && assessed.passed && assessed.exitCode === 0)
|
|
log(
|
|
passed
|
|
? `quality gate passed (${METHODS[built].label})`
|
|
: `quality gate failed with exit ${(assessed && assessed.exitCode) ?? '?'} — the database exists but is below threshold`,
|
|
)
|
|
|
|
return {
|
|
// Two success statuses, because a database that built but did not pass is a real outcome the
|
|
// caller has to decide about rather than a failure to hide or a pass to claim.
|
|
status: passed ? 'built' : 'built-below-threshold',
|
|
lang,
|
|
target: detected.target,
|
|
outputDir: detected.outputDir,
|
|
dbPath: detected.dbPath,
|
|
method: METHODS[built].label,
|
|
attempts,
|
|
quality: assessed || null,
|
|
logFile: `${detected.outputDir}/build.log`,
|
|
}
|