Files
trailofbits__skills/.github/workflows/lint.yml
Eric Kilmer ec2450d05e Trim dwarf-expert: single-file skill, effort level, contract test (#223)
* Trim dwarf-expert: single-file skill, effort level, contract test

- Consolidate SKILL.md from four files into one (115 lines); delete
  reference/. Cut generic behavior rules, duplicate When-to-Use
  sections, a 9-line readelf reference hop, and an ASCII decision
  tree; fix five typos.
- Fix --lookup=<offset> misuse carried over from the original: DIE at
  a section offset is --debug-info=<offset>; --lookup takes a program
  address. Verified live against llvm-dwarfdump.
- Drop the invented v3-v5 version scoping from the skill, plugin.json,
  marketplace.json, and both READMEs. Version-specific judgment that
  earned its place (the -gdwarf-N pin inference, v2 surface forms)
  lives in the Verifying section.
- Frontmatter: add effort: medium, rewrite the trigger description,
  extend allowed-tools with Write/Edit/WebFetch.
- Add tests/test_skill_contract.py (stdlib + pytest): every dwarfdump
  flag documented in SKILL.md must exist in a live llvm-dwarfdump
  (>= 19, the verified floor for --error-display/--verify-json), plus
  a frontmatter contract check. Zero extracted flags or a missing
  tool fails rather than skips.
- CI: python-tests installs llvm-19, falling back to the default llvm
  on future runner images.
- Bump version 1.0.1 -> 1.1.0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address review: fence-aware extraction, capability-gated resolver

- body_sections() no longer strips fenced blocks from section text:
  fences are ignored only for heading detection, so flags documented
  inside examples are extracted and verified. New unit test covers
  both properties; a fenced --bogus-flag mutation now goes red.
- Replace the parsed-version LLVM floor with a capability gate: the
  resolver requires --error-display/--verify-json in --help, because
  Apple's LLVM numbering does not track upstream and a numeric major
  is not comparable across toolchains. Failure lists rejected tools
  with their version banners. README wording matches.
- Raise the extraction floor from 10 to 15 (21 flags documented).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: kz-tob <kara.zaffarano@trailofbits.com>
2026-08-03 11:45:58 -04:00

145 lines
5.7 KiB
YAML

name: Lint
on:
push:
branches: [main]
pull_request:
branches: [main]
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
pre-commit:
name: Pre-commit
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.13"
- uses: pre-commit/action@2c7b3805fd2a0fd8c1884dcaebf91fc102a13ecd # v3.0.1
bats:
name: Shell (bats)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Install bats and uv
run: |
sudo apt-get update
sudo apt-get install -y bats
curl -LsSf https://astral.sh/uv/install.sh | sh
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Run bats tests
# --no-run-if-empty would turn "discovery broke" into a silent pass. This repo
# ships bats suites, so finding none is a failure, not a clean run.
run: |
set -euo pipefail
mapfile -d '' files < <(find plugins -name '*.bats' -type f -print0)
if [ "${#files[@]}" -eq 0 ]; then
echo "ERROR: no .bats files found — discovery is broken"
exit 1
fi
echo "Running ${#files[@]} bats file(s)"
bats "${files[@]}"
- name: Run shell regression suites
# Matched by no glob in the previous discovery, so these never ran in CI.
run: |
# find, not a glob: `**` needs globstar and silently degrades to `*`
# without it, so a suite one directory deeper stops running with no signal.
# And an empty result is a failure, not a pass — this repo ships suites.
set -euo pipefail
mapfile -d '' suites < <(
find plugins -type f -path '*/tests/*' -name 'run_*.sh' -print0
)
if [ "${#suites[@]}" -eq 0 ]; then
echo "ERROR: no shell regression suites found — discovery is broken"
exit 1
fi
echo "Running ${#suites[@]} shell regression suite(s)"
for s in "${suites[@]}"; do
echo "::group::$s"
bash "$s"
echo "::endgroup::"
done
python-tests:
name: Python tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.13"
- name: Install pytest
# The rust-review / c-review test files are pytest-based
run: python3 -m pip install pytest
- name: Install toolchain dependencies
# constant-time-analysis tests cross-compile C to aarch64 via
# `clang --target=aarch64-unknown-linux-gnu`. Without the cross
# sysroot, clang fails with:
# fatal error: 'bits/libc-header-start.h' file not found
# libc6-dev-arm64-cross supplies the aarch64 headers and crt files;
# gcc-aarch64-linux-gnu pulls in binutils-aarch64-linux-gnu (the
# cross linker clang invokes). --no-install-recommends suppresses
# Recommends, so each package must be listed explicitly.
#
# llvm-19 supplies llvm-dwarfdump for the dwarf-expert contract test,
# which fails rather than skips when the tool is missing. 19 is the
# floor: --error-display and --verify-json landed in LLVM 19, and
# ubuntu-24.04's default llvm is 18. The fallback covers future
# runner images whose default llvm is >= 19 but that no longer
# package llvm-19 itself.
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
clang \
gcc-aarch64-linux-gnu \
libc6-dev-arm64-cross
sudo apt-get install -y --no-install-recommends llvm-19 \
|| sudo apt-get install -y --no-install-recommends llvm
- name: Discover and run plugin Python tests
shell: bash
run: |
# Intentionally omit -e: we want every directory to run even if one
# fails, then exit with a combined failure code via `failed`.
#
# pytest per directory, not `python3 <file>` per file. A test file with no
# `if __name__ == "__main__"` block exits 0 under the old loop having run
# nothing at all, which is indistinguishable from a pass. Running pytest
# from inside each directory (via `python3 -m`, which puts the CWD on
# sys.path) preserves the sibling imports these suites rely on, and
# --import-mode=importlib keeps c-review and rust-review from colliding on
# their identically-named test_split.py.
set -uo pipefail
mapfile -t dirs < <(
find plugins -type f \( -name 'test_*.py' -o -name '*_test.py' \) \
-exec dirname {} \; | sort -u
)
if [ "${#dirs[@]}" -eq 0 ]; then
echo "ERROR: no Python test files found — discovery is broken"
exit 1
fi
failed=0
for d in "${dirs[@]}"; do
echo "::group::$d"
if ! ( cd "$d" && python3 -m pytest -q --import-mode=importlib . ); then
failed=1
fi
echo "::endgroup::"
done
echo "Ran ${#dirs[@]} test directory/ies"
exit "$failed"