mirror of
https://github.com/proffesor-for-testing/agentic-qe.git
synced 2026-09-19 08:45:47 +08:00
Dragan Spiridonov
a39d4ad010
Closes the gap surfaced by the @xenova/transformers → protobufjs
investigation: `npm audit` in this repo lies about user-facing CVEs
because the top-level `overrides` block in package.json forces patched
versions of vulnerable transitives — but npm `overrides` are root-only
by design and do NOT propagate to downstream consumers. A clean
`npm install agentic-qe` from a user's perspective can therefore see
high/critical CVEs that our in-repo audit reports as 0.
Two new gates run against an actual consumer install tree:
1. `consumer-audit` job in npm-publish.yml (BLOCKING pre-publish).
Packs the local build with `npm pack`, installs the tarball into a
fresh tmp project, runs `npm audit --audit-level=high` there. Now a
`needs:` of the publish job, so a CVE invisible to repo-level audit
still blocks publish.
2. Consumer-audit step + P1 issue creation in post-publish-canary.yml
(NON-BLOCKING). Same idea, but against the freshly-published tarball
from the registry — catches discrepancies between what we built and
what the CDN serves. Findings open an auto-labelled P1 issue
(security/canary/release) and surface in the run's step summary.
The two together replicate the actual user view at both the
pre-publish gate and the post-publish safety net.
Local simulation of the consumer-audit logic on the current head:
183 packages installed, 0 vulnerabilities — confirming the
@huggingface/transformers migration in 122f223c actually fixed the
consumer view (was 4 high + 1 critical pre-migration).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
414 lines
15 KiB
YAML
414 lines
15 KiB
YAML
name: Publish to npm
|
|
|
|
on:
|
|
release:
|
|
types: [published]
|
|
workflow_dispatch:
|
|
inputs:
|
|
dry_run:
|
|
description: 'Perform a dry run (no actual publish)'
|
|
required: false
|
|
default: 'false'
|
|
type: boolean
|
|
|
|
# Required for npm trusted publishers (OIDC)
|
|
permissions:
|
|
contents: read
|
|
id-token: write
|
|
|
|
jobs:
|
|
build:
|
|
name: Build and Verify
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 30
|
|
|
|
steps:
|
|
- name: Checkout code
|
|
uses: actions/checkout@v4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@v4
|
|
with:
|
|
node-version: '24.13.0'
|
|
cache: 'npm'
|
|
registry-url: 'https://registry.npmjs.org'
|
|
|
|
- name: Install dependencies
|
|
run: npm ci
|
|
|
|
# Defense in depth: blocks publish if a high/critical CVE has landed in
|
|
# the production dep tree since the last main-branch CI run. Optimized
|
|
# CI already runs the same audit on every PR; this is the final gate.
|
|
- name: Block on high+critical CVEs in production deps
|
|
run: npm audit --audit-level=high --omit=dev
|
|
|
|
- name: Run type check
|
|
run: npm run typecheck
|
|
|
|
- name: Build project
|
|
run: npm run build
|
|
|
|
- name: Verify build output
|
|
run: |
|
|
if [ ! -d "dist" ]; then
|
|
echo "Error: dist directory not found"
|
|
exit 1
|
|
fi
|
|
if [ ! -f "dist/cli/bundle.js" ]; then
|
|
echo "Error: CLI bundle not found"
|
|
exit 1
|
|
fi
|
|
if [ ! -f "dist/index.js" ]; then
|
|
echo "Error: Main entry point not found"
|
|
exit 1
|
|
fi
|
|
echo "Build verification passed"
|
|
|
|
- name: Verify CLI and MCP bundles work
|
|
run: |
|
|
# CLI bundle is self-contained (esbuild) — no ESM resolution issues
|
|
node dist/cli/bundle.js --version
|
|
echo "CLI bundle: OK"
|
|
# MCP bundle is also self-contained
|
|
node -e "import('./dist/mcp/bundle.js').then(m => { console.log('MCP bundle: OK'); process.exit(0); }).catch(e => { console.error(e); process.exit(1); })"
|
|
# Verify dist files exist and are non-empty
|
|
test -s dist/index.js && echo "dist/index.js: OK"
|
|
test -s dist/cli/bundle.js && echo "dist/cli/bundle.js: OK"
|
|
test -s dist/mcp/bundle.js && echo "dist/mcp/bundle.js: OK"
|
|
timeout-minutes: 2
|
|
|
|
- name: Upload build artifacts
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: dist
|
|
path: dist/
|
|
retention-days: 1
|
|
|
|
# ============================================================================
|
|
# Pre-publish gate (issue #401)
|
|
#
|
|
# Runs `aqe init --auto` against the freshly-built tarball on a corpus of
|
|
# pinned real public repos. Blocks publish if any fixture fails.
|
|
#
|
|
# Why this exists:
|
|
# v3.9.1 through v3.9.4 all shipped broken init flows (deadlocks, hangs,
|
|
# missing files) because the only verification before publish was a build +
|
|
# bundle smoke test. Real-world init was tested by users *after* npm
|
|
# publish, by which point the damage was done. This job is the structural
|
|
# fix from issue #401's post-mortem.
|
|
#
|
|
# Budget: ~5 minutes total. The 4-fixture corpus is deliberately small —
|
|
# adding fixtures here trades publish latency for marginal coverage.
|
|
# Pathological encodings, symlink loops, and other chaos tests belong in a
|
|
# separate weekly workflow, not on the release path.
|
|
# ============================================================================
|
|
pre-publish-gate:
|
|
name: Pre-publish init gate (corpus)
|
|
needs: [build]
|
|
runs-on: ubuntu-latest
|
|
# 20 min budget. The gate itself is fast (init is 0-5s per fixture on
|
|
# CI), but the one-time shared `npm install` of agentic-qe takes 3-5
|
|
# min on CI runners including native-binary postinstalls
|
|
# (better-sqlite3, hnswlib-node). The 4 fixtures combined add ~30s of
|
|
# init + cleanroom tar work. 20 min leaves headroom for cold runner
|
|
# + uncached npm. A previous 10-min cap hit the timeout on v3.9.7's
|
|
# first publish attempt, which is how we learned the CI npm install
|
|
# is materially slower than a dev box. See run-gate.sh for the
|
|
# shared-install optimization that made this tractable.
|
|
timeout-minutes: 20
|
|
|
|
steps:
|
|
- name: Checkout code
|
|
uses: actions/checkout@v4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@v4
|
|
with:
|
|
node-version: '24.13.0'
|
|
cache: 'npm'
|
|
|
|
- name: Install dependencies
|
|
run: npm ci
|
|
|
|
- name: Download build artifacts
|
|
uses: actions/download-artifact@v4
|
|
with:
|
|
name: dist
|
|
path: dist/
|
|
|
|
- name: Pack agentic-qe tarball
|
|
id: pack
|
|
run: |
|
|
PACKAGE_VERSION=$(node -p "require('./package.json').version")
|
|
npm pack --pack-destination /tmp
|
|
TARBALL=/tmp/agentic-qe-${PACKAGE_VERSION}.tgz
|
|
if [ ! -f "${TARBALL}" ]; then
|
|
echo "Error: expected tarball ${TARBALL} not produced by npm pack" >&2
|
|
ls -la /tmp/*.tgz 2>/dev/null || true
|
|
exit 1
|
|
fi
|
|
echo "tarball=${TARBALL}" >> "$GITHUB_OUTPUT"
|
|
echo "Packed: ${TARBALL} ($(du -h ${TARBALL} | cut -f1))"
|
|
|
|
- name: Cache corpus tarballs
|
|
uses: actions/cache@v4
|
|
with:
|
|
path: tests/fixtures/init-corpus/.cache
|
|
# Cache key tied to the manifest content — bumping a fixture
|
|
# SHA invalidates the cache automatically.
|
|
key: init-corpus-${{ hashFiles('tests/fixtures/init-corpus/MANIFEST.json') }}
|
|
|
|
- name: Download + extract fixture corpus
|
|
run: ./tests/fixtures/init-corpus/setup.sh
|
|
|
|
- name: Run init gate against corpus
|
|
env:
|
|
AQE_LOCAL_TARBALL: ${{ steps.pack.outputs.tarball }}
|
|
run: ./tests/fixtures/init-corpus/run-gate.sh
|
|
|
|
- name: Upload gate logs (always)
|
|
if: always()
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: init-corpus-logs
|
|
path: tests/fixtures/init-corpus/run-logs/
|
|
retention-days: 7
|
|
|
|
- name: Append gate summary to job summary
|
|
if: always()
|
|
run: |
|
|
if [ -f tests/fixtures/init-corpus/run-logs/summary.txt ]; then
|
|
echo "## Init corpus gate" >> $GITHUB_STEP_SUMMARY
|
|
echo '```' >> $GITHUB_STEP_SUMMARY
|
|
cat tests/fixtures/init-corpus/run-logs/summary.txt >> $GITHUB_STEP_SUMMARY
|
|
echo '```' >> $GITHUB_STEP_SUMMARY
|
|
fi
|
|
|
|
# ============================================================================
|
|
# Tests on tag SHA (issue #401)
|
|
#
|
|
# Re-runs the fast unit suite on release tags. This is the safety net for
|
|
# the rare case where a release tag points to a SHA that never went
|
|
# through PR CI (e.g. a hotfix committed directly to main).
|
|
#
|
|
# The earlier "conditional" version of this job tried to skip re-running
|
|
# tests when optimized-ci.yml had already passed on the same SHA, by
|
|
# querying the GitHub Actions API. The devils-advocate review of #401
|
|
# found four failure modes in that conditional:
|
|
# 1. Matrix workflows with `continue-on-error: true` can report
|
|
# workflow-level success even when one shard failed.
|
|
# 2. Tag SHAs from `gh release create --target main` may differ from
|
|
# the PR-CI head SHA when merge commits are involved.
|
|
# 3. GITHUB_TOKEN rate limits silently break the optimization.
|
|
# 4. Renaming optimized-ci.yml silently breaks the optimization.
|
|
#
|
|
# Just always running the fast unit suite costs ~2 minutes, has no API
|
|
# calls, and is the simpler design. Workflow_dispatch dry-runs are
|
|
# excluded so manual debug runs don't burn CI minutes.
|
|
# ============================================================================
|
|
tests-on-tag-sha:
|
|
name: Tests on tag SHA
|
|
needs: [build]
|
|
if: github.event_name == 'release'
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 15
|
|
|
|
steps:
|
|
- name: Checkout code
|
|
uses: actions/checkout@v4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@v4
|
|
with:
|
|
node-version: '24.13.0'
|
|
cache: 'npm'
|
|
|
|
- name: Install dependencies
|
|
run: npm ci
|
|
|
|
- name: Run fast unit tests
|
|
run: npm run test:unit:fast
|
|
|
|
# ============================================================================
|
|
# Integration tests on tag SHA (issue #491)
|
|
#
|
|
# Runs the focused daemon-runtime seam suite — the surfaces that, when
|
|
# broken, ship as "looks initialized, silently dies" failures users only
|
|
# discover at runtime. The fast subset (~6s) covers:
|
|
# - fleet_init wires the daemon WorkerManager to the kernel (Bug 1)
|
|
# - every production worker can reach its declared targetDomains
|
|
# - LearningCoordinator's write→read round trip works through the
|
|
# real HybridMemoryBackend (Bug 2+3)
|
|
# - CapturedExperienceBridge writes the expected kv keys
|
|
#
|
|
# Why this is its own gate rather than part of tests-on-tag-sha:
|
|
# `test:unit:fast` excludes integration/ by design — and the four bugs
|
|
# in #491 all lived in the daemon-runtime seam that unit tests
|
|
# explicitly punt on. Until v3.9.31, integration coverage was present
|
|
# but not run on release; this job closes that loop.
|
|
# ============================================================================
|
|
integration-tests-on-tag-sha:
|
|
name: Integration tests on tag SHA
|
|
needs: [build]
|
|
if: github.event_name == 'release'
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 10
|
|
|
|
steps:
|
|
- name: Checkout code
|
|
uses: actions/checkout@v4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@v4
|
|
with:
|
|
node-version: '24.13.0'
|
|
cache: 'npm'
|
|
|
|
- name: Install dependencies
|
|
run: npm ci
|
|
|
|
- name: Run integration seam tests
|
|
run: npm run test:integration:fast
|
|
|
|
# ============================================================================
|
|
# Consumer-side audit gate
|
|
#
|
|
# `npm audit` against this repo runs from our perspective, where the
|
|
# top-level `overrides` block in package.json forces patched versions of
|
|
# otherwise-vulnerable transitives (e.g. protobufjs >=7.5.6). But npm
|
|
# `overrides` only apply when WE are the root package — they do NOT
|
|
# propagate to downstream consumers. So a repo audit can be clean while
|
|
# `npm install -g agentic-qe` simultaneously surfaces real high/critical
|
|
# CVEs to users.
|
|
#
|
|
# This job packs the local build, installs the tarball into a fresh tmp
|
|
# project (the same way a consumer would), and runs `npm audit` there.
|
|
# The fail signal is the consumer view, not ours.
|
|
#
|
|
# Caught after the v3.9.32 audit-gate work: 5 vulns (4 high, 1 critical)
|
|
# in @xenova/transformers → onnxruntime-web → onnx-proto → protobufjs
|
|
# were invisible to in-repo `npm audit` but live for every user.
|
|
# ============================================================================
|
|
consumer-audit:
|
|
name: Consumer-side audit (tarball install)
|
|
needs: [build]
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 10
|
|
|
|
steps:
|
|
- name: Checkout code
|
|
uses: actions/checkout@v4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@v4
|
|
with:
|
|
node-version: '24.13.0'
|
|
cache: 'npm'
|
|
|
|
- name: Install dependencies
|
|
run: npm ci
|
|
|
|
- name: Build
|
|
run: npm run build
|
|
|
|
- name: Pack tarball
|
|
id: pack
|
|
run: |
|
|
TARBALL=$(npm pack --json | jq -r '.[0].filename')
|
|
echo "tarball=${TARBALL}" >> "$GITHUB_OUTPUT"
|
|
echo "Packed: ${TARBALL}"
|
|
|
|
- name: Audit against clean consumer install
|
|
run: |
|
|
CONSUMER_DIR=$(mktemp -d)
|
|
pushd "${CONSUMER_DIR}" >/dev/null
|
|
echo '{"name":"audit-consumer","version":"0.0.0","private":true}' > package.json
|
|
# Install our tarball as a real consumer would.
|
|
npm install --no-audit --no-fund "${GITHUB_WORKSPACE}/${{ steps.pack.outputs.tarball }}"
|
|
# Now the moment of truth: audit from the consumer's POV, where
|
|
# our root-only overrides do NOT apply.
|
|
npm audit --audit-level=high
|
|
popd >/dev/null
|
|
|
|
publish:
|
|
name: Publish to npm
|
|
needs: [build, pre-publish-gate, consumer-audit, tests-on-tag-sha, integration-tests-on-tag-sha]
|
|
# tests-on-tag-sha and integration-tests-on-tag-sha are skipped on
|
|
# workflow_dispatch (they only run on release events), so we explicitly
|
|
# accept skipped as a non-blocker. Without this, the default GH Actions
|
|
# behavior would skip publish whenever either is skipped — breaking the
|
|
# dry-run path.
|
|
if: >-
|
|
always()
|
|
&& needs.build.result == 'success'
|
|
&& needs.pre-publish-gate.result == 'success'
|
|
&& needs.consumer-audit.result == 'success'
|
|
&& (needs.tests-on-tag-sha.result == 'success' || needs.tests-on-tag-sha.result == 'skipped')
|
|
&& (needs.integration-tests-on-tag-sha.result == 'success' || needs.integration-tests-on-tag-sha.result == 'skipped')
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 10
|
|
environment: npm-publish
|
|
|
|
steps:
|
|
- name: Checkout code
|
|
uses: actions/checkout@v4
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@v4
|
|
with:
|
|
node-version: '24.13.0'
|
|
cache: 'npm'
|
|
registry-url: 'https://registry.npmjs.org'
|
|
|
|
- name: Download build artifacts
|
|
uses: actions/download-artifact@v4
|
|
with:
|
|
name: dist
|
|
path: dist/
|
|
|
|
- name: Install dependencies
|
|
run: npm ci
|
|
|
|
- name: Verify package version matches tag
|
|
if: github.event_name == 'release'
|
|
run: |
|
|
PACKAGE_VERSION=$(node -p "require('./package.json').version")
|
|
TAG_VERSION="${{ github.ref_name }}"
|
|
# Remove 'v' prefix if present
|
|
TAG_VERSION="${TAG_VERSION#v}"
|
|
|
|
if [ "$PACKAGE_VERSION" != "$TAG_VERSION" ]; then
|
|
echo "Error: Package version ($PACKAGE_VERSION) does not match tag version ($TAG_VERSION)"
|
|
exit 1
|
|
fi
|
|
echo "Version verification passed: $PACKAGE_VERSION"
|
|
|
|
- name: Publish to npm (dry run)
|
|
if: github.event.inputs.dry_run == 'true'
|
|
run: npm publish --access public --dry-run
|
|
env:
|
|
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
|
|
|
|
- name: Publish to npm
|
|
if: github.event.inputs.dry_run != 'true'
|
|
run: npm publish --access public --provenance
|
|
env:
|
|
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
|
|
|
|
- name: Create publish summary
|
|
run: |
|
|
PACKAGE_NAME=$(node -p "require('./package.json').name")
|
|
PACKAGE_VERSION=$(node -p "require('./package.json').version")
|
|
|
|
echo "## Published to npm" >> $GITHUB_STEP_SUMMARY
|
|
echo "" >> $GITHUB_STEP_SUMMARY
|
|
echo "- **Package**: $PACKAGE_NAME" >> $GITHUB_STEP_SUMMARY
|
|
echo "- **Version**: $PACKAGE_VERSION" >> $GITHUB_STEP_SUMMARY
|
|
echo "- **Registry**: https://www.npmjs.com/package/$PACKAGE_NAME" >> $GITHUB_STEP_SUMMARY
|
|
echo "" >> $GITHUB_STEP_SUMMARY
|
|
if [ "${{ github.event.inputs.dry_run }}" == "true" ]; then
|
|
echo "> **Note**: This was a dry run. No package was actually published." >> $GITHUB_STEP_SUMMARY
|
|
else
|
|
echo "Install with: \`npm install $PACKAGE_NAME@$PACKAGE_VERSION\`" >> $GITHUB_STEP_SUMMARY
|
|
fi
|