Files
vercel__workflow/scripts/create-community-worlds-matrix.mjs
Pranay Prakash 54ba1888cd fix: compare benchmarks against PR base branch instead of main (#560)
* fix: compare benchmarks against PR base branch instead of main

- Use github.event.pull_request.base.ref instead of hardcoded main
- Remove search_artifacts: true to ensure most recent baseline is used
- For stacked PRs, this compares against the parent PR's baseline

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: group failed e2e tests by category and app in summary

Instead of listing each failed test as a separate item, group them by:
1. Category (world): e.g., "Community Worlds", "Vercel Production"
2. App (framework): e.g., "mongodb", "turso", "nextjs-turbopack"

This makes the summary much more readable when there are many failures.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: ensure local E2E tests always produce JSON output

- Add 'fastify' to app detection list in aggregate-e2e-results.js
- Change && to ; so e2e tests run even if dev.test.ts fails
- This ensures local-dev, local-prod, and local-postgres categories
  appear in the E2E summary comment

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: ensure local E2E tests always produce JSON output

- Add 'fastify' to app detection list in aggregate-e2e-results.js
- Change && to ; so e2e tests run even if dev.test.ts fails
- This ensures local-dev, local-prod, and local-postgres categories
  appear in the E2E summary comment

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: publish CI results to GitHub Pages for docs

- Add generate-docs-data.js script to create JSON summaries from CI artifacts
- Add publish-results job to tests.yml and benchmarks.yml workflows
- Update docs/lib/worlds-data.ts to fetch from GitHub Pages URLs
- Results published to https://vercel.github.io/workflow/ci/

This allows the docs worlds page to display actual test/benchmark
results without requiring a GITHUB_TOKEN.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: correct outputFile path for local E2E test artifacts

The --outputFile path was using ../../ which placed files outside the
repo because pnpm run test:e2e executes from workspace root, not from
the cd'd workbench directory. This prevented local-dev, local-prod, and
local-postgres test results from being uploaded as artifacts.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: show green checkmark for skipped tests instead of warning

Skipped tests are intentional and shouldn't show as warnings in the
E2E test summary comments.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: use collapsible sections in benchmark PR comment

Wrap each benchmark, stream benchmarks section, and summary tables in
<details> toggles to make the PR comment more compact and readable.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: add Vercel observability links to benchmark PR comments

- Store runId in benchmark timing data
- Add project-slug to Vercel benchmark matrix
- Pass WORKFLOW_VERCEL_PROJECT_SLUG env var to benchmarks
- Store Vercel metadata (teamSlug, projectSlug, environment) in timing files
- Generate observability deep links for each Vercel world benchmark
- Show observability links below Production (Vercel) tables

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: use correct Vercel project slugs for observability links

- nextjs-turbopack → example-nextjs-workflow-turbopack
- nitro-v3 → workbench-nitro-workflow

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-12-06 18:30:25 -08:00

73 lines
2.0 KiB
JavaScript

#!/usr/bin/env node
/**
* Generates a GitHub Actions matrix for community world testing.
* Reads from worlds-manifest.json and filters to testable community worlds.
*
* Usage: node scripts/create-community-worlds-matrix.mjs
*
* Output format (JSON):
* {
* "world": [
* {
* "id": "starter",
* "name": "Starter",
* "package": "@workflow-worlds/starter",
* "service-type": "none",
* "env-vars": "{\"WORKFLOW_TARGET_WORLD\":\"@workflow-worlds/starter\"}"
* },
* ...
* ]
* }
*/
import fs from 'fs';
import path from 'path';
import { fileURLToPath } from 'url';
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const rootDir = path.join(__dirname, '..');
// Read the manifest
const manifestPath = path.join(rootDir, 'worlds-manifest.json');
const manifest = JSON.parse(fs.readFileSync(manifestPath, 'utf-8'));
// Filter to community worlds that can be tested in CI
const testableWorlds = manifest.worlds.filter((world) => {
// Only community worlds
if (world.type !== 'community') return false;
// Skip worlds that require external credentials (e.g., Jazz needs API keys)
if (world.requiresCredentials) return false;
return true;
});
// Build the matrix
const matrix = {
world: testableWorlds.map((world) => {
// Determine service type based on services array
let serviceType = 'none';
if (world.services && world.services.length > 0) {
// Use the first service's name as the service type
// Currently supports: mongodb, redis
const serviceName = world.services[0].name;
if (['mongodb', 'redis'].includes(serviceName)) {
serviceType = serviceName;
}
}
return {
id: world.id,
name: world.name,
package: world.package,
'service-type': serviceType,
'env-vars': JSON.stringify(world.env || {}),
'setup-command': world.setup || '',
};
}),
};
// Output JSON for GitHub Actions
console.log(JSON.stringify(matrix));