2 Commits

Author	SHA1	Message	Date
Pranay Prakash	54ba1888cd	fix: compare benchmarks against PR base branch instead of main (#560) ... * fix: compare benchmarks against PR base branch instead of main - Use github.event.pull_request.base.ref instead of hardcoded main - Remove search_artifacts: true to ensure most recent baseline is used - For stacked PRs, this compares against the parent PR's baseline 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: group failed e2e tests by category and app in summary Instead of listing each failed test as a separate item, group them by: 1. Category (world): e.g., "Community Worlds", "Vercel Production" 2. App (framework): e.g., "mongodb", "turso", "nextjs-turbopack" This makes the summary much more readable when there are many failures. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: ensure local E2E tests always produce JSON output - Add 'fastify' to app detection list in aggregate-e2e-results.js - Change && to ; so e2e tests run even if dev.test.ts fails - This ensures local-dev, local-prod, and local-postgres categories appear in the E2E summary comment 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: ensure local E2E tests always produce JSON output - Add 'fastify' to app detection list in aggregate-e2e-results.js - Change && to ; so e2e tests run even if dev.test.ts fails - This ensures local-dev, local-prod, and local-postgres categories appear in the E2E summary comment 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: publish CI results to GitHub Pages for docs - Add generate-docs-data.js script to create JSON summaries from CI artifacts - Add publish-results job to tests.yml and benchmarks.yml workflows - Update docs/lib/worlds-data.ts to fetch from GitHub Pages URLs - Results published to https://vercel.github.io/workflow/ci/ This allows the docs worlds page to display actual test/benchmark results without requiring a GITHUB_TOKEN. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: correct outputFile path for local E2E test artifacts The --outputFile path was using ../../ which placed files outside the repo because pnpm run test:e2e executes from workspace root, not from the cd'd workbench directory. This prevented local-dev, local-prod, and local-postgres test results from being uploaded as artifacts. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: show green checkmark for skipped tests instead of warning Skipped tests are intentional and shouldn't show as warnings in the E2E test summary comments. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: use collapsible sections in benchmark PR comment Wrap each benchmark, stream benchmarks section, and summary tables in <details> toggles to make the PR comment more compact and readable. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: add Vercel observability links to benchmark PR comments - Store runId in benchmark timing data - Add project-slug to Vercel benchmark matrix - Pass WORKFLOW_VERCEL_PROJECT_SLUG env var to benchmarks - Store Vercel metadata (teamSlug, projectSlug, environment) in timing files - Generate observability deep links for each Vercel world benchmark - Show observability links below Production (Vercel) tables 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: use correct Vercel project slugs for observability links - nextjs-turbopack → example-nextjs-workflow-turbopack - nitro-v3 → workbench-nitro-workflow 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>	2025-12-06 18:30:25 -08:00
Pranay Prakash	5dd15452cd	Test and benchmark community worlds against e2e tests (#482) ... * Add new github workflow * Enable pull_request trigger for community worlds workflow 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add community worlds manifest and generation scripts - Add community-worlds.json manifest as single source of truth - Add scripts/generate-community-worlds-workflow.mjs to generate CI workflow - Add scripts/generate-community-worlds-docs.mjs to generate docs section - Update aggregate-benchmarks.js to load community worlds dynamically - Add pnpm generate:community-worlds script - Update docs/deploying/world/index.mdx with community worlds The manifest-based approach allows: - E2E tests to be auto-generated from the manifest - Benchmark aggregation to include community worlds - Docs to stay in sync with tested worlds 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix YAML syntax error - quote strings starting with @ The @ symbol has special meaning in YAML, so package names like @workflow-worlds/turso need to be quoted. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix Redis health-cmd quoting, remove unpublished starter world - Quote health-cmd when it contains spaces (fixes Docker arg parsing) - Remove @workflow-worlds/starter as it's not published to npm 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add benchmarks and summary job for community worlds - Add build job to share artifacts between benchmark jobs - Add benchmark jobs for Turso, MongoDB, and Redis worlds - Update summary job to show both E2E and benchmark status matrix - Add left border/indent to sidebar child items for visual hierarchy - Update workflow generator to support benchmark generation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Reuse build artifacts for E2E tests E2E jobs now depend on the shared build job and download artifacts instead of rebuilding packages from scratch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add Worlds Ecosystem dashboard to docs - Create worlds-manifest.json with official and community worlds - Add aggregate-worlds-data.mjs script for processing E2E and benchmark results - Create WorldsDashboard, WorldCard, and BenchmarkChart components - Add /docs/worlds page showing compatibility status and performance - Include sample data for development The dashboard shows: - E2E test progress per world (pass/fail/skip counts) - Benchmark performance comparison across all worlds - Filter by official vs community worlds 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix CI to output JSON test results and add Jazz world - Update workflow generator to output JSON test results from vitest - Upload E2E results as artifacts for parsing in summary job - Summary job now shows actual pass/fail/skip counts per world - Add Jazz world to worlds-manifest.json (requires external credentials) - Add update-worlds-status.yml workflow to auto-update dashboard data - Update TypeScript types to support null lastRun and metrics 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Refactor community worlds to use reusable workflows Instead of a generated workflow file, integrate community world testing directly into tests.yml and benchmarks.yml using reusable workflows. - Add reusable workflows for E2E tests: e2e-community-world.yml (no services), e2e-community-world-mongodb.yml, e2e-community-world-redis.yml - Add reusable workflows for benchmarks: benchmark-community-world.yml, benchmark-community-world-mongodb.yml, benchmark-community-world-redis.yml - Update tests.yml to call reusable workflows for Turso, MongoDB, Redis - Update benchmarks.yml to include community world benchmarks in summary - Delete generated community-worlds.yml and generator script This approach: - Inherits proper Rust/SWC setup from the main workflows - Keeps all CI in the established patterns - Makes adding new community worlds straightforward 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix benchmark timing file naming for community worlds Add WORKFLOW_BENCH_BACKEND env var support to bench.bench.ts so community world benchmarks generate timing files with the correct backend suffix (e.g., bench-timings-nextjs-turbopack-turso.json instead of -local.json). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add @workflow-worlds/starter to community worlds test matrix 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Unify worlds manifest and add dynamic GitHub API fetching - Merge community-worlds.json into worlds-manifest.json with type field - Add server-side data fetching from GitHub API for worlds dashboard - Remove static worlds-status.json, fetch CI artifacts dynamically - Update all references to use unified manifest format - Remove obsolete community-worlds.yml and update-worlds-status.yml workflows 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Skip community worlds for non-nextjs-turbopack in benchmark summary Community worlds only run against nextjs-turbopack, so hide the "missing" rows for Express and Nitro frameworks in the benchmark comparison tables. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix stream benchmark detection to check for actual TTFB data The previous check `!== null` incorrectly returned true for undefined, causing all benchmarks to show TTFB columns. Now explicitly checks for a number type. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Hide Worlds Ecosystem page from sidebar The page is still accessible via direct link at /docs/worlds but won't appear in the navigation until it's been further iterated on. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix review comments: trailing newline and division by zero - Add trailing newline when replacing Community Worlds section in docs - Fix division by zero in WorldCard benchmark calculation when metrics is empty 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Consolidate community world workflows with service-type parameter - Create setup-workflow-dev composite action for common setup steps - Add service-type input to benchmark-community-world.yml and e2e-community-world.yml - Use conditional job execution (if: inputs.service-type == 'mongodb') to handle different services - Update benchmarks.yml and tests.yml to pass service-type parameter - Delete redundant workflow files: - benchmark-community-world-mongodb.yml - benchmark-community-world-redis.yml - e2e-community-world-mongodb.yml - e2e-community-world-redis.yml Reduces workflow files from 11 to 7 and eliminates ~500 lines of duplicated YAML. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Generate community world test matrix from worlds-manifest.json Replace hardcoded community world jobs with dynamic matrix generation using scripts/create-community-worlds-matrix.mjs. This allows adding/removing community worlds by editing the manifest instead of multiple workflow files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Apply suggestion from @vercel[bot] Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> * Add Samples column and separate local/production benchmarks - Add Samples column to all benchmark tables showing iteration count - Separate benchmark results into Local Development and Production sections - Add explanatory context for each section (localhost vs Vercel deployment) - Add GitHub action step summaries to e2e community world tests - Create aggregate-e2e-results.js script for parsing vitest JSON output 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Remove obsolete generate-community-worlds-docs script The Worlds Ecosystem page now fetches from worlds-manifest.json at runtime, making this script unnecessary. The npm script also referenced a non-existent workflow generator script. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Maximize composite action usage and reorganize benchmark output - Update setup-workflow-dev composite action with optional Rust, install-dependencies, and install-args inputs - Update tests.yml to use composite action in unit, e2e-vercel-prod, getTestMatrix, e2e-local-*, and getCommunityWorldsMatrix jobs - Update benchmarks.yml to use composite action in build, benchmark-local, benchmark-postgres, benchmark-vercel, and getCommunityWorldsMatrix jobs - Reorganize benchmark output to group by benchmark test with local/production tables within each benchmark - Remove invalid $schema reference from worlds-manifest.json 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add beads stealth mode stuff (for personal claude memory - will remove stealth if people want) * Add E2E test results PR comment summary - Add pr-comment-start job to create/update PR comment when tests start - Add artifact uploads to all e2e test jobs (vercel-prod, local-dev, local-prod, local-postgres, windows) - Update e2e-community-world.yml with consistent artifact naming (e2e-community-*) - Add summary job to aggregate all e2e results and update PR comment - Extend aggregate-e2e-results.js with --mode aggregate for multi-job PR summary - Group results by category (Vercel Production, Local Development, etc.) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add step summaries to all e2e test jobs Add "Generate E2E summary" step to each individual e2e job: - e2e-vercel-prod - e2e-local-dev - e2e-local-prod - e2e-local-postgres - e2e-windows Each job now outputs pass/fail/skip counts to GITHUB_STEP_SUMMARY. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Consolidate community world workflows to single job Replace 3 mutually exclusive jobs (e2e/e2e-mongodb/e2e-redis) with a single job that starts services via docker run when needed. This eliminates the skipped job entries that appear in the GitHub Actions UI. - Use conditional docker run steps instead of services: block - Add health check loops to wait for service readiness - Add cleanup step to stop containers 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix extractWorldId to handle community world artifact naming Add handling for `e2e-results-community-{world}` pattern so community world test results are properly extracted (e.g., `e2e-results-community-turso` now correctly extracts `turso` instead of `community-turso`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>	2025-12-02 15:12:44 -08:00