mirror of
https://github.com/vercel/workflow.git
synced 2026-09-14 19:59:43 +08:00
Nathan Rajlich
f8f6e17aeb
* Add opt-in QuickJS WASM VM engine (WORKFLOW_VM=quickjs) with full event replay * QuickJS engine: AbortController, setAttributes, terminal drain, turbo-safe requeue, stable PRNG seed * QuickJS engine: hook.getConflict support, cross-run writable forwarding symbols * QuickJS engine: stream framing round-trip, bound step proxies, webhook fidelity * Apply biome fixes to QuickJS engine files * Address review feedback: anchor source-map strip to end-of-input, use getWorkflowQueueName for conflict requeue, Buffer-free asset decoding, function replacers for payload injection, maxEventsLimit guard * CI: include generated QuickJS source assets in shared e2e build artifacts * Fix same-token hook ordering and conflicted-hook disposal in the QuickJS engine * CI: run both VM engines across all frameworks and worlds; label jobs with the engine * Fix stack overflow stripping inline source maps from webpack dev bundles; harden step-listing e2e assertions against eventually-consistent reads * e2e: poll step listings until analytics rows include attempt (optional column can lag terminal status) * e2e: use --withData to force storage-backed step listings for attempt assertions (analytics listing can omit attempt entirely) * Sort imports in QuickJS serialization files (biome organizeImports) * QuickJS engine: resolve the run's full payload-key capability so sealed (encp) hook payloads open Main's sealed-box work (#3096) makes cross-deployment resumeHook() seal hook payloads to the target run's published X25519 public key. The shared start() path publishes that key regardless of engine, so QuickJS runs receive sealed payloads too — but the QuickJS entrypoint resolved only the bare symmetric key via importKey(), which cannot open encp envelopes. The first sealed hook payload wedged the run right after hook_received, timing out every hook/webhook e2e on Vercel prod (node:vm legs were fine — the node engine resolves the full capability via memoizeEncryptionKey). Resolve deriveRunPayloadKeys() in the entrypoint instead and widen the runtime's key types from CryptoKey to DecryptionKey. Writes stay symmetric (encrypt() with RunPayloadKeys takes the encr path). Regression test seals a payload exactly as resumeHook does and round-trips it through the VM. * Address review: crypto/process parity, loud Intl guards, lazy engine import, VM-leak guard, telemetry namespace, eval-string escaping - Deterministic crypto.getRandomValues/randomUUID in the VM bootstrap, drawing from the seeded Math.random (identical sequences to the node engine's vm/index.ts implementations); all crypto.subtle methods throw with step-function guidance. process.env exposed as a frozen copy, matching node. - Intl: throwing constructors (no ICU in QuickJS), and toLocale*-family methods (incl. localeCompare) throw when given an explicit locale so cross-engine divergence is loud instead of silently writing different values into the event log. No-argument forms keep working. - runtime.ts lazy-imports the QuickJS entrypoint at dispatch, keeping the ~1.3MB embedded WASM assets out of node-engine deployments. - runQuickJSWorkflow wraps the per-run phase so an exceptional exit disposes the VM instead of leaking it in a reused compute instance; corrected the misleading fail-loud comment (run_failed, not retry); warn when the event drain loop exhausts its iteration bound. - Telemetry attributes renamed quickjs.* → workflow.vm.* to stay in the file's workflow.* namespace. - Eval-string correlation-id interpolation uses JSON.stringify instead of quote-only escaping. - common-vm.test.ts pins the reducer/reviver superset invariant against common.ts so the duplicated sets can't silently drift. - Docs enumerate the remaining global-surface differences (subtle.digest, Intl, WebAssembly, Atomics); quickjs-entrypoint documents the known precondition-guard gap. * QuickJS engine: implement resilient resumeHook (hookInput materialization + resumeId dedup) #1834 made resumeHook() fall back to enqueueing the run with a hookInput payload when the direct hook_received write fails transiently, with the runtime materializing the missing event on delivery. Only the node:vm path implemented it — the QuickJS dispatch returned before the node block, so the resilient payload was silently dropped and the new e2e timed out on every quickjs leg. - runtime.ts threads hookInput into runWorkflowWithQuickJS; the entrypoint materializes the missing hook_received after loading the event log (resumeId-keyed dedup, occurredAt from the resumeId ULID, local eventData substitution for lazy/ref responses, EntityConflict / HookNotFound handling) — mirroring the node block. - processEvents drops duplicate hook_received rows sharing a resumeId (first-in-log wins), matching the node engine's EventsConsumer dedup; the seen-set lives in the VM heap so it is deterministic per replay. Verified against the dev server with WORKFLOW_VM=quickjs: the resilient resume e2e passes and the materialization is observable in the logs; all 27 hook e2e tests green. * Rerun CI * QuickJS engine: split VM-local class/step-function reducers off the hardened host codec The hardened host-side serialization (#3257) made the shared reducers/class.ts and reducers/step-function.ts depend on serialization/hardened.ts, which imports node:util and captures host intrinsics — unbundleable and meaningless inside the QuickJS guest, where the codec already runs in the guest realm. Point the VM codec at pre-hardening copies with identical wire format; the host/guest boundary hardening for this engine arrives with the host-side serde that retires the VM bundle. * QuickJS engine: enqueue explicit wait continuations instead of same-message redelivery Scheduling sleep wakeups by returning { timeoutSeconds } redelivers the CURRENT queue message. When that message is a hook-resume delivery (carrying hookInput), its redelivery re-runs the lazy-resume re-ensure in the handler prologue; if the workflow disposed the hook during the first delivery (dispose -> sleep), the re-ensure gets HookNotFound, the prologue acks the message as 'nothing left to resume', and the wait timer it carried is silently lost — the run wedges (caught by the hookDisposeTestWorkflow e2e). Enqueue fresh continuation messages instead, matching the node engine's suspension handler: getWaitContinuationDispatch for pending waits (gaining delay clamping/hop chaining and pending-wait dedup keys) and a plain immediate message for elapsed-wait / attr_set / getConflict requeues. A fresh message carries only runId, so its delivery always reaches replay. Also: read hook_received resumeId from the canonical top-level event field (eventData.resumeId is the deprecated legacy fallback), and stop passing hookInput into the entrypoint — the shared prologue in runtime.ts materializes the event for both engines. Adds a VM replay test for the hook -> dispose -> sleep shape. * Sort imports in quickjs-entrypoint (biome organizeImports) * Address review: dispatch inside run-level try/catch, queue namespace + run-origin trace carrier threading, configurable interrupt budget - Move the QuickJS engine dispatch inside the replay loop's try so escaping engine failures (MaxEventsExceededError, WASM OOM, bundle-eval errors) reach the catch that classifies and records run_failed, instead of nacking the message and burning all 48 queue redeliveries into MAX_DELIVERIES_EXCEEDED. Transient world errors still rethrow for redelivery. Updated the two comments that describe the propagation. - Thread the queue namespace from runtime.ts through runWorkflowWithQuickJS into every message publish (step dispatch, hook_conflict requeue, immediate requeue, wait continuation) — without it, publishes on a namespaced deployment land on __wkf_workflow_* while consumers listen on __<ns>_wkf_workflow_*. - Thread the run-origin nextTraceCarrier accessor through instead of capturing the current invocation context, so linked-mode invocations form a star around workflow.start rather than chaining; the hook_conflict requeue now carries a traceCarrier and requestedAt. - Replace the hardcoded 30s VM interrupt budget with the configurable replay budget (getReplayTimeoutMs, default 240s), matching the node engine. * Sort imports in quickjs-runtime (biome organizeImports)
184 lines
5.5 KiB
JavaScript
184 lines
5.5 KiB
JavaScript
// Framework-specific dev test configurations
|
|
const DEV_TEST_CONFIGS = {
|
|
'nextjs-turbopack': {
|
|
generatedStepRegistrationPath:
|
|
'app/.well-known/workflow/v1/flow/__step_registrations.js',
|
|
generatedWorkflowPath: 'app/.well-known/workflow/v1/flow/route.js',
|
|
apiFilePath: 'app/api/chat/route.ts',
|
|
apiFileImportPath: '../../..',
|
|
},
|
|
'nextjs-webpack': {
|
|
generatedStepRegistrationPath:
|
|
'app/.well-known/workflow/v1/flow/__step_registrations.js',
|
|
generatedWorkflowPath: 'app/.well-known/workflow/v1/flow/route.js',
|
|
apiFilePath: 'app/api/chat/route.ts',
|
|
apiFileImportPath: '../../..',
|
|
},
|
|
nitro: {
|
|
generatedStepRegistrationPath: 'node_modules/.nitro/workflow/steps.mjs',
|
|
generatedWorkflowPath: 'node_modules/.nitro/workflow/workflows.mjs',
|
|
apiFilePath: 'routes/api/chat.post.ts',
|
|
apiFileImportPath: '../..',
|
|
},
|
|
nuxt: {
|
|
generatedStepRegistrationPath: '.nuxt/workflow/steps.mjs',
|
|
generatedWorkflowPath: '.nuxt/workflow/workflows.mjs',
|
|
apiFilePath: 'server/api/chat.post.ts',
|
|
apiFileImportPath: '../..',
|
|
},
|
|
sveltekit: {
|
|
generatedStepRegistrationPath:
|
|
'src/routes/.well-known/workflow/v1/flow/__step_registrations.js',
|
|
generatedWorkflowPath: 'src/routes/.well-known/workflow/v1/flow/+server.js',
|
|
apiFilePath: 'src/routes/api/chat/+server.ts',
|
|
apiFileImportPath: '../../../..',
|
|
workflowsDir: 'src/workflows',
|
|
},
|
|
vite: {
|
|
generatedStepRegistrationPath: 'node_modules/.nitro/workflow/steps.mjs',
|
|
generatedWorkflowPath: 'node_modules/.nitro/workflow/workflows.mjs',
|
|
apiFilePath: 'routes/api/chat.post.ts',
|
|
apiFileImportPath: '../..',
|
|
},
|
|
hono: {
|
|
generatedStepRegistrationPath: 'node_modules/.nitro/workflow/steps.mjs',
|
|
generatedWorkflowPath: 'node_modules/.nitro/workflow/workflows.mjs',
|
|
apiFilePath: './src/index.ts',
|
|
apiFileImportPath: '..',
|
|
},
|
|
express: {
|
|
generatedStepRegistrationPath: 'node_modules/.nitro/workflow/steps.mjs',
|
|
generatedWorkflowPath: 'node_modules/.nitro/workflow/workflows.mjs',
|
|
apiFilePath: './src/index.ts',
|
|
apiFileImportPath: '..',
|
|
},
|
|
fastify: {
|
|
generatedStepRegistrationPath: 'node_modules/.nitro/workflow/steps.mjs',
|
|
generatedWorkflowPath: 'node_modules/.nitro/workflow/workflows.mjs',
|
|
apiFilePath: './src/index.ts',
|
|
apiFileImportPath: '..',
|
|
},
|
|
nest: {
|
|
generatedStepRegistrationPath: '.nestjs/workflow/steps.mjs',
|
|
generatedWorkflowPath: '.nestjs/workflow/workflows.mjs',
|
|
apiFilePath: './src/app.controller.ts',
|
|
apiFileImportPath: '..',
|
|
workflowsDir: 'src/workflows',
|
|
},
|
|
astro: {
|
|
generatedStepRegistrationPath:
|
|
'src/pages/.well-known/workflow/v1/__step_registrations.js',
|
|
generatedWorkflowPath: 'src/pages/.well-known/workflow/v1/flow.js',
|
|
apiFilePath: 'src/pages/api/chat.ts',
|
|
apiFileImportPath: '../..',
|
|
workflowsDir: 'src/workflows',
|
|
},
|
|
'tanstack-start': {
|
|
generatedStepRegistrationPath: 'node_modules/.nitro/workflow/steps.mjs',
|
|
generatedWorkflowPath: 'node_modules/.nitro/workflow/workflows.mjs',
|
|
apiFilePath: 'src/routes/api/chat.ts',
|
|
apiFileImportPath: '../../..',
|
|
},
|
|
};
|
|
|
|
function createMatrixEntry(name, project, config, overrides = {}) {
|
|
const canary = overrides.canary === true;
|
|
|
|
return {
|
|
name,
|
|
project,
|
|
...config,
|
|
runLabel: canary ? 'canary' : 'stable',
|
|
artifactSuffix: canary ? 'canary' : 'stable',
|
|
...overrides,
|
|
};
|
|
}
|
|
|
|
const matrix = {
|
|
app: [],
|
|
};
|
|
|
|
for (const app of [
|
|
{
|
|
name: 'nextjs-turbopack',
|
|
project: 'example-nextjs-workflow-turbopack',
|
|
},
|
|
{
|
|
name: 'nextjs-webpack',
|
|
project: 'example-nextjs-workflow-webpack',
|
|
},
|
|
]) {
|
|
matrix.app.push(
|
|
createMatrixEntry(app.name, app.project, DEV_TEST_CONFIGS[app.name])
|
|
);
|
|
matrix.app.push(
|
|
createMatrixEntry(app.name, app.project, DEV_TEST_CONFIGS[app.name], {
|
|
canary: true,
|
|
})
|
|
);
|
|
}
|
|
|
|
matrix.app.push(
|
|
createMatrixEntry('nitro', 'workbench-nitro-workflow', DEV_TEST_CONFIGS.nitro)
|
|
);
|
|
matrix.app.push(
|
|
createMatrixEntry(
|
|
'sveltekit',
|
|
'workbench-sveltekit-workflow',
|
|
DEV_TEST_CONFIGS.sveltekit
|
|
)
|
|
);
|
|
matrix.app.push(
|
|
createMatrixEntry('nuxt', 'workbench-nuxt-workflow', DEV_TEST_CONFIGS.nuxt)
|
|
);
|
|
matrix.app.push(
|
|
createMatrixEntry('hono', 'workbench-hono-workflow', DEV_TEST_CONFIGS.hono)
|
|
);
|
|
matrix.app.push(
|
|
createMatrixEntry('vite', 'workbench-vite-workflow', DEV_TEST_CONFIGS.vite)
|
|
);
|
|
matrix.app.push(
|
|
createMatrixEntry(
|
|
'express',
|
|
'workbench-express-workflow',
|
|
DEV_TEST_CONFIGS.express
|
|
)
|
|
);
|
|
matrix.app.push(
|
|
createMatrixEntry(
|
|
'fastify',
|
|
'workbench-fastify-workflow',
|
|
DEV_TEST_CONFIGS.fastify
|
|
)
|
|
);
|
|
matrix.app.push(
|
|
createMatrixEntry('nest', 'workbench-nest-workflow', DEV_TEST_CONFIGS.nest)
|
|
);
|
|
matrix.app.push(
|
|
createMatrixEntry('astro', 'workbench-astro-workflow', DEV_TEST_CONFIGS.astro)
|
|
);
|
|
|
|
matrix.app.push({
|
|
name: 'tanstack-start',
|
|
project: 'workbench-tanstack-start-workflow',
|
|
...DEV_TEST_CONFIGS['tanstack-start'],
|
|
});
|
|
|
|
// Cross-product with the workflow VM engine axis: every app is tested
|
|
// against both the default node:vm engine and the opt-in QuickJS WASM
|
|
// engine (WORKFLOW_VM=quickjs). Each engine gets its own artifactSuffix
|
|
// and runLabel so CI artifacts and job names are unique. The `vm` field
|
|
// is surfaced to the workflow dev server via the WORKFLOW_VM env var in
|
|
// tests.yml.
|
|
const VMS = ['node', 'quickjs'];
|
|
matrix.app = matrix.app.flatMap((app) =>
|
|
VMS.map((vm) => ({
|
|
...app,
|
|
vm,
|
|
runLabel: [app.runLabel, vm].filter(Boolean).join(' '),
|
|
artifactSuffix: [app.artifactSuffix, vm].filter(Boolean).join('-'),
|
|
}))
|
|
);
|
|
|
|
console.log(JSON.stringify(matrix));
|