Built using @sokra's Fleet! This re-arranges all of our chunking
experimental features under `experimental.turbopackChunking` and adds
the following options:
```javascript
/**
* Avoid creating more than one chunk smaller than this size, in bytes. Smaller
* chunks are merged into bigger ones to avoid that. Defaults to `50000` (50 KB).
*/
minChunkSize?: number
/**
* Avoid creating more than this number of chunks per chunk group. Chunks are
* merged into bigger ones to avoid that. Defaults to `40`.
*/
maxChunkCountPerGroup?: number
/**
* Never merge chunks bigger than this size, in bytes, with other chunks. This keeps code
* in big chunks from being duplicated across multiple chunks. Defaults to `200000` (200 KB).
*/
maxMergeChunkSize?: number
/**
* Minimum size, in bytes, for a component chunk to be emitted on its own when
* `generateComponentChunks` is enabled. Component chunks smaller than this are folded into a
* single remainder chunk. Defaults to `20000` (20 KB).
*/
minComponentChunkSize?: number
```
Currently, importing a new module causes a complete eviction (`clear()`)
and re-evaluation of server chunks, both in the Turbopack runtime's
module cache and Node's `require.cache`.
Here's what happens today:
- A new module is imported into a chunk's graph
- This changes its `availability_info`, which in dev for non-entry
chunks is encoded into the chunk's filepath
- `VersionedContentMap` works entirely on chunk paths. When we construct
instructions to transition a chunk into its new state, we fail to find
the prior state and fall back to `clear()` as described above.
An ideal version of this is a refactor that implements
`VersionedContentMap` on a per-module basis, not a per-chunk one. This
PR doesn't do that, but achieves consistent module-level updates when a
chunk's availability info changes.
It does this by implementing chunk lists for server entry chunks the
same way the client hmr implementation does: an entry chunk's version
aggregates the versions of its dependent chunks (including dynamically
imported ones), keyed by merger rather than path. Entry chunks do not
encode availability info in their paths, so they are insulated from
missing versions. Updates are applied in Node.js through the same shared
merged-update machinery in the unified hmr runtime that the client
already uses, plus a new `ChunkListUpdate` branch in the server hmr
client to unwrap the merged updates.
Details:
- The merged-update wire format (`EcmascriptMergedUpdate` etc.) moves
from turbopack-browser into shared `turbopack_ecmascript::chunk_list`,
along with runtime-agnostic `ChunkListVersion` and `update_chunk_list`
implementations.
- `turbopack-nodejs` gains a chunk content merger mirroring the browser
one. `EcmascriptBuildNodeEntryChunk`'s versioned content is now a
chunk-list content over its sync and async chunks. It is version-only
and never emitted; the entry chunk still inlines its loader calls.
- The aggregate server hmr subscription now tracks only entry chunks.
Shared chunks under `server/chunks/` ride the entry's `ChunkListUpdate`
as module deltas, so their content-hash paths no longer matter.
- The Node hmr client applies `ChunkListUpdate` by feeding each nested
merged update through the shared apply path. Modules that appear in an
"added" chunk but already exist in the module cache were moved by a
chunk rename and are treated as modified.
- On a successful partial apply, the hot reloader clears the manifest
cache for updated chunks and notifies browsers to refetch RSC, without
clearing `require.cache`.
### Test Plan
Adds a series of e2e tests to catch the described manual test plan in
both entry and dynamic chunks.
This moves modules out of `turbopack-browser` and into
`turbopack-ecmascript` so it can be used in the server hmr
implementation as well.
Move the chunk-list version, update, and merged-update wire types out of
`turbopack-browser` into a shared `turbopack-ecmascript::chunk_list`
module so both the browser and node chunking contexts can build on them.
Rename `EcmascriptDevChunkListVersion` to `ChunkListVersion` and make
`update_chunk_list` runtime-agnostic (plain path->`VersionedContent`
maps).
Pure refactor: no behavior change.
<!-- NEXT_JS_LLM_PR -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
This replaces per-chunk Server HMR turbo tasks with a single firehose
subscription that diffs every HMR chunk. This significantly cuts the
number of tokio task churn on projects with many server chunks and
centralizes the diff/clear logic. It leads to a multi-second saving in
both cold and warm builds in a large app.
This PR also renames `clear()` to `reEvaluateAllModulesExpensive()` to
label directly that this is a costly operation and should only be done
in exceptional circumstances.
A following PR will bring this to client chunks.
Rust:
- New `aggregate_hmr` module: `AggregateHmrVersion` keyed by chunk path,
`merged_partial_update` builder, and `is_hmr_eligible_chunk` (excludes
`.map` files, which would force every diff to `Total`).
- `Project::all_hmr_version_state` / `all_hmr_update` aggregate over the
whole `hmr_root_path`. The seed transition emits an empty `Partial` so
the JS consumer doesn't treat it as a restart and wipe handlers the
triggering request just populated. Any chunk requiring `Total`/`Missing`
escalates the batch.
- `VersionedContentMap::hmr_chunks_in_path` lists eligible chunks with
their `VersionedContent`.
NAPI:
- `projectAllHmrEvents(target)` returns a single subscription.
JS:
- `setupServerHmr` subscribes once via `allHmrEvents` instead of fanning
out over `hmrChunkNamesSubscribe`.
- `reEvaluateAllModulesExpensive()` evicts every chunk under
`server/chunks/` from `require.cache` directly rather than tracking
subscriptions. This is a bit fragile as it relies on the path prefix and
scanning require.cache.
The previous `@mswjs/interceptors` has a bug where the response it
forwarded never set the [`complete`
field](https://nodejs.org/api/http.html#messagecomplete). This wasn't an
issue so far (or not reported) but is an issue for `httpxy` which relies
on an accurate `complete` field.
Bumping `@mswjs/interceptor` now so that we can switch from `http-proxy`
to `httpxy`.
## Summary
Add Vite-compatible `caseSensitive` support to `import.meta.glob()`.
Matching remains case-sensitive by default, while `caseSensitive: false`
applies case-insensitive matching to directory traversal, filenames,
positive patterns, and negative patterns.
The option is included in matcher equality and non-default virtual
module identity, and is exposed through diagnostics, TypeScript
declarations, and the Turbopack options reference.
## Verification
- `cargo test -p turbo-tasks-fs glob`
- `cargo nextest run -p turbopack-tests -E "test(import_meta_glob) and
not test(import_meta_glob_errors)"`
- `cargo fmt --all -- --check`
- `pnpm build-all` and `pnpm --filter=next types` are blocked by
current-canary `sharp.default` type errors in
`packages/next/src/server/image-optimizer.ts`
<!-- NEXT_JS_LLM -->
I was seeing failures on https://github.com/vercel/next.js/pull/95628
and went down this rabbit hole.
- We were (incorrectly) not canonicalizing the `DiskFileSystem` root
dir. This happened to work out for us because pnpm also wasn't
canonicalizing junction point targets, and so the root dir prefix
matched. However after `#95628`, pnpm started canonicalizing junction
point targets because we added `pnpm-workspace.yaml` files. This was
just coincidence and indicative of much deeper problems.
- `dunce` isn't a good fit for us. It picks a win32 or verbatim
representation based on path length, which would mean that long paths
inside of a short root path would never work because we'd never be able
to strip the prefix, since we'd be looking for a win32 path prefix.
Omnipath is better for what we want to do:
https://docs.rs/omnipath/latest/omnipath/windows/trait.WinPathExt.html
- The windows verbatim path format works with long paths (>260
characters) and behaves a lot closer to cross-platform unix paths, which
is what we want. E.g. if somebody tries to import a JS module from a
Windows 8.3 short path, it should fail. This will fail with verbatim
paths (good!). This PR *always* uses the verbatim path format internally
within `DiskFileSystem`.
- `read_link` was re-inventing `try_from_sys_path` but badly. Just call
`try_from_sys_path`.
- `turbopack/crates/turbo-tasks-fs/src/embed/file.rs` was dead code.
- `validate_path_length_inner` was incorrectly referring to verbatim
paths as UNC paths. These are two different things.
- Canonicalize any absolute symlink targets outside of the
`DiskFileSystem` root path, resolving any symlinks they may have,
normalizing case-insensitive paths, and handling Windows 8.3 short name
format.
## Some Context About Windows Path Formats:
- win32 paths: These are your normal user-friendly `C:\foo\blah` paths.
When calling most filesystem APIs, these are limited to 260 characters
unless the user has enabled long paths on the system, and avoiding the
limit with these paths also requires a [`longPathAware` application
manifest](https://learn.microsoft.com/en-us/windows/win32/sbscs/application-manifests#longPathAware).
You can mix `/` and `\` in win32 paths, and Windows will normalize the
path separator for you.
- verbatim paths: This is a rust-specific name (used by the stdlib) for
the "extended length" prefixed paths. These are paths starting with
`\\?\`. These paths can be up to ~32,767 characters long. They do not
support Windows 8.3 shortnames. They do not normalize between `/` and
`\`, and do not support `.` or `..` components, however
[`Path::join`](https://doc.rust-lang.org/std/path/struct.Path.html#method.join)
will do this normalization if given a verbatim path.
- Windows 8.3 shortnames: A legacy format where paths can be represented
by 8 characters, followed by a tilde and a digit (or hash in some
extreme cases where there are many collisions), followed by a 3
character file extension. NTFS supports this,
[ReFS](https://learn.microsoft.com/en-us/windows-server/storage/refs/refs-overview)
(used by [Windows Dev
Drive](https://learn.microsoft.com/en-us/windows/dev-drive/)) does not.
- UNC ("Universal Naming Convention"): This is a specific representation
of paths used for remote files on network servers.
- `file://` URIs: These must be formed from [win32 style
paths](https://docs.rs/url/latest/url/struct.Url.html#method.from_file_path),
though the normal 260 character limit typically does not apply here.
- canonicalize: Corresponds to
[`std::fs::canonicalize`](https://doc.rust-lang.org/stable/std/fs/fn.canonicalize.html).
This involves actual fs operations (that we can't easily track), and
recursively resolves symlinks. It will fail if the file path does not
exist. On windows, this always returns a verbatim path.
- wide format: Windows paths are utf-16. Generally we don't support
paths that can't be encoded to utf-8, but we should try to fail as
obviously as we can in these cases. We don't have good filesystem issue
handling yet though.
- case sensitivity: Generally paths are not case-sensitive on Windows or
macOS, but are case-sensitive on most Linux filesystems. The exact
definition of how case sensitivity works with unicode paths is highly
platform and filesystem dependent.
- junction points: The windows equivalent of symlinks (symlinks are also
supported, but require developer mode to be turned on). These can
contain an arbitrary path reference in any format (win32, verbatim, etc)
with any case and with windows 8.3 short names. Junction points are
always target absolute paths to directories.
This has been causing a lot of confusion when contributors have been
debugging tree shaking, eg.
https://github.com/vercel/next.js/issues/95698 and the linked issues.
Turbopack's standard tree-shaking features are controlled through the
`turbopackRemoveUnusedExports` and `turbopackRemoveUnusedImports`
options which default to true.
This PR builds on top of the other PRs in this stack, it is the PR that
switches to inlining the bootstrap code and dropping the runtime file
that each route had.
#95815 refactored the NFT flow and ended up only tracing the RSC
template and it's subgraph. Not the server actions loader module whose
module graph is completely separate.
---
Clauded.
Attempts to fix
https://github.com/vercel/next.js/discussions/95130#discussioncomment-17652155.
---
<details>
## Summary
Fixes the NFT tracing regression behind #95815 and the deploy failures
reported in
https://github.com/vercel/next.js/issues/87737#issuecomment-4897366831.
Since the 16.3 NFT rewrite (#94224 / #92901), a `serverExternalPackages`
entry that is imported **only from a `'use server'` action referenced by
a client component** ends up in the route's trace as the content-hashed
alias symlink (`.next/node_modules/<pkg>-<hash>`) **without the store
files the symlink points to**. Any output assembled from the traces
(Vercel lambdas, `output: 'standalone'`) then contains a dangling
symlink, and invoking the action throws `Failed to load external module
<pkg>-<hash>` (HTTP 500). 16.2.9 traces both. Both CJS and ESM externals
are affected, with any package manager — the ESM/bun/monorepo framing in
the issue reports was a coincidence of which packages were only
reachable through actions.
The cause: `AppEndpoint::trace_result` started the trace DFS only from
`rsc_entry`, but the server actions loader is a separate module-graph
entry (`additional_entries`). It is chunked into the endpoint output
(which emits the alias symlink via the chunk's output assets), yet its
subgraph — including the externals' traced target files — was never
visited by the tracer. `trace_endpoint` now accepts multiple entry
modules and the app endpoint passes the actions loader alongside
`rsc_entry`.
The test observes the user-visible failure: it builds with `output:
'standalone'`, deletes everything except the standalone output (so only
traced files are available, like a deployed lambda), runs `server.js`,
and invokes the server action in a browser. Without the fix the action
fails with `Failed to load external module lodash-<hash>`; on 16.2.9 the
same fixture responds with the action result. Note the fixture must not
import the externals from any other route: the standalone output is the
union of all route traces, so a route that traces them correctly would
mask the missing entries.
## Verification
- `pnpm test-start-turbo
test/production/standalone-mode/server-action-externals/standalone-mode-server-action-externals.test.ts`
— fails without the fix (`Failed to load external module
lodash-fb78ad0536ffcdab` from the standalone server), passes with it.
The equivalent standalone flow on `next@16.2.9` returns the action
result with HTTP 200.
- `pnpm test-start-turbo
test/e2e/externals-transitive/externals-transitive.test.ts`
- `pnpm test-start-turbo
test/production/standalone-mode/server-actions/standalone-mode-server-actions.test.ts`
<!-- NEXT_JS_LLM -->
</details>
Fixes#95125 in two ways
The OP admits that they misread the message in a few ways, but we can
fix all of them
1. Be specific about what function might need an annotation in the error
message. A user with a nearby `path.join` might think that that is where
the annotation belongs instead of on the function with the warning.
2. Be more flexible with how these `ignore` comments are interpreted
`fs.readFileSync(path.join(process.cwd(), someVar))` shouldn't require 2
annotations. Handle this by checking for and detecting `turbopackIgnore`
comments in all expressions! This will then bubble up through the linker
and just cause us to ignore corresponding fs access apis.
I'm introducing a new endpoint that doesn't compile SSR when doing a
page navigation, I'd like to call it a `rscEndpoint` but currently this
name is taken. `rscEndpoint` appears to be used for detecting changes
for HMR so I've renamed it to `rscHmrEndpoint` so I can use that name
for the new `rscEndpoint`.
This computes the implementation hash of a given `use cache`:
- it includes the module containing the `use cache` function, plus all transitive imports of that module (including externals via NFT)
- what does it hash
- for bundled code, it hashes the generated code (so including any AST transforms, inlined env vars, etc)
- for NFTd files, it hashes the file content on disk
- So this is basically equivalent to hashing the generated JS chunks, except that it's scoped to only what the given actions module uses
- (so notably, if you have multiple use cache functions in a given file, then they will all have the same hash)
<br/>
- [x] Gate behind experimental flag
- [x] Only compute hash for `use cache` functions, not all server actions
- [ ] `final_read_hint` is a problem. Some chunk items are now codgened twice, so it recomputes the AST
- [x] disable for dev for now
- [ ] Measure perf impact
```jsonc
// .next/server/server-reference-manifest.json
{
"node": {
"806f4954cfbb75404a19d6d405065ed9059cc0cab2": {
"workers": {
"app/rsc/page": {
"moduleId": "[project]/bench/app-router-server/.next-internal/server/app/rsc/page/actions.js { ACTIONS_MODULE0 => \"[project]/bench/app-router-server/app/rsc/logic.js [app-rsc] (ecmascript)\" } [app-rsc] (server actions loader, ecmascript)",
"async": false,
"exportedName": "$$RSC_SERVER_CACHE_0",
"filename": "bench/app-router-server/app/rsc/logic.js",
"codeHash": "413e394d9597b35df12828a6a7fd6363" // <------------
}
},
"filename": "bench/app-router-server/app/rsc/logic.js",
"exportedName": "$$RSC_SERVER_CACHE_0"
}
},
"edge": {},
"encryptionKey": "jOF2TbpNLjp2oOhl3VPBwGE7luodnei9clJPq/gaYQo="
}
```
This takes `chunkingHeuristics` from `next.config.js` and puts it on
`ChunkGroupEntry` so it can be on `ChunkGroupInfo` which will then make
it available in `make_production_chunks`
### What
Turbopack's `server-reference-manifest` emitter stores `filename` and
`exportedName` **twice**: once hoisted to the action-entry level
(correct), **and** redundantly inside *every* per-page
`ActionManifestWorkerEntry`. This PR removes the per-worker copies so
the worker map only carries `{ moduleId, async }` — matching the webpack
backend, which already emits them at the entry level only.
The emitter even comments the intent (`// Hoist the filename and
exported_name to the entry level`) and does the hoist — the per-worker
fields are vestigial.
```rust
// crates/next-api/src/server_actions.rs
entry.workers.insert(&key, ActionManifestWorkerEntry {
module_id: loader_id.clone(),
is_async: …,
exported_name, // ← removed (already on entry)
filename: filename.as_ref(), // ← removed (already on entry)
});
entry.exported_name = exported_name; // hoist (kept)
entry.filename = filename.as_ref(); // hoist (kept)
```
### Why it matters
`filename` and `exportedName` are invariant per action, but the worker
map has **one entry per page that can reach the action**, so these
strings get repeated `O(actions × pages)` times. On a large App Router
app (`output: 'standalone'`, `cacheComponents: true`) this is the
dominant cost of the manifest:
- ~1,650 routes, **1,847 references → 589,349 `workers` entries**
- `server-reference-manifest.json` = **170 MB** / `.js` = **182 MB**
(plus the duplicate copy under `.next/standalone`, plus ~314 MB of
per-route manifests)
- A single `'use cache'`/action reachable from a shared client is
registered on **1,638 pages**, each repeating the full `filename` string
Dropping the two redundant per-worker fields removes the largest
repeated strings from every worker entry, shrinking the manifest
substantially with no information loss.
### Why it's safe (consumer-compatible)
- The webpack backend already emits worker entries as `{ moduleId, async
}` only, with `filename`/`exportedName` at the entry level — so this
aligns Turbopack with webpack, not introduces a new shape.
- The only first-party reader,
`packages/next/src/server/mcp/tools/get-server-action-by-id.ts`, reads
`entry.filename` / `entry.exportedName` (entry level) and types
`workers` as `Record<string, any>` — it never reads the per-worker
copies.
### Changes
- `crates/next-core/src/next_manifests/mod.rs` — drop `exported_name` +
`filename` from `ActionManifestWorkerEntry` (keep `module_id`,
`is_async`).
- `crates/next-api/src/server_actions.rs` — stop assigning them in the
worker insert (entry-level hoist unchanged).
### Notes
- Verified by reading the emitter, the manifest types, and the consumer;
I don't have a local Turbopack build, so CI is the source of truth here
— if any manifest **snapshot fixtures** include per-worker
`filename`/`exportedName`, they'll need regenerating.
- Context / measurements: vercel/next.js#95241.
This is the big change in the `next-` part of the repo to support this.
The code goes through the module graph to find any service workers that
need compiling and then compiles them. It also has the code changes to
`packages/next/src/server/lib/router-utils/filesystem.ts` that serve
these files on their respective routes.
`next` already disallows reading out of `.next` during a build, this
extends that to the new `.next-profiles` directory
This is currently hardcoded (not end user configurable), so we just
hardcode the path in the next entrypoints.
The sourcemap file of the polyfill chunk wasn't actually content hashed.
it just used the name of the corresponding JS chunk.
It was still emitted into `_next/static/immutable`. So it wasn't
actually immutable.
Now, it's content hashed (and this also removed the ad-hoc sourcemap
construction from app.rs and instead uses the usual pattern that all
other chunks use to reference their corresponding sourcemap chunks).
We didn't catch this because it only happens with
`nextConfig.productionBrowserSourceMaps: true` which isn't the default.
The test now covers this as well
`EcmascriptParsable::parse_original` is used to access the original
module when using 'split modules', the only place that needed this was
already checking for split modules for other reasons so we can simply
remove this function from the trait and access the split module
'original' directly.
Also remove EcmascriptParseable::ty which was dead
https://github.com/vercel/next.js/pull/94197 had a bug in the Turbopack
case, where it didn't properly trace all necessary files referenced from
`cacheHandler` and `cacheHandlers`.
If you imported `lodash` in the cache handler, then
- `node_modules/.pnpm/lodash@4.18.1/node_modules/lodash/lodash.js` was
correctly traced
but missing were
- `node_modules/.pnpm/lodash@4.18.1/node_modules/lodash/package.json`
and
- `node_modules/lodash` (the symlink)
The problem was that it wasn't including affecting sources of these
entrypoints, because it was treating those entrypoints as bundled:
```
SingleModuleGraph::new_inner(
entries: Vec<ChunkGroupEntry>,
visited_modules: &FxIndexMap<ResolvedVc<Box<dyn Module>>, GraphNodeIndex>,
include_traced: bool,
include_binding_usage: bool,
entries_are_traced: bool, // <------ problematic
) -> Result<Vc<Self>>
```
should really have been
```
SingleModuleGraph::new_inner(
entries: &GraphEntries, // contains Vec<ChunkGroupEntry> and traced_modules: Vec<Modules
visited_modules: &FxIndexMap<ResolvedVc<Box<dyn Module>>, GraphNodeIndex>,
include_traced: bool,
include_binding_usage: bool,
) -> Result<Vc<Self>> {
```
because the single `entries_are_traced: bool` can be different per
entry.
The filtering logic is generally very cheap, many issues are
short-lived, and filtering is unlikely to trigger more than 1-2 turbo
tasks per issue.
Wrapping it in another task per issue doesn't make a ton of sense.
# What
Replace the `derive` macros for TaskInput with a proc macro
`[turbo_tasks::task_input]` (and analogous support inside of the
`turbo_tasks::value` macro.
Introduce a hand written `CloneResolved` future type for TaskInput
implementations that are always resolved.
# Why
This allows us to take advantage of the `NonLocalValue` marker trait. If
a type is `NonLocalValue` then `TaskInput` can have trivial
`is_resolved` and `resolve_input` implementations. This applies to the
vast majority of TaskInputs but the current `derive` implementation
always produces a recursive `async` `resolve_input` implementation but
it is rarely needed and rustc/llvm do a bad job of optimizing it.
## Code-size impact
`next-swc.darwin-arm64.node` — release profile, `pnpm swc-build-native
--release`, macOS aarch64.
| Metric | canary | this branch | Δ bytes | Δ % |
| --- | ---: | ---: | ---: | ---: |
| Raw `.node` | 124,494,592 | 124,217,280 | −277,312 | −0.22% |
| Stripped (`strip -x`) | 84,294,000 | 84,013,136 | −280,864 | −0.33% |
| Stripped + gzip -9 | 29,123,538 | 28,898,920 | −224,618 | −0.77% |
## Performance
A similar measurement of performance was neutral
| Metric | canary | temporary | Δ | % |
|---|---|---|---|---|
| wall time (s) | 42.659 ± 1.851 | 42.501 ± 2.028 | −0.158 | −0.4% |
| user time (s) | 294.322 ± 1.834 | 293.207 ± 2.070 | −1.115 | −0.4% |
| sys time (s) | 74.158 ± 4.197 | 73.543 ± 5.725 | −0.615 | −0.8% |
| MaxRSS (MB) | 13650.1 ± 66.0 | 13610.1 ± 57.5 | −40.0 | −0.3% |
All deltas are smaller than one standard deviation — statistically
indistinguishable from noise on this benchmark, as expected for a small
optimization.
### What?
Add focused production adapter coverage for configured `cacheHandler`
and `cacheHandlers` module graphs in Node `output.assets`, including
transitive helper dependencies and Turbopack `assetsHashes` assertions.
### Why?
Adapter deployments dynamically load configured cache handlers at
runtime. This coverage demonstrates that Node adapter outputs currently
omit those handler files and their transitive dependencies.
### How?
Add an App Router Cache Components fixture using `adapterPath`, both
cache-handler APIs, and a `use cache: remote` route. Its adapter writes
`onBuildComplete` context to disk and the test inspects Node outputs for
each handler and uniquely named helper.
### Verification
- `NEXT_TEST_PREFER_OFFLINE=1 pnpm --filter=next build`
- Expected failure: `NEXT_TEST_PREFER_OFFLINE=1 pnpm test-start-turbo
test/production/app-dir/adapter-cache-handlers/adapter-cache-handlers.test.ts`
(no Node output contains all four handler assets)
- Expected failure: `NEXT_TEST_PREFER_OFFLINE=1 pnpm test-start-webpack
test/production/app-dir/adapter-cache-handlers/adapter-cache-handlers.test.ts`
(same adapter handoff gap)
- `git diff canary --check`
- Not run continuously: `pnpm --filter=next dev` (`EMFILE: too many open
files` occurred after startup in the local watcher)
<!-- NEXT_JS_LLM_PR -->
---------
Co-authored-by: Niklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
Recreation of https://github.com/vercel/next.js/pull/90314
---
## Summary
- Fix `--debug-build-paths` pages normalization in Turbopack so
`pages/<name>/index.tsx` maps to `/<name>` (matching JS
`getPageFromPath()` behavior).
- Handle index page edge case: `/index.tsx -> /`.
- Keep regression coverage in
`test/production/debug-build-path/debug-build-paths.test.ts` with a
generic fixture `pages/with-index/index.tsx`.
## Current Behavior Reference
- CI run (before this fix) that failed as expected for this bug:
-
https://github.com/vercel/next.js/actions/runs/22266469801/job/64413677099?pr=90314#step:35:216
## Root Cause
Rust-side debug-build-path normalization removed file extensions but did
not remove `/index`, producing route keys like `/with-index/index` while
pages route keys are `/with-index`. That mismatch filtered out the page
and caused `PageNotFoundError` during page-data collection.
---------
Co-authored-by: devjiwonchoi <devjiwonchoi@gmail.com>
Refactor the module-graph bit of NFT so that we can provide
`Endpoint.traced_files()` which is needed for the bundle analyzer
There should be no logic change whatsoever
Previously, we created chunk list register chunks for every reachable
chunk in the chunk graph on a page. Now, we only create one that
subscribes to all recursively reachable assets.
This has to pass around an explicit list of client references chunks as
those cannot be discovered via the chunk graph alone.
This results in a significant performance improvement when loading pages
with the dev server, improving performance of a 60s cold build in a
large app by about 10s.
- [x] skip traced edges properly in all graph traversals
- [x] iter_nodes and enumerate_nodes include the traced modules
- [x] Don't propagate is_self_async inside of traced references as a perf optimization
- [x] fix performance
- [x] validate that NFT JSON files are identical
- [x] ~~validate/adjust bundle analyzer output~~ will be in followup
- [x] audit `fn chunking_type()` changes
Benchmark:
```
* ced85b3108 - (2 weeks ago) refactor - Niklas Mischkulnig (HEAD -> mischnic/nft-module-graph, origin/mischnic/nft-module-graph)
TURBOPACK_PERSISTENT_CACHE=0 NEXT_TURBOPACK_TRACING= pnpm next build 334.96s user 44.57s system 788% cpu 48.152 total
TURBOPACK_PERSISTENT_CACHE=0 NEXT_TURBOPACK_TRACING= pnpm next build 343.39s user 42.61s system 809% cpu 47.659 total
* 5edf48238f - (77 minutes ago) v16.3.0-canary.32 - next-js-bot[bot] (HEAD, tag: v16.3.0-canary.32)
TURBOPACK_PERSISTENT_CACHE=0 NEXT_TURBOPACK_TRACING= pnpm next build 333.42s user 43.57s system 712% cpu 52.877 total
TURBOPACK_PERSISTENT_CACHE=0 NEXT_TURBOPACK_TRACING= pnpm next build 332.95s user 43.15s system 791% cpu 47.515 total
TURBOPACK_PERSISTENT_CACHE=0 NEXT_TURBOPACK_TRACING= pnpm next build 337.59s user 41.28s system 771% cpu 49.117 total
TURBOPACK_PERSISTENT_CACHE=0 NEXT_TURBOPACK_TRACING= pnpm next build 335.25s user 45.53s system 718% cpu 52.975 total
```
<details>
<summary>Previous benchmarks</summary>
```
52212418699f362904d64c08e544d4994e8d27fa mischnic/nft-module-graph
pnpm next build 371.41s user 66.39s system 737% cpu 59.353 total
pnpm next build 380.13s user 59.62s system 751% cpu 58.548 total
pnpm next build 374.25s user 60.55s system 761% cpu 57.102 total
8e4cfc5062 canary
pnpm next build 370.77s user 53.89s system 762% cpu 55.725 total
pnpm next build 383.16s user 55.61s system 744% cpu 58.921 total
pnpm next build 382.34s user 53.51s system 757% cpu 57.512 total
pnpm next build 381.15s user 54.50s system 751% cpu 57.943 total
```
```
b0d5411a79 mischnic/nft-module-graph
pnpm next build --experimental-build-mode=compil 368.75s user 62.05s system 749% cpu 57.445 total
pnpm next build --experimental-build-mode=compil 368.95s user 58.42s system 763% cpu 56.004 total
0e3ad2768bd3b036e96f66e4b770a16bb0b38705 mischnic/nft-module-graph
pnpm next build --experimental-build-mode=compil 369.48s user 60.62s system 748% cpu 57.494 total
pnpm next build --experimental-build-mode=compil 374.89s user 52.27s system 730% cpu 58.449 total
pnpm next build --experimental-build-mode=compil 375.02s user 52.89s system 747% cpu 57.274 total
f37073e1d9 mischnic/speed-next-dynamic-map
pnpm next build --experimental-build-mode=compil 389.58s user 55.98s system 704% cpu 1:03.21 total
e599b11e2d6b09a92600cfca6d9b338ed1f7537d mischnic/fix-side-effect-edge-removal
pnpm next build --experimental-build-mode=compil 386.60s user 53.34s system 720% cpu 1:01.09 total
f68f70b044 canary
pnpm next build --experimental-build-mode=compil 387.46s user 53.72s system 738% cpu 59.716 total
pnpm next build --experimental-build-mode=compil 390.33s user 53.54s system 734% cpu 1:00.43 total
pnpm next build --experimental-build-mode=compil 385.33s user 56.55s system 722% cpu 1:01.18 total
pnpm next build --experimental-build-mode=compil 386.49s user 56.44s system 726% cpu 1:00.96 total
```
```
b1ed4553e6 origin/mischnic/nft-module-graph
pnpm next build --experimental-build-mode=compile 467.28s user 75.24s system 791% cpu 1:08.56 total
pnpm next build --experimental-build-mode=compile 465.00s user 76.80s system 831% cpu 1:05.20 total
pnpm next build --experimental-build-mode=compile 473.86s user 70.73s system 790% cpu 1:08.90 total
3cb77c844b canary
pnpm next build --experimental-build-mode=compile 367.91s user 58.06s system 726% cpu 58.672 total
pnpm next build --experimental-build-mode=compile 362.97s user 57.68s system 746% cpu 56.379 total
pnpm next build --experimental-build-mode=compile 364.59s user 57.01s system 724% cpu 58.167 total
```
</details>
### What?
Threads the layer module graph through `ServerActionsGraph` and
`map_server_actions` as an `OperationVc` instead of a resolved `Vc`,
calling `connect()` at each consumer.
### Why?
Fixes the invalidation priority of `map_server_actions` so it is
executed after the module graph on incremental builds. Previously,
because the graph was resolved at construction time, the dependency
ordering caused `map_server_actions` to be re-run before the module
graph on incremental rebuilds.
### How?
By passing the layer graph as an `OperationVc` and calling `connect()`
within each consumer task, the graph is resolved within the consuming
task rather than at construction time. This restores the intended
invalidation/scheduling order between the module graph and
`map_server_actions`.
<!-- NEXT_JS_LLM_PR -->
### What?
Replaces the unsound by-value `IntoIterator` impl for `ReadRef<T>` in
`turbo-tasks` with a sound clone-free variant, and adapts the callers
across `turbopack-*` and `next-api`.
### Why?
The previous impl used `transmute_copy` to fabricate `&'static`-typed
items so it could expose them through the standard `Iterator` trait.
Those references were only really valid as long as the `ReadRef` inside
the iterator stayed alive — but `Iterator::Item` is a fixed associated
type, so once the items were stashed in futures, `Vec`s, or `serde_json`
map keys, the lifetime was completely unenforced.
This produced a latent use-after-free whenever something else
(turbo-tasks cell eviction, an intermediate `Drop`, etc.) released the
underlying storage between the iteration site and the next dereference.
The observed symptom was a panic in `RcStr::as_str` during JSON
serialization of `AssetHashesManifestAsset`'s manifest:
```
thread 'tokio-rt-worker' panicked at turbopack/crates/turbo-rcstr/src/lib.rs:132:52:
range end index 13 out of range for slice of length 7
```
The byte read at the inline-length position was junk left over from
freed/reused memory — `len = 13` is unreachable for any
legitimately-constructed inline `RcStr` (max inline length is 7 on
64-bit). The bug site was
`crates/next-api/src/project_asset_hashes_manifest.rs`, which consumed
an `OutputAssetsWithPaths` `ReadRef`, kept `&RcStr` references in
`asset_paths` past the `try_join` that dropped the iterator, then
serialized them.
### How?
**`turbopack/crates/turbo-tasks/src/read_ref.rs`** — new by-value impl:
```rust
pub struct ReadRefIter<T, I, J>
where
T: VcValueType,
I: Copy + 'static,
J: Iterator<Item = &'static I>,
{
iter: J,
_read_ref: ReadRef<T>,
}
impl<T, I, J> Iterator for ReadRefIter<T, I, J> /* … */ {
type Item = I;
fn next(&mut self) -> Option<I> { self.iter.next().copied() }
}
impl<T, I, J> IntoIterator for ReadRef<T>
where
T: VcValueType,
I: Copy + 'static,
J: Iterator<Item = &'static I> + 'static,
&'static VcReadTarget<T>: IntoIterator<Item = &'static I, IntoIter = J>,
{
type Item = I;
type IntoIter = ReadRefIter<T, I, J>;
fn into_iter(self) -> Self::IntoIter {
let r: &VcReadTarget<T> = &self;
// SAFETY: the fabricated `&'static` reference is only stored inside
// `iter`, which lives inside the returned `ReadRefIter` alongside
// the `ReadRef` that owns the data. `next()` only ever yields
// `Copy`-ed-out values — no reference (with the fake `'static`
// lifetime or otherwise) ever leaves the iterator. Struct-field drop
// order (`iter` then `_read_ref`) drops the borrow before the
// backing storage.
let r = unsafe { std::mem::transmute::<&VcReadTarget<T>, &'static VcReadTarget<T>>(r) };
ReadRefIter { iter: r.into_iter(), _read_ref: self }
}
}
```
Key properties:
- **No cloning.** Setup is one borrow + `transmute`; `next()` is
`Option::copied()` (bitwise copy via the `Copy` bound), not
`Clone::clone`. Nothing in the iterator clones the backing collection or
its elements.
- **Contained `unsafe`.** The fake `'static` reference never leaves
`ReadRefIter`. `Iterator::next` yields `I` by value, so the lifetime
never escapes into futures, `Vec`s, or other persistence outside the
iterator.
- **Drop order safe.** Struct fields drop in declaration order: `iter`
(and any borrows it holds) drops before `_read_ref` (the backing `Arc`).
- **`Copy` bound.** The impl is restricted to element types that are
`Copy` — `ResolvedVc<_>`, integer ids, owned-tuple-of-`Copy`, etc. For
non-`Copy` element types (`RcStr`, `FileSystemPath`, `PatternMatch`,
`(String, _)`, `(ModuleId, ReadRef<_>)`, …) callers iterate by reference
via the existing `IntoIterator for &'a ReadRef<T>` impl (`for x in
&read_ref` or `read_ref.iter()`).
The original buggy site in `project_asset_hashes_manifest.rs` now uses
`output_assets.iter()` and keeps `&'a RcStr` references in the manifest
struct. The borrow checker now enforces the lifetime that used to be
faked via `transmute` — `output_assets` outlives the references because
nothing consumes it, and there are no clones at the call site either.
**Caller adjustments.** Touching the impl forced a sweep of all call
sites that were implicitly leaning on the unsound shape (yielding
`&'static`-typed items as a stand-in for owned items). The fixes fall
into a small number of categories:
- Drop redundant `.copied()` / `.cloned()` / `|&x| f(x)` patterns after
`into_iter()` (items are owned `Copy` values now, no need to
deref-and-copy).
- Switch non-`Copy` element iteration to `&read_ref` / `read_ref.iter()`
(e.g. `PatternMatches`, `CodeAndIds`, `UnresolvedUrlReferences`,
`GraphEntries`, `Vec<RcStr>`).
- Reshape `crates/next-api/src/paths.rs` helpers from `impl
IntoIterator<Item = &ResolvedVc<_>>` to `impl IntoIterator<Item =
ResolvedVc<_>>` — `ResolvedVc` is `Copy`, so by-value is the natural
shape and it composes directly with the new by-value
`ReadRef::into_iter`. Callers in `app.rs`, `pages.rs`, `middleware.rs`,
`instrumentation.rs`, `font.rs` updated to match (either passing the
`ReadRef`/`Vec` directly, or `.iter().copied()` for borrowed sources).
- A few small follow-ups: `for (key, EndpointGroup { primary, .. }) in
&entrypoint_groups` in `routes_hashes_manifest.rs` (with a borrowed `&'l
str` key in the manifest); `compute_async_module_info_single(graph,
result)` (no `*graph`, it's already `Copy`); `&(ty, batch)` → `(ty,
batch)` destructures in `chunking/mod.rs`.
### Testing
- `cac` clean across the workspace.
- `ca clippy --all-targets` clean.
- `ca test -p turbo-tasks-backend` — all unit + integration tests pass.
- `ca test -p turbopack-tests --tests` — execution snapshot suite (218
passed, 0 failed, 1 ignored) and snapshot suite (87 passed, 0 failed).
Closes NEXT-
Fixes #
We want to use this to potentially improve efficiency in e2e tests where
we're running many instances of Turbopack at once. By default,
turbo-tasks will try to use every available CPU core.
**This environment variable does not strictly limit Turbopack to using 4
cores, it just restricts the number of main tokio worker threads and
changes how we shard dashmaps.**
This option will not make sense for the vast majority of users, it only
makes sense when you are running *many* instances of Turbopack at once,
and the exact semantics are potentially confusing, as it is not a hard
limit.
## Test Plan
Build vercel-site with this set to 1 and see a little over 100% CPU
usage. Set it to 8 and see around 800% CPU usage.
```
TURBO_TASKS_AVAILABLE_PARALLELISM=1 pnpm next build --experimental-build-mode=compile
```
Luke Sandberg