Disable `experimentalRawTransfer` in `oxc-parser` NAPI package on
unsupported platforms.
Tested working on:
* NodeJS v22.0.0
* NodeJS v22.14.0
* NodeJS v23.0.0
* NodeJS v23.9.0
* Deno v2.0.0
* Deno v2.2.3
Not working on:
* NodeJS v20.0.0
* NodeJS v20.18.1
* Bun v1.2.4
The reason NodeJS v20 and Bun are unsupported is because they cannot
create an `ArrayBuffer` larger than 4 GiB. We may be able to relax that
requirement in future in order to support those platforms.
Duplicate of #9555, but just stacked on top of #9566.
`napi-rs` for some reason omits the `export * from '@oxc-project/types';` line from `index.d.ts` when building for WASM.
I don't know how to fix that, so added a hacky script to fix up `index.d.ts` manually. This is not at all ideal, but at least it now compiles and CI passes.
Prevent compilation errors on 32-bit platforms by avoiding calculating `usize: 1 << 32` unless on a 64-bit platform.
`parseSync` will throw an error if called with `experimentalRawTransfer` on an unsupported system.
Add a function `rawTransferSupported` to check if raw transfer is supported on current system or not, so user can avoid these errors.
Similar to #9546. Acorn outputs `RegExpLiteral`'s flags in same order as in source text. There is nothing wrong with this, but it makes our serialization/deserialization code more complex than it needs to be, just to match Acorn. I don't think we need to follow Acorn here - there's no particular "good" order for the flags.
Bump `acorn-test262` to include commit https://github.com/oxc-project/acorn-test262/commit/3d7013e145499268dc8f985d087208aa013e0ba2, which reorders the flags in alphabetical order in Acorn's AST.
This allows simplifying serialization/deserialization code on our side.
Add benchmarks for NAPI parser. Benchmarks cover both "standard" mode and "raw transfer" mode.
Codspeed say they've sorted out the problems with wide variance in NodeJS benchmarks, so hopefully they won't wave all over the place, the way they did last year when we first tried to add these benchmarks.
What we also found last year is that Codspeed's measures didn't reflect real-world wallclock measurements at all when comparing code running on Rust side vs JS side. Raw transfer moves a lot of work from Rust side to JS side, and so likely the bench results will be misleading if comparing "standard" vs "raw".
I've added benchmarks for both, not to compare the 2, but because I think there's room to improve the performance of both "standard" and "raw" independently.
Speed up the tests for raw transfer by serializing output to JSON, and comparing as string to the `acorn-test262` JSON snapshots. This is faster than running `parseSync` twice with "raw" and "standard" settings, and comparing the 2 - time to run the NAPI parser tests is reduced by half.
There are still 5 tests which compare 5 sample JS/TS files (same ones as used in benchmarks) with a full raw-vs-standard comparison. These tests cover the other parts of `parseSync`'s return value - `comments`, `module` and `errors`. They also give some limited test coverage for AST of TS files.
Add an `astType` option to `parseSync` and `parseAsync`.
When set to `'js'`, the returned AST does not include TypeScript-related properties. When set to `'ts'`, it does.
If not specified, the option defaults to same as source type. So when parsing JS / JSX files, the AST returned does not include TS properties by default.
The motivation for allowing the user to override that behavior, is in case someone is parsing a mix of JS and TS files, and want all the ASTs in the same format. In that case they'd set `astType: 'ts'` for all files.
This is a breaking change as it alters what properties appear in the AST on JS side.
First version of raw transfer (#2409).
Provides a solid speed-up transferring data from Rust to JS. Further iterations will speed it up further.
Tests check that output via raw transfer matches output via JSON transfer exactly for all of Test262 which Acorn is able to parse. It should also match for TypeScript and JSX, but that's not covered by tests as yet.
However, I think we should consider this experimental for now because there are a few rough edges (discussed below).
Therefore I've put it behind an "experimental" flag:
```js
const ret = parseSync(filename, code, { experimentalRawTransfer: true });
console.log(ret.program);
```
### How it works
* JS creates an `ArrayBuffer` and passes it to Rust.
* Rust creates an `Allocator` using that buffer as its backing memory.
* Rust parses the AST into that allocator (including comments).
* Rust also converts module record and errors into arena types, and writes them into the allocator.
* Rust writes into the end of the buffer the offset at which data begins in the buffer.
* Control passes back to JS.
* JS code decodes data in the buffer, and creates JS objects fitting the ESTree shape.
There is *no* serialization step on Rust side. There is no JSON encoding or decoding involved. These are the main sources of the speed gain.
### Preconditions
The reason all this works is that all AST types (and other transferred types) are `#[repr(C)]`. So the memory layouts of those types are specified, and can be statically calculated.
`oxc_ast_tools` does those layout calculations, and generates the JS-side deserializer code based on its knowledge of what offsets struct fields are at, what the discriminants of enums are etc.
### Rough edges
There are a few rough edges.
#### `Allocator::from_raw_parts`
This PR adds a method `Allocator::from_raw_parts`. It creates a `bumpalo::Bump` that uses an existing block of memory as the allocator's backing memory. This ability is required, but it's not an API that `bumpalo` offers (and the maintainer was not willing to add it).
The implementation of `Allocator::from_raw_parts` is extremely hacky. It depends on internal implementation details of `bumpalo` which are not specified. And the method used to determine the memory layout of `Bump` depends on unspecified details of Rust's memory model. So, while it does seem to work in practice, it is, strictly speaking, UB.
It could break in a future Rust version, or with esoteric compiler flags e.g. [`-Zrandomize-layout`](https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/randomize-layout.html). And it could break if we update `bumpalo` from the version we're currently using. For this reason, this PR pins the version of `bumpalo` in `Cargo.toml`.
I think this OK for now, but it's unpleasantly fragile.
We can resolve all these problems by replacing `bumpalo` with our own arena allocator (which I think we should do anyway, for other reasons).
In the meantime, `Allocator::from_raw_parts` is behind a cargo feature `from_raw_parts`, to avoid it being used anywhere else in our codebase.
#### Unspecified type layouts
As noted above, all AST types are `#[repr(C)]`, so their layout is specified and stable. There are a few types which are outside of our control, though:
1. `Vec<T>`. We use `allocator_api2::vec::Vec`, which is not `#[repr(C)]`.
2. `&str`. I don't believe the layout of this type is specified.
3. `Option<T>`.
For production-grade stability, we need to try to work around these.
`Vec` - we should replace `allocator_api2::vec::Vec` with our own `Vec` type. This will also allow us to reduce its size (https://github.com/oxc-project/backlog/issues/18).
`&str` - again, we need our own string slice type, to work around the problem of lone surrogates (#3526) and to make it more efficient (https://github.com/oxc-project/backlog/issues/46). We can make its memory layout stable at the same time.
`Option<T>` is tricky. We don't want to replace Rust's `Option` because of the niche optimization benefits it gives. I'm not sure this one is 100% soluable, but Rust gives at least *some* guarantees about the layout of `Option`. Maybe we can avoid using `Option` in the AST in ways which go outside that specification.
#### Large buffers
For speed, raw transfer requires the entire AST to be in a single contiguous memory region, and for the start of that region to be aligned on 4 GiB.
JS does not support 64-bit integers, so offset calculations are much cheaper when the buffer is aligned on 4 GiB and no larger than 4 GiB - because then all pointers have the same value in their top 32 bits. So the pointer can be treated as a 32-bit value (bottom 32 bits only). JS can handle 32 bit integers no problem.
When creating a large buffer on JS side, it *mostly* ends up aligned on a 4 GiB boundary anyway, but occasionally it doesn't. So in order to ensure the buffer has at least 1 region within it which is aligned on 4 GiB, and 2 GiB in size, we have to create a 6 GiB buffer.
I *think* this is OK. On systems with virtual memory, allocating 6 GiB only reserves 6 GiB of *virtual* memory. Physical memory is only consumed when the pages of that allocation are actually written to.
But I may be missing something here, and memory exhaustion might be a danger. I think we need some real-world usage to find out.
*Possibly* we could reduce the need for so much memory if JS deserializer called into a small WASM module to do offset calculations. WASM can work with `i64` values. Or there may be other solutions.
#### Endianness
Currently only little-endian systems are supported. Probably in practice this doesn't matter much, but it'd be ideal to cover big-endian too.
Speed up UTF-8 to UTF-16 `Span` conversion by processing span offsets in ascending order.
This involves, for each node, processing `span.start`, then visiting all children and processing their `Span`s, and then finally processing `span.end`.
When the AST has come direct from the parser, this means all offsets are processed 100% in ascending order (and the visitor used here has manually-written visitation methods for the few types where that's not always the case otherwise e.g. `export {x}`).
If the AST *has* been modified, code may have moved around, so ascending order won't necessarily be preserved, but still it *mostly* will.
Optimize for this visitation order. Instead of doing a binary search through the whole UTF8-UTF16 translation table on every single span offset, assume the current offset is in same region as the last. Only resort to binary search as a de-opt when that's not the case. Most JS/TS files don't contain many non-ASCII characters, so these cases will be rare.
Pure refactor. Rename:
* `Program::to_json` to `to_estree_ts_json`.
* `Program::to_pretty_json` to `to_pretty_estree_ts_json`.
This is preparatory work for #9285, which adds `*_js_*` variants of these 2 methods, which omit TypeScript fields.
Also rename the serializers, to include `TS` in their names, for the same reason.
hi-ogawa