`FixedSizeAllocator`s allocate a block of memory larger than what the `Allocator` controls, and often with a different start pointer. So when `FixedSizeAllocator` is dropped, it needs to deallocate that block of memory using the original pointer.
That pointer way previously stored in `FixedSizeAllocator` itself. Instead store it in a `FixedSizeAllocatorMetadata` struct, which is written into the memory block itself, after the section controlled by the `Allocator`, and other metadata (`RawTransferMetadata`).
This allows getting the original `alloc_ptr` when you only have access to the `Allocator`, but not the `FixedSizeAllocator`.
Correct comment for `get_buffer_offset` (used by raw transfer) about buffer size, and add an additional rounding down. The latter makes no difference because the offset is truncated to `u32` straight after anyway, but better to code defensively in case we change buffer size / alignment later on.
`FixedSizeAllocator` create an inner `Allocator` 16 bytes smaller than the buffer size. The last 16 bytes will be used to store `RawTransferMetadata` (offset of the AST data etc). This prepares `FixedSizeAllocator` for use with raw transfer for JS linter plugins.
Reduce the size of the buffer used for raw transfer by 16 bytes so that there's 16 bytes space for `malloc`'s metadata without committing another memory page. I don't know for sure that `malloc` would store metadata in the allocation for allocations this large, but this is what Bumpalo does, so it's probably a good idea.
Pure refactor. Introduce a `RawTransferMetadata` struct to hold metadata that gets written to end of buffer. Codegen use knowledge of memory layout of this struct to generate constants added in #12268. This is more robust than manually calculating offsets.
Similar to #12268. Instead of hard-coding the size and alignment of raw transfer buffers in multiple places, codegen add these to generated files, and everywhere else import the values from those files. This is more robust.
Reduce size of `oxc-parser` WASM binary by omitting code for raw transfer. Raw transfer is not supported on WASM.
This also allows removing the workaround to prevent compilation failure on 32-bit systems due to `1 << 32` overflowing `usize`. This code is no longer compiled on 32-bit systems.
In ESTree serializer, reduce repeated code by introducing a `serialize_span` method which outputs `start` and `end`, and also `range` if the `range` option is enabled.
Related to #10307.
Re-arrange fields in ESTree AST, so `start` and `end` are last, instead of first.
Also bump `acorn-test262` submodule to include https://github.com/oxc-project/acorn-test262/pull/54, which rearranges the fields in test snapshots.
Part of #10307. Add `range` option to `oxc-parser`, to add `range` property to nodes in ESTree AST.
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: overlookmotel <theoverlookmotel@gmail.com>
Add `codeframe` property to errors returned by `parseSync` / `parseAsync` when using raw transfer. This matches the shape of error objects without raw transfer.
Previously, only `parseSync` supported raw transfer. Extend support to `parseAsync` too.
This requires:
1. A queue to limit the number of files being parsed simultaneously, to avoid the possibility of memory exhaustion.
2. A more complicated mechanism for recycling buffers, since now multiple buffers can be in use at the same time.
All our AST structs are `#[repr(C)]`, which gives us guaranteed stable memory layouts. This is essential for raw transfer, and helpful for other purposes too.
However, the downside is that we lose out on the automatic field-reordering that default `#[repr(Rust)]` provides, where structs are packed to minimize padding bytes.
This PR implements a similar field-reordering scheme, to get the same effect.
* `oxc_ast_tools` determines optimal field order for compact memory layout.
* `oxc_ast_tools` generates a file (`structs.rs`) which details the desired field order for each type.
* `#[ast]` proc macro follows the instructions in `structs.rs` and re-orders the fields.
This reduces the size of various AST types by 8 bytes. No type now has more than 7 padding bytes (previously e.g. `Function` had 15 bytes of padding).
Possible future improvements:
1. Get Rust Analyser to display original field order.
2. Remove `syn` dependency from `oxc_ast_macros` crate, to improve compile times. It should be possible to do all the AST manipulation using just `proc_macro` crate, by generating more code in `oxc_ast_tools`, so the macro's logic can be simpler.
Note: This PR doesn't add `syn` dependency, it was already there. But it makes it easier to remove it.
Related:
* #6488
* #9706
Finished first step(32 -> 24) of #9706.
Change `len` and `cap` fields from `usize` to `u32`, so that the `Vec` size can reduced size to `24`. Also, to avoid some unnecessary unit conversion. I changed many `RawVec` methods to take `u32` rather than `usize`.
The performance looks great now! And we reduced 1%-3% memory usage in the parser.
Add comments to `assert_layouts.rs` files stating the number of padding bytes each time contains. This is useful when trying to reduce size of AST types.
closes#10980
Previously final source type was only computed when `lang` is not set.
It is changed to:
* compute source type from `lang`, use filename extension if `lang` is
not set
* reset the computed source type with 'script` or `module` if provided
Add an `#[estree(prepend_to = ...)]` attribute to accompany `#[estree(append_to = ...)]`.
* `#[estree(prepend_to = target)]` prepends the field to *start* of the target field.
* `#[estree(append_to = target)]` appends the field to *end* of the target field.
This also allow both prepending *and* appending to the same field. e.g. combining `TSThisParameter`, `FormalParameter`s, and `BindingRestElement` into a single array.
Pure refactor. Replace `AppendTo` and `AppendToConcat` which are used in ESTree serializer to concatenate 2 fields with a more generalized `Concat2`, which performs both functions.
#10791 fixed the performance of `oxc-parser` NPM package by removing the reviver from `JSON.parse` call. However, it replaced it with a complete traversal of the AST on JS side to locate `Literal` nodes and update them.
Instead, generate a list of paths to `Literal` nodes needing fixing on Rust side in `ESTree` serializer.
e.g. for this program:
```js
123n;
foo(/xyz/);
```
the fix paths are:
```json
[
["body", 0, "expression"],
["body", 1, "expression", "arguments", 2]
]
```
Having the location of nodes which need updating reduces work on JS side, as no need to search the entire AST.
Running `pnpm run bench` (in `napi/parser`) locally shows between 8% and 15% speed-up from this change.
In ESTree (plain ESTree for JS i.e. Acorn), a hashbang is recorded as a line comment. Follow that behavior in our ESTree AST.
In TS-ESTree, hashbangs are ignored. That's not how it looks in AST explorer, because there's some post-processing going on, but `@typescript-eslint/parser` itself ignores hashbangs. So only add the comment in ESTree AST (not TS-ESTree AST).
More info on this rather confusing subtlety: https://github.com/typescript-eslint/typescript-eslint/issues/6500
Prevent compilation errors on 32-bit platforms by avoiding calculating `usize: 1 << 32` unless on a 64-bit platform.
`parseSync` will throw an error if called with `experimentalRawTransfer` on an unsupported system.
Add a function `rawTransferSupported` to check if raw transfer is supported on current system or not, so user can avoid these errors.
Add an `astType` option to `parseSync` and `parseAsync`.
When set to `'js'`, the returned AST does not include TypeScript-related properties. When set to `'ts'`, it does.
If not specified, the option defaults to same as source type. So when parsing JS / JSX files, the AST returned does not include TS properties by default.
The motivation for allowing the user to override that behavior, is in case someone is parsing a mix of JS and TS files, and want all the ASTs in the same format. In that case they'd set `astType: 'ts'` for all files.
This is a breaking change as it alters what properties appear in the AST on JS side.
First version of raw transfer (#2409).
Provides a solid speed-up transferring data from Rust to JS. Further iterations will speed it up further.
Tests check that output via raw transfer matches output via JSON transfer exactly for all of Test262 which Acorn is able to parse. It should also match for TypeScript and JSX, but that's not covered by tests as yet.
However, I think we should consider this experimental for now because there are a few rough edges (discussed below).
Therefore I've put it behind an "experimental" flag:
```js
const ret = parseSync(filename, code, { experimentalRawTransfer: true });
console.log(ret.program);
```
### How it works
* JS creates an `ArrayBuffer` and passes it to Rust.
* Rust creates an `Allocator` using that buffer as its backing memory.
* Rust parses the AST into that allocator (including comments).
* Rust also converts module record and errors into arena types, and writes them into the allocator.
* Rust writes into the end of the buffer the offset at which data begins in the buffer.
* Control passes back to JS.
* JS code decodes data in the buffer, and creates JS objects fitting the ESTree shape.
There is *no* serialization step on Rust side. There is no JSON encoding or decoding involved. These are the main sources of the speed gain.
### Preconditions
The reason all this works is that all AST types (and other transferred types) are `#[repr(C)]`. So the memory layouts of those types are specified, and can be statically calculated.
`oxc_ast_tools` does those layout calculations, and generates the JS-side deserializer code based on its knowledge of what offsets struct fields are at, what the discriminants of enums are etc.
### Rough edges
There are a few rough edges.
#### `Allocator::from_raw_parts`
This PR adds a method `Allocator::from_raw_parts`. It creates a `bumpalo::Bump` that uses an existing block of memory as the allocator's backing memory. This ability is required, but it's not an API that `bumpalo` offers (and the maintainer was not willing to add it).
The implementation of `Allocator::from_raw_parts` is extremely hacky. It depends on internal implementation details of `bumpalo` which are not specified. And the method used to determine the memory layout of `Bump` depends on unspecified details of Rust's memory model. So, while it does seem to work in practice, it is, strictly speaking, UB.
It could break in a future Rust version, or with esoteric compiler flags e.g. [`-Zrandomize-layout`](https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/randomize-layout.html). And it could break if we update `bumpalo` from the version we're currently using. For this reason, this PR pins the version of `bumpalo` in `Cargo.toml`.
I think this OK for now, but it's unpleasantly fragile.
We can resolve all these problems by replacing `bumpalo` with our own arena allocator (which I think we should do anyway, for other reasons).
In the meantime, `Allocator::from_raw_parts` is behind a cargo feature `from_raw_parts`, to avoid it being used anywhere else in our codebase.
#### Unspecified type layouts
As noted above, all AST types are `#[repr(C)]`, so their layout is specified and stable. There are a few types which are outside of our control, though:
1. `Vec<T>`. We use `allocator_api2::vec::Vec`, which is not `#[repr(C)]`.
2. `&str`. I don't believe the layout of this type is specified.
3. `Option<T>`.
For production-grade stability, we need to try to work around these.
`Vec` - we should replace `allocator_api2::vec::Vec` with our own `Vec` type. This will also allow us to reduce its size (https://github.com/oxc-project/backlog/issues/18).
`&str` - again, we need our own string slice type, to work around the problem of lone surrogates (#3526) and to make it more efficient (https://github.com/oxc-project/backlog/issues/46). We can make its memory layout stable at the same time.
`Option<T>` is tricky. We don't want to replace Rust's `Option` because of the niche optimization benefits it gives. I'm not sure this one is 100% soluable, but Rust gives at least *some* guarantees about the layout of `Option`. Maybe we can avoid using `Option` in the AST in ways which go outside that specification.
#### Large buffers
For speed, raw transfer requires the entire AST to be in a single contiguous memory region, and for the start of that region to be aligned on 4 GiB.
JS does not support 64-bit integers, so offset calculations are much cheaper when the buffer is aligned on 4 GiB and no larger than 4 GiB - because then all pointers have the same value in their top 32 bits. So the pointer can be treated as a 32-bit value (bottom 32 bits only). JS can handle 32 bit integers no problem.
When creating a large buffer on JS side, it *mostly* ends up aligned on a 4 GiB boundary anyway, but occasionally it doesn't. So in order to ensure the buffer has at least 1 region within it which is aligned on 4 GiB, and 2 GiB in size, we have to create a 6 GiB buffer.
I *think* this is OK. On systems with virtual memory, allocating 6 GiB only reserves 6 GiB of *virtual* memory. Physical memory is only consumed when the pages of that allocation are actually written to.
But I may be missing something here, and memory exhaustion might be a danger. I think we need some real-world usage to find out.
*Possibly* we could reduce the need for so much memory if JS deserializer called into a small WASM module to do offset calculations. WASM can work with `i64` values. Or there may be other solutions.
#### Endianness
Currently only little-endian systems are supported. Probably in practice this doesn't matter much, but it'd be ideal to cover big-endian too.
Speed up UTF-8 to UTF-16 `Span` conversion by processing span offsets in ascending order.
This involves, for each node, processing `span.start`, then visiting all children and processing their `Span`s, and then finally processing `span.end`.
When the AST has come direct from the parser, this means all offsets are processed 100% in ascending order (and the visitor used here has manually-written visitation methods for the few types where that's not always the case otherwise e.g. `export {x}`).
If the AST *has* been modified, code may have moved around, so ascending order won't necessarily be preserved, but still it *mostly* will.
Optimize for this visitation order. Instead of doing a binary search through the whole UTF8-UTF16 translation table on every single span offset, assume the current offset is in same region as the last. Only resort to binary search as a de-opt when that's not the case. Most JS/TS files don't contain many non-ASCII characters, so these cases will be rare.
Pure refactor. Rename:
* `Program::to_json` to `to_estree_ts_json`.
* `Program::to_pretty_json` to `to_pretty_estree_ts_json`.
This is preparatory work for #9285, which adds `*_js_*` variants of these 2 methods, which omit TypeScript fields.
Also rename the serializers, to include `TS` in their names, for the same reason.
Replace `serde` with a custom serializer specialized for writing JSON.
`impl Serialize` -> `impl ESTree`. I've called it `ESTree` in case we
want to add e.g. `impl Babel` later on.
The main motivation is that with control of the serializer, we can make
it stateful, so we can extend it to, for example, omit TS fields based
on a flag. This is pretty much impossible with `serde`'s interface.
But a side-effect of the simpler implementation is that it's also more
performant.
The usage of `ESTree` trait is very similar to `serde::Serialize`. Main
differences:
* `serialize` method is infallible. Does not return a `Result`.
* `state = serializer.serialize_struct();` instead of `map =
serializer.serialize_map();`.
* `state.serialize_field(...)` instead of `map.serialize_entry(...)`.
* Struct keys must be strings, and JSON-safe (not including `"` or other
characters that require escaping in JSON).
This is a breaking change, as it's possible downstream consumers are
using `serde` to serialize our AST to other formats. I'd guess unlikely,
but possible. We can always implement `serde::Serialize` as well as
`ESTree` further down the line if there's a demand.
I (and mostly copilot) wrote `VisitMutModuleRecord` to visit all spans as I wasn't sure how `VisitMut` is generated. I'm looking into visit generator now if it's something I can do. Please let me know if manually writing it down is fine for now or I should try macro (or maybe a completely different approach).
overlookmotel