Oxlint's raw transfer implementation stores the source text at end of the buffer. Prior to this PR `napi/parser` stored source text at the start of the buffer.
Bring `napi/parser` into line with Oxlint, by also storing source text at end of the buffer.
Making all versions of raw transfer that we use align with each other removes complication (all versions now have the same implementation of `deserializeStr`) and also removes some fragile code where it'd be easy to accidentally trigger UB (see the updated comments in `arena/fixed_size/windows.rs` and `arena/alloc_impl.rs`).
Unfortunately, storing source at end of buffer is slightly less efficient than storing it at the start. We'll switch all implementations to storing source at start of buffer when we move to storing all strings (source text and all other strings) together in a single contiguous block. But in meantime, this slight inefficiency is outweighed by the gain in safety.
Large change to how raw transfer stores metadata about the allocations it uses.
Previously the two metadata structures `RawTransferMetadata` and `FixedSizeAllocator` after the chunk's `ChunkFooter`.
This had a few problems:
1. It was confusing and unwieldy - a recipe for bugs.
2. `ChunkFooter`'s memory was exposed on JS side - a bit unsafe as altering bytes in this region could easily trigger UB.
3. It entwined `Arena` (which is just the thing that allocates) with the details of exactly _what_ raw transfer allocates.
4. Imposed annoying alignment requirements, because `ChunkFooter` must be aligned on 16, and so anything after it must ensure it doesn't break that invariant.
Old layout:
```
WHOLE BLOCK - aligned on 4 GiB
<-----------------------------------------------------> Allocated block (`BLOCK_SIZE` bytes)
ALLOCATOR
<-----------------------------------------> `Allocator` chunk (`CHUNK_SIZE` bytes)
<----> `ChunkFooter` (aligned on 16)
<-----------------------------------> `Allocator` chunk data storage (for AST)
(`ACTIVE_SIZE` bytes)
METADATA
<----> `RawTransferMetadata`
<----> `FixedSizeAllocatorMetadata`
BUFFER SENT TO JS
<-----------------------------------------------> Buffer sent to JS (`BUFFER_SIZE` bytes)
```
This PR moves `RawTransferMetadata` and `FixedSizeAllocatorMetadata` into the chunk itself. New layout:
```
WHOLE BLOCK - size 2 GiB - 16, aligned on 4 GiB
<-----------------------------------------------------> Allocated block (`BLOCK_SIZE` bytes)
ARENA
<-----------------------------------------------------> Chunk (fills whole block)
<--------------------------------------> Allocatable region for AST (`ACTIVE_SIZE` bytes)
<---> `RawTransferMetadata`
<---> `FixedSizeAllocatorMetadata`
<---> `ChunkFooter` (aligned on 16, last in block)
BUFFER SENT TO JS
<-------------------------------------------> Buffer sent to JS (`BUFFER_SIZE` bytes)
```
`FixedSizeAllocatorMetadata` and `ChunkFooter` are no longer in the region which is shared with JS side. As far as `Arena` is concerned, they're now just some data (like any other data) which is allocated in the arena.
Also:
- Introduce more consistency to the naming of constants which specify the size and position of these various data structures in the arena.
- Add more const assertions to ensure everything is laid out and aligned as it should be.
Continuation of #21132. All uses of the `Uint32Array` view of the buffer in raw transfer code have been removed in preceding PRs. So we can now remove it entirely.
Continuation of #21132. Use `Int32Array` view of the buffer for all code in lazy deserialization. Lazy deserialization is not currently in use, but keeping it in line with the main raw transfer deserializer allows removing the `Unint32Array` view of the buffer entirely in #21140 without breaking the lazy deserializer tests.
Small perf optimization. When deserializing tokens in `oxc-parser`, use only the `Int32Array` view of the buffer, instead of both `Int32Array` and `Uint8Array`. Less values to keep in registers, and `int32` doesn't need to be obtained with a property lookup on each call.
Implement the beginnings of support for tokens in `napi/parser`.
This PR only adds support for tokens via raw transfer, and behind an undocumented `experimentalTokens` option.
Add tests checking that tokens received on JS side via raw transfer match snapshots for all Test262, AcornJSX, and TypeScript test cases. They do!
The tests are the main purpose of this PR, making sure it works before we switch over to tokens via raw transfer in Oxlint.
We can add full tokens support to `napi/parser` (including via JSON transfer) later on.
Similar to #16442.
There's no need to remove the `experimental*` properties from `options` object on JS side. They'll be ignored by NAPI-RS on Rust side anyway, so creating a new object on JS side is pointless overhead.
Pure refactor. Just rename generated `types.js` files to `type_ids.js`. This name better describes what the files contain, and it avoids naming clash with `types.d.ts` in `apps/oxlint/src-js/generated`.
Add an option to `oxc-parser` to add `parent` to all AST nodes. This option is only available with raw transfer (not really feasible with JSON transfer, as JSON doesn't support circular references).
This is really for the purposes of JS plugins in Oxlint, so have not included the new deserializers which add `parent` in the `oxc-parser` NPM package, to avoid bloating download size. But it's ideal home is in `napi/parser` so can add it to the conformance tests.
`parent` is correct for all nodes in all Test262 and TypeScript test cases.
We had a runtime `preserveParens` option in deserializers to cater for linter, but it was not used in parser. Convert it to a compile-time option instead.
Small fix to JSDoc comments for raw transfer deserialization.
`options` must be an object at this point because raw transfer code can only be entered if `options` contains `experimentalRawTransfer: true`.
Now that `apps/oxlint` and `napi/parser` are both ESM packages (#13723 and #14042), we can use `.js` file extensions for all ESM files, rather than the mix of `.js` and `.mjs` file we had previously.
It's less confusing having to remember what's `.js` and what's `.mjs`, and avoids ugly workarounds like #14038 (`.d.mts`??).
Pure refactor. Root of `napi/parser` was getting really crowded with tons of files, and I'm about to add a load more for the ESTree walker.
Move JS source files into a `src-js` directory, like in `apps/oxlint`, so that the root of the package is just config files and scripts.
Boshen