Add a method `Utf8ToUtf16::convert_program_and_comments`, and use it in a couple of places.
This is marginally faster than `convert_program` followed by `convert_comments`, as it avoids branching on the same condition twice.
Previously, whether to serialize AST to ESTree JSON with/without TS fields was a compile-time option - via 2 different configs `CompactTSSerializer` and `CompactJSSerializer` (and their pretty-printing and with-fixes counterparts).
Turn this into a runtime option instead -
This make serialization a little slower (more branches in either mode, and dead code in JS-only mode) but reduces the size of `oxc-parser` binary as only one `ESTreeSerializer` is generated, instead of 2 (JS-only, and with-TS).
I added these benchmarks while working on improving perf of tokens APIs in linter. That work is now complete, so these benchmarks are no longer needed. We can always restore them later on if we start working on perf of tokens again.
Add a function `update_tokens` which converts tokens to ESTree format in place, mutating the `Token`s in the `Vec<Token>` in place.
This is what will be used for sending tokens to JS side via raw transfer.
Support for serializing tokens to JSON is also retained. The 2 implementations share the same AST visitor, and hook into it via separate `Context` implementations.
Add a benchmark for preparing tokens on Rust side to be sent to JS with raw transfer. This isn't implemented yet, so initially this benchmark measures perf of serializing to JSON.
Introduce `ESTreeTokenConfig` trait and allow option for the "flavor" of tokens to be set at compile time.
When producing Espree-style tokens, this allows cutting out all the special-case logic related to `JSXIdentifier` tokens and removing 2 branches from `emit_identifier_at`.
Small effect on perf - only +0.3% on ESTree tokens benchmark.
Previously we used a 3-pass process for serializing tokens in ESTree format.
* Pass 1: Walk AST and produce a list of "overrides" - which tokens need their `Kind` changed.
* Pass 2: Loop through all tokens, apply any overrides, convert to `ESTreeToken`s, and save them in a `Vec`.
* Pass 3: Serialize the `Vec<ESTreeToken>` with `ESTree` trait (JSON serializer).
Compress this all to a single pass:
* Create an `ESTreeSequenceSerializer` which can have items sent to it to serialize one-by-one (streaming).
* Walk AST.
* When encounter a token which needs its `Kind` altered:
* Flush all previous tokens, serializing each in turn, using its default `Kind`.
* Serialize the current token with the amended `Kind`.
* Continue walking.
* At the end, flush any remaining tokens to the serializer.
This avoids multiple passes, and also removes the `Vec<ESTreeToken>` which was just intermediate data, and grew very large.
+18% perf improvement on ESTree tokens benchmark.
Refactor.
Previously we converted spans from UTF-8 to UTF-16 in AST first, then serialized tokens. This meant that when serializing tokens, we had to convert spans to UTF-16 before comparing offsets of overrides to the token's start offset (because the override offsets are taken from the AST nodes during the AST walk pass).
Instead, convert token spans first, *then* the AST. This removes span conversion from the critical path. On a CPU with instruction-level parallelism, other work can start without being blocked on waiting for the result of span conversion, and the two can happen in parallel.
This should be a slight perf improvement, though it likely won't show up on CodSpeed benchmarks as CodSpeed's CPU simulation to basic to emulate instruction-level parallelism.
But mainly, the motivation is that this makes the process simpler and easier to reason about.
Fix a bug where we didn't produce any tokens for files which start with a BOM.
As a side effect, re-use the existing `Utf8ToUtf16` span converter created for converting spans in the AST again for tokens, rather than creating another one.
Refactor. We had pointlessly complex logic where caller would call `collect_token_context` and then pass the result back into some other tokens code `to_estree_tokens_json`. Combine the 2 functions to make the call sites simpler.
Add benchmarks for:
1. Parsing with tokens.
2. Serializing tokens to ESTree-style JSON.
Neither of these benchmarks needs to be kept indefinitely, but they'll be useful while working on the tokens code. There's a lot of room for optimization.
Part of #10307. Add `range` option to `oxc-parser`, to add `range` property to nodes in ESTree AST.
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: overlookmotel <theoverlookmotel@gmail.com>
The benchmark jobs take too long to run, change them to a minimal set.
This set of files are smaller but should provide basic coverage to real
world examples.
#10791 fixed the performance of `oxc-parser` NPM package by removing the reviver from `JSON.parse` call. However, it replaced it with a complete traversal of the AST on JS side to locate `Literal` nodes and update them.
Instead, generate a list of paths to `Literal` nodes needing fixing on Rust side in `ESTree` serializer.
e.g. for this program:
```js
123n;
foo(/xyz/);
```
the fix paths are:
```json
[
["body", 0, "expression"],
["body", 1, "expression", "arguments", 2]
]
```
Having the location of nodes which need updating reduces work on JS side, as no need to search the entire AST.
Running `pnpm run bench` (in `napi/parser`) locally shows between 8% and 15% speed-up from this change.
In benchmark for ESTree conversion, pass result of `program.to_estree_ts_json()` to `black_box`. Without it, if compiler was sufficiently clever, it could remove the entire call to `to_estree_ts_json` because the result is never used.
Speed up UTF-8 to UTF-16 `Span` conversion by processing span offsets in ascending order.
This involves, for each node, processing `span.start`, then visiting all children and processing their `Span`s, and then finally processing `span.end`.
When the AST has come direct from the parser, this means all offsets are processed 100% in ascending order (and the visitor used here has manually-written visitation methods for the few types where that's not always the case otherwise e.g. `export {x}`).
If the AST *has* been modified, code may have moved around, so ascending order won't necessarily be preserved, but still it *mostly* will.
Optimize for this visitation order. Instead of doing a binary search through the whole UTF8-UTF16 translation table on every single span offset, assume the current offset is in same region as the last. Only resort to binary search as a de-opt when that's not the case. Most JS/TS files don't contain many non-ASCII characters, so these cases will be rare.
Pure refactor. Rename:
* `Program::to_json` to `to_estree_ts_json`.
* `Program::to_pretty_json` to `to_pretty_estree_ts_json`.
This is preparatory work for #9285, which adds `*_js_*` variants of these 2 methods, which omit TypeScript fields.
Also rename the serializers, to include `TS` in their names, for the same reason.
Pure refactor. Use `bench_function` instead of `bench_with_input` and
just borrow data from outside closure. This shortens the code and (I
think) makes it easier to read.
Re-use allocator in parser + lexer benchmarks.
I believe this is the recommended usage when parsing a bunch of files -
to re-use one allocator rather than create a fresh one for each run, so
it makes sense to me that this is what the benchmark should measure.
Doesn't show much difference on CodSpeed because it only runs the
benchmark once, and it treats allocations as free anyway. But I imagine
the difference may show up a bit more in a standard criterion benchmark.
overlookmotel