Tiny optimization to lazy visitor. It's marginally cheaper in a loop to compare to 0, rather than to a constant. I also assume that `!== 0` is cheaper than `> 0` as it doesn't involve type coercion.
Improve raw transfer benchmarks. Use `require` to load internal modules instead of `import`. `oxc-parser` uses `require` internally, so using `import` here was resulting in a lot of code being loaded and instantiated twice.
Add lazy visitor to `oxc-parser`.
This API is experimental and not ready for use yet. AST is not ESTree-compatible at present (same as lazy deserializer).
Usage:
```js
import { parseSync, experimentalGetLazyVisitor } from 'oxc-parser';
// Get `Visitor` class.
// `Visitor` is not exported directly from `oxc-parser` as all code
// related to lazy deserialization is lazy-loaded.
const Visitor = experimentalGetLazyVisitor();
// Define visitor.
// This visitor counts how many identifiers in AST.
let identCount = 0;
const visitor = new Visitor({
BindingIdentifier(ident) {
identCount++;
},
IdentifierReference(ident) {
identCount++;
},
IdentifierName(ident) {
identCount++;
},
});
const code = 'let x, y; x = { z: 2 };';
const { visit, dispose } = parseSync('test.js', code, { experimentalLazy: true });
visit(visitor);
assert(identCount === 4);
// Dispose of AST now we're done with it.
// This returns buffer storing AST data to the pool for reuse.
dispose();
```
Pure refactor. Adapt lazy deserialization codegen in preparation for adding lazy visitor. Involves introducing a `State` struct, renaming things, and moving code around. No substantive changes.
Tiny perf optimization to raw transfer. #11807 made this whole file lazy-loaded, so no need to lazily create the `TextEncoder` any more. If the file is imported, it's going to be used.
Pure refactor. Move `parseSync` to be before `parseAsync`, for consistency with all other code in `napi/parser`. Hoist all exports to top of file to make it a little more readable.
Small perf optimization to `oxc-parser`. Destructure `bindings` object when importing it. Calling those methods repeatedly later on avoids overhead of an object property lookup each time.
The effect is infinitesimal, except when using lazy deserialization, which is so fast for small files that it makes sense to optimize every step of its call path.
Raw transfer and raw lazy deserialization are experimental and not publicly advertised. Lazy-load all code for these features only when they're first used.
Lazy deserialization in particular requires a non-trivial amount of code.
`rawTransferSupported` still needs to be eagerly loaded, as user would call it before calling `parseSync` or `parseAsync`. So move it into a separate module.
Pure refactor. Add more `#[estree]` attributes to `RegExpFlagsAlias` type. These have no effect on generated code for JSON serialization or raw transfer deserialization, as `RegExpFlagsAlias` already has a custom converter. But it simplifies the codegen for lazy deserialization.
Small perf optimization to raw transfer. Don't use 2 different counters in the loop when deserializing `Vec`s. Do what Rust's slice iterators do, and use current and end pointers.
Remove the custom ESTree serializer for `MethodDefinition`'s `key` field. This serializer was required to replicate a bug in TS-ESLint, which they've now fixed. This fixes the test case which was broken by #11762.
Move `getElement` function into `NodeArray` class definition, to avoid extra function call to intermediate `getInternal`. This is a hot path, so streamlining it is important for perf.
Simple object shapes are faster. Adding extra property to `NodeArray.prototype` alters its shape. Prevent this by defining a placeholder method in class definition, and then mutating this property later, rather than adding a new property.
Implement an optimized `slice` method for `NodeArray` in lazy-deserialized AST. It returns a another `NodeArray`, instead of a plain `Array`, and doesn't deserialize any of the contents.
Lazy deserialization return `NodeArray`s which deserialize their contents lazily, instead of plain JS arrays.
`program.body.find(stmt => stmt.type === 'ImportDeclaration')` now only deserializes the nodes up until first `ImportDeclaration`, rather than deserializing every statement in the program into an array, and then searching it. Similarly getting `program.body.length` does not deserialize any nodes, and all other array methods are similarly optimized.
Also provide an optimized iterator for the common case of:
```js
for (const stmt of program.body) {
doStuff(stmt);
}
```
Pure refactor. We prefix some property names on `internal` objects with `$` to avoid clashes between static properties (`pos`, `ast`) and cached fields. Switch the prefixing around - now prefix keys for cached fields, so the internal properties are called e.g. `pos` not `$pos`. This shortens generated code, and makes writing implementations simpler.
Fix a bug in lazy deserializer where `Vec`s were not getting cached because they were wrapped in `Option`. When determining if a node's field needs to be locally cached, drill down through any wrapping `Option`s / `Box`es / `Cell`s / enums.
Improve the output from `console.log` when printing a lazy-deserialized AST.
Previously `console.log(program)` printed `Program { type: 'Program' }`. Now it prints much the same as a normal (eagerly deserialized) AST:
```
Program {
type: 'Program',
start: 0,
end: 17,
sourceType: SourceType { sourceType: 'module' },
hashbang: null,
body: [
VariableDeclaration {
type: 'VariableDeclaration',
start: 0,
end: 10,
kind: 'let',
declarations: [Array],
declare: false
}
]
}
```
In lazy-deserialized AST, cache node objects, so accessing the same field twice returns the same object. Previously `assert(binaryExpression.left == binaryExpression.left)` would fail, as each access returned a new object.
`pos` (i.e. memory address) is used as the cache key, with 2 exceptions: `RegExp` and `Program`.
`RegExp` doesn't have a `span: Span` field, so its first field `RegExpPattern` has the same memory address as the `RegExp`. So `pos` would not be a unique key.
```rs
pub struct RegExp<'a> {
pub pattern: RegExpPattern<'a>,
pub flags: RegExpFlags,
}
pub struct RegExpPattern<'a> {
pub text: Atom<'a>,
pub pattern: Option<Box<'a, Pattern<'a>>>,
}
```
The codegen identifies these cases, and applies a workaround - `pos + 1` is used as the cache key instead of `pos`, so the parent and child have different cache keys.
Drop support for NodeJS before v20 in `oxc-parser` NPM package.
All versions of NodeJS earlier than v20 are now EOL. I don't think we need to support them.
https://nodejs.org/en/about/previous-releases
I would like to use `FinalizationRegistry`, which is not available in Node v14 (our current lowest supported version).
Fix#10519.
Run raw transfer tests across multiple threads. Vitest can run multiple test *files* in parallel on separate threads, but it doesn't have ability to run test *cases* in the same file in parallel.
So create a worker pool and send details of each test case to the workers, which then run them in parallel.
On my machine, this cuts down the time to run `napi/parser` tests by a factor of 3 - from 19s to 6s.
Also: Fix `parseAsync` tests: `Promise`s weren't being `await`-ed, and cut down number of iterations on `parseAsync` test from 100,000 to 10,000 so it completes in a reasonable amount of time.
Use the term "construct" for creating an AST node object in lazy deserializer. "Deserialize" was the wrong word to use, because the whole point is that it *doesn't* deserialize until you ask it to.
First iteration of lazy deserialization. It works (does not throw an error on any of Test262 or TS test cases), but has some major shortcomings at present:
* Does not match ESTree - shape of AST is more like our Rust types.
* Some missing fields.
* `Vec`s are not lazily deserialized.
* No caching of nodes, so you get a different object each time you get a node e.g. `assert(binaryExpression.left == binaryExpression.left)` fails!
Note: This is a lazy deserializer, which is first step towards a lazy visitor. But there's no visitor implemented yet.
I will build on this starting point in future PRs.
Pure refactor. Move code for raw transfer into separate directory. This will enable exposing some internals to benchmarks without exposing them publicly from the package. Also lazy deserialization is going to add a lot more code - don't want `index.js` to get too bloated.
Also make main entry point `index.js` run in strict mode (doesn't much matter, but it's good practice).
Add `value` field to `BigIntLiteral`, same as the other literal types.
Once upon a time, we stored the value as a `BigInt` which normalized the representation (removing `_`s, and converting e.g. `0xFF` to `255`). But we had to remove it because `BigInt` owns a heap allocation, and this caused a memory leak.
https://github.com/oxc-project/oxc/commit/50fe0fef73027f10740e68a65a78216933b5d8a0
This PR brings back the `value` field, but avoids memory leak by storing the normalized value as a string (`Atom`).
This has effects in various places in the codebase:
* Brings our JS-side AST into line with ESTree (fixing the test cases broken by #11563).
* Fixes edge cases in a couple of lint rules.
* Codegen now prints bigints in normalized form e.g. `0xFFn` is printed as `255n`.
* Changes behavior of 1 function in `oxc_ecmascript`.
I've made comments below on the parts I'm not so sure about. Review would be very welcome!
overlookmotel