Drop support for NodeJS before v20 in `oxc-parser` NPM package.
All versions of NodeJS earlier than v20 are now EOL. I don't think we need to support them.
https://nodejs.org/en/about/previous-releases
I would like to use `FinalizationRegistry`, which is not available in Node v14 (our current lowest supported version).
Fix#10519.
Run raw transfer tests across multiple threads. Vitest can run multiple test *files* in parallel on separate threads, but it doesn't have ability to run test *cases* in the same file in parallel.
So create a worker pool and send details of each test case to the workers, which then run them in parallel.
On my machine, this cuts down the time to run `napi/parser` tests by a factor of 3 - from 19s to 6s.
Also: Fix `parseAsync` tests: `Promise`s weren't being `await`-ed, and cut down number of iterations on `parseAsync` test from 100,000 to 10,000 so it completes in a reasonable amount of time.
Use the term "construct" for creating an AST node object in lazy deserializer. "Deserialize" was the wrong word to use, because the whole point is that it *doesn't* deserialize until you ask it to.
First iteration of lazy deserialization. It works (does not throw an error on any of Test262 or TS test cases), but has some major shortcomings at present:
* Does not match ESTree - shape of AST is more like our Rust types.
* Some missing fields.
* `Vec`s are not lazily deserialized.
* No caching of nodes, so you get a different object each time you get a node e.g. `assert(binaryExpression.left == binaryExpression.left)` fails!
Note: This is a lazy deserializer, which is first step towards a lazy visitor. But there's no visitor implemented yet.
I will build on this starting point in future PRs.
Pure refactor. Move code for raw transfer into separate directory. This will enable exposing some internals to benchmarks without exposing them publicly from the package. Also lazy deserialization is going to add a lot more code - don't want `index.js` to get too bloated.
Also make main entry point `index.js` run in strict mode (doesn't much matter, but it's good practice).
Add `value` field to `BigIntLiteral`, same as the other literal types.
Once upon a time, we stored the value as a `BigInt` which normalized the representation (removing `_`s, and converting e.g. `0xFF` to `255`). But we had to remove it because `BigInt` owns a heap allocation, and this caused a memory leak.
https://github.com/oxc-project/oxc/commit/50fe0fef73027f10740e68a65a78216933b5d8a0
This PR brings back the `value` field, but avoids memory leak by storing the normalized value as a string (`Atom`).
This has effects in various places in the codebase:
* Brings our JS-side AST into line with ESTree (fixing the test cases broken by #11563).
* Fixes edge cases in a couple of lint rules.
* Codegen now prints bigints in normalized form e.g. `0xFFn` is printed as `255n`.
* Changes behavior of 1 function in `oxc_ecmascript`.
I've made comments below on the parts I'm not so sure about. Review would be very welcome!
Related to #11489.
Remove the duplicate properties from TS-ESTree JSON for `BindingPattern` by using custom serializers.
This isn't the ideal solution, but it does solve the immediate problem.
Add `codeframe` property to errors returned by `parseSync` / `parseAsync` when using raw transfer. This matches the shape of error objects without raw transfer.
There were some test cases we needed to skip because they'd cause parser to panic. That bug is now fixed (#11473), so no need to skip these tests any more.
Add NAPI parser tests to check that TS-ESTree AST via raw transfer matches snapshots.
Previous PRs have fixed all remaining issues, so all the tests pass!
Re-order fields in ESTree and TS-ESTree ASTs to be in visitation order (which is the order in which they're defined in Rust types).
This does not align with either Acorn or TS-ESLint, but for the reasons discussed in https://github.com/oxc-project/oxc/issues/9705#issuecomment-2859250834, I feel this isn't a concern. The field order doesn't matter much per se, only that it's consistent, and this change has other benefits - it will make it simpler to build a visitor which visits fields in the correct order, and allows adding other parsers to `acorn-test262` in order to produce test cases for decorators.
Visitation order aligns with TS-ESLint except for the following types:
* `ExportSpecifier`
* `TSImportType`
* `TSIndexedAccessType`
* `TSMethodSignature`
* `TSPropertySignature`
* `TSTypePredicate`
In these 6 cases, TS-ESLint's `visitorKeys` has the wrong visitation order, because it does not visit fields in source order. I will open an issue and hopefully they can fix.
This PR also bumps `acorn-test262` to include https://github.com/oxc-project/acorn-test262/pull/37 which re-orders the fields in fixture snapshots to match the new field order here.
Previously we automatically re-ordered fields to put TS-only fields last. Don't do that - use definition order, unless a field order is specified with `#[estree(field_order(...))]`.
Previously, only `parseSync` supported raw transfer. Extend support to `parseAsync` too.
This requires:
1. A queue to limit the number of files being parsed simultaneously, to avoid the possibility of memory exhaustion.
2. A more complicated mechanism for recycling buffers, since now multiple buffers can be in use at the same time.
Related to #10978. Add `phase` property to `ImportExpression` in TS-ESTree AST.
This aligns with ESTree spec, but does *not* align with TS-ESLint - TS-ESLint outputs a `CallExpression` with a `MetaProperty` as its `callee` instead.
We don't have the necessary information in our AST to match TS-ESLint, because we don't have the spans of `import`, `defer`, or `import.defer`, to use for the `MetaProperty`, or the `IdentifierName`s it contains.
This is not completely ideal, but I figure it's preferable to represent this syntax *somehow* in the AST, rather than completely ignoring it, in which case there's no way to distinguish between `import("x")`, `import.defer("x")`, and `import.source("x")`.
Presumably TS-ESLint will change their AST shape to conform to ESTree spec once these reach stage 4, so I don't think we should expend effort changing our AST to align with them in the meantime. TypeScript doesn't even have test cases for these (we'd know if they did, as we'd be failing them) - so I doubt these are popular constructs in TS.
`acorn-test262` submodule is bumped to include https://github.com/oxc-project/acorn-test262/pull/36 which adds the `phase` field to `ImportExpression`s in test snapshots.
Related to #10978. Add `phase` property to `ImportExpression` in ESTree AST.
This property is *not* added in TS-ESTree, because TS-ESLint supports `import.defer(...)` and `import.source(...)`, but does not align with ESTree - it outputs a `CallExpression` with a `MetaProperty` as its `callee` instead.
We don't have the necessary information in our AST to match TS-ESLint, because we don't have the spans of `import`, `defer`, or `import.defer`, to use for the `MetaProperty`, or the `IdentifierName`s it contains.
Presumably TS-ESLint will change their AST shape to conform to ESTree spec once these reach stage 4, so I don't think we should expend and changing our AST to align with them in the meantime. TypeScript doesn't even have test cases for these (we'd know if they did, as we'd be failing them) - so I doubt these are popular constructs in TS.
`acorn-test262` submodule is bumped to include https://github.com/oxc-project/acorn-test262/pull/31.
Codegen for `ESTree` impls supports a `#[ts]` attribute on struct fields which only appear in TS-ESTree AST. Add a `#[js_only]` attribute which does the same thing in reverse - indicates a field only appears in the JS ESTree AST, and not in TS-ESTree.
This allows removing a custom serializer and hacky workaround for `JSXOpeningFragment`.
Fixes#10978. Add `phase` property to `ImportDeclaration` in ESTree and TS-ESTree ASTs.
`ImportExpression` is not covered by this PR. That's more tricky as TS-ESLint already supports `import.defer(...)` and `import.source(...)`, but does not align with ESTree - it outputs a `MetaProperty` instead.
`acorn-test262` submodule is bumped to include https://github.com/oxc-project/acorn-test262/pull/30.
Update docs for `oxc-parser` NPM package to clarify:
1. TS AST now aligns with TS-ESLint.
2. We align with Acorn for plain JS ESTree.
3. We add a non-standard `hashbang` property to `Program`.
When I first implemented this proposal, it was Stage 3 and the specs
were not even fixed.
But for now, this spec is in Stage 4 and has been cleared up, so I've
updated it.
In the process, I have done some refactoring.
- Shorten code by `bitflags` like `RegExpFlags`
- This requires some lines to make `ast_tools` works
- Update readme
NOTE: The basic behavior remains unchanged.
All our AST structs are `#[repr(C)]`, which gives us guaranteed stable memory layouts. This is essential for raw transfer, and helpful for other purposes too.
However, the downside is that we lose out on the automatic field-reordering that default `#[repr(Rust)]` provides, where structs are packed to minimize padding bytes.
This PR implements a similar field-reordering scheme, to get the same effect.
* `oxc_ast_tools` determines optimal field order for compact memory layout.
* `oxc_ast_tools` generates a file (`structs.rs`) which details the desired field order for each type.
* `#[ast]` proc macro follows the instructions in `structs.rs` and re-orders the fields.
This reduces the size of various AST types by 8 bytes. No type now has more than 7 padding bytes (previously e.g. `Function` had 15 bytes of padding).
Possible future improvements:
1. Get Rust Analyser to display original field order.
2. Remove `syn` dependency from `oxc_ast_macros` crate, to improve compile times. It should be possible to do all the AST manipulation using just `proc_macro` crate, by generating more code in `oxc_ast_tools`, so the macro's logic can be simpler.
Note: This PR doesn't add `syn` dependency, it was already there. But it makes it easier to remove it.
Related:
* #6488
* #9706
Finished first step(32 -> 24) of #9706.
Change `len` and `cap` fields from `usize` to `u32`, so that the `Vec` size can reduced size to `24`. Also, to avoid some unnecessary unit conversion. I changed many `RawVec` methods to take `u32` rather than `usize`.
The performance looks great now! And we reduced 1%-3% memory usage in the parser.
I noticed that most of `Comment` consisted of enums which were only 1 byte in size, but only used a few bits in each byte. There are few enough fields that we can actually store all of them in a single `u16` bit flag, which reduces the size of `Comment` from 24 bytes to 16 bytes.
overlookmotel