Move work from `#[ast]` macro into `ast_tools`. Calculate whether an enum is fieldless or not in `ast_tools` and pass that info to `#[ast]` macro via an `EnumDetails` struct. This avoids having to iterate over all variants of each enum in the macro.
This is preferable because `ast_tools` runs only after changes to AST, whereas the macro code runs every time `oxc_ast` crate is compiled.
For historical reasons, `oxc_span` crate was re-exporting string types which now live in `oxc_str` crate.
Remove these re-exports, and import these types directly from `oxc_str`.
Main motivation is just that it makes sense, but it should also have a minor positive effect on compile times - `oxc_span` and `oxc_str` crates can now be compiled in parallel.
One annoyance with `ast_tools`, our codegen, has been that when you want it to generate code from types in some file, you had to manually add the file paths to a list in `ast_tools` itself.
This PR removes that list and instead finds crates in the monorepo which depend on `oxc_ast_macros` crate, via `cargo metadata`, and searches all files in those crates for types with `#[ast]` attributes etc - types that it need to act on.
There is no longer a list to keep updated, you just mark types `#[ast]` and the codegen will find them.
This produces some churn in generated files. None of the generated code changes, it just gets re-ordered in some places, due to the different order that files get found in now. This order is deterministic, and it should not change again.
## Summary
- Add blank lines between generated items using `///@@line_break` in traverse trait, walk functions, ancestor types, Without structs, `is_*`/`is_parent_of_*` methods
- Fix match arm ordering in inherited enums (BFS instead of DFS) to match JS output
- Restore SAFETY comment block and "Compiler should reduce" comment using `///@ ` syntax
Remaining diff (~360 lines across 3 files) is exclusively intentional improvements:
- Header comment referencing new generator location
- V8 naming consistency (`v_8_` matching `convert_case` crate, consistent with other Rust-generated files)
- Type ordering (literals before JSX, following `SOURCE_PATHS` order)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Generate a file defining the vars that each environment (`env`) provides.
This isn't used for anything yet, but it will be in next PR #17293.
Vars for each environment are stored in this form:
```js
{
readonly: ["var1", "var2", "var3"],
writable: ["var4", "var5"],
}
```
instead of the more obvious:
```js
{
var1: false,
var2: false,
var3: false,
var4: true,
var5: true,
}
```
...because it's shorter code when there's lots of vars (which there usually are), and iterating through arrays I assume is faster than iterating through properties of objects (which are essentially hash maps when they're large).
this change forces ast changes to run when package.json (oxfmt version)
OR oxfmtrc.jsonc (oxfmt config) changes, hopefully avoiding failing CI
due to this not running, and changes to either of those two happening
#15565 made `oxc_ast_tools` depend on local workspace versions of various crates. Add those crates to the watch list for "AST Changes" CI task, so if changes are made in them, `ast_tools` codegen will run to ensure generated code has been updated to reflect those changes.
In particular, this will prevent generated code getting out of sync when changes to minifier affect the generated code.
Previously we lazy-loaded visitor keys file, because it would be rarely used. But it's required for selectors DSL, so statically import it, and include it in bundle.
Also, import it as a TS file, since it's valid TS, even without type annotations.
Pure refactor. Generated files `constants.js` and `type_ids.js` are JS files, but they only define `const`s, so are also valid TS, without adding any type annotations. So rename them to `.ts` to allow TS to infer the types of their exports.
Pure refactor. Just rename generated `types.js` files to `type_ids.js`. This name better describes what the files contain, and it avoids naming clash with `types.d.ts` in `apps/oxlint/src-js/generated`.
Refactor. Previously build script for `oxlint` package copied files from `napi/parser`. Remove this dependency, by instead making `oxc_ast_tools` generate the files in `apps/oxlint/src-js/generated`, and then copy the files from there to `dist/generated` in the build script.
This ends up with the same result, but divorces `oxlint` package from `oxc-parser`, which is cleaner.
Refactor. Include generated files `types.js` and `walk.js` in bundle, instead of externalizing them. It's only the deserializer code that we want to keep unbundled to make it easier to debug without source maps. `types.js` and `walk.js` aren't too complicated.
Add `loc` to all AST nodes.
`loc` is computed lazily, as it's quite an expensive operation - because it requires searching the source text for line breaks and building a translation table.
We can consider making `loc` a "real" property once we have a faster translation table built on Rust side.
Export AST types directly from `oxlint` package, instead of depending on `@oxc-project/types`. This enables altering the types to reflect that `range` field is always present on AST node in Oxlint's AST (#14354).
Add an option to `oxc-parser` to add `parent` to all AST nodes. This option is only available with raw transfer (not really feasible with JSON transfer, as JSON doesn't support circular references).
This is really for the purposes of JS plugins in Oxlint, so have not included the new deserializers which add `parent` in the `oxc-parser` NPM package, to avoid bloating download size. But it's ideal home is in `napi/parser` so can add it to the conformance tests.
`parent` is correct for all nodes in all Test262 and TypeScript test cases.
We had a runtime `preserveParens` option in deserializers to cater for linter, but it was not used in parser. Convert it to a compile-time option instead.
Now that `apps/oxlint` and `napi/parser` are both ESM packages (#13723 and #14042), we can use `.js` file extensions for all ESM files, rather than the mix of `.js` and `.mjs` file we had previously.
It's less confusing having to remember what's `.js` and what's `.mjs`, and avoids ugly workarounds like #14038 (`.d.mts`??).
the latest version of `oxc-parser` (on npm) does not have resolvable
types because the entry is `.mjs` and the types are `.d.ts` (these need
to match - `.d.mts`). you can see this in the failing CI for [this
renovate
PR](https://github.com/nuxt/nuxt/actions/runs/17940186619/job/51014702361?pr=33290)
but I'm happy to provide a more minimal reproduction if it helps
I also spotted that `napi/parser/test/visit.test.ts` was importing
`visitorKeys` which was not exported from the corresponding declaration,
so took the liberty to add that export as well.
let me know if this needs any changes 🙏
I hope it helps
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: overlookmotel <theoverlookmotel@gmail.com>
Add an AST visitor to `oxc-parser`.
```js
import { parseSync, Visitor } from 'oxc-parser';
const { program } = parseSync('test.js', 'let x = { y: z };');
const idents = [];
const visitor = new Visitor({
Identifier(node) {
idents.push(node.name);
}
});
visitor.visit(program);
// Logs [ 'x', 'y', 'z' ]
console.log(idents);
```
Primarily we need a visitor for Oxlint plugins, but it's easier to develop and test in `oxc-parser`, so I thought I'd add it there first. Oxlint build process will copy the generated files for tree-walking from `napi/parser` dir, as it does with other files.
Notes:
* The `walk.mjs` file is large, so it's lazy-loaded only when user first constructs a `Visitor`. Adding `Visitor` to the package will not affect start-up time where user only uses `parseSync` etc.
* TS type definition for `Visitor` and the visitor object it takes are provided.
* Visitor supports enter and exit visitors e.g. `Program` and `Program:exit`.
* `src-js/visit/visitor.mjs` is copied from `apps/oxlint` with only minor modifications, and conversion from TS to JS. It'd be better if we only had this impl in one place, but that would require converting `oxc-parser` to TS and building with TSDown, like `oxlint` is. Leaving that for later - #13935.
Add visitor keys export to `oxc-parser` NPM package. This is useful in itself, but also the first step towards an ESTree walker.
Visitor keys are based on `@typescript-eslint/visitor-keys`.
It'd be ideal to generate visitor keys direct from Oxc's own types, but that's proving difficult due to all the custom serializers we use to translate to ESTree. So using the shortcut of borrowing from TS-ESLint for now.
Oxc's AST is slightly different from TS-ESTree (notably Oxc's AST adds `ParenthesizedExpression`). But apart from those few differences, TS-ESLint's AST and Oxc's are identical, and I made a PR to TS-ESLint earlier this year (https://github.com/typescript-eslint/typescript-eslint/pull/11279) to ensure visitor key order is correct, and matches Oxc.
To avoid adding a runtime dependency on `@typescript-eslint/visitor-keys`, extract the data from TS-ESLint in `oxc_ast_tools`, amend for Oxc/TS-ESTree differences, and re-output it as a simple object literal.
This approach will be faster at runtime, as the object's shape is consistent, and also avoids a dependency.
Convert `oxlint` to a NAPI package.
This PR:
1. Moves files from `napi/oxlint` into `apps/oxlint`.
2. Converts `oxlint` into a NAPI package.
3. Alters the [release workflow](https://github.com/oxc-project/oxc/pull/13723/files#diff-80cb497c859bbaf7756da6ced2bca3d31933affc96102983eee1ad7d4bcc60fd) and [script](https://github.com/oxc-project/oxc/pull/13723/files#diff-4c4b7045c68805aeeae7c0af94cf7cf7487cb0403496da974a6a23accdc457d0) for building `oxlint` packages.
Most of necessary code modifications and refactoring was done in previous PRs (#13701, #13712, #13722, #13745), so this PR only includes changes necessary for packaging as NAPI.
I've done my best to update release workflow, but am mostly "flying blind" without ability to test it, so very likely there are mistakes.
A few notes and oddities:
* I've kept the `oxlint` NPM package as CommonJS. We could probably switch to ESM, but I thought it better to minimize the changes in this PR.
* NAPI-RS creates `.node` files named with a `-msvc` postfix for Windows. The scripts here remove that postfix so that filename of `.node` files and NPM package names match for all platforms, for simplicity.
* `tasks/website` has a dependency on `oxlint` as a normal library. This resulted in linker errors, so have had to put all the NAPI stuff behind `napi` cargo feature. Tests for `tasks/website` are run separately in CI, to avoid feature unification turning on the `napi` feature for `website`.
I've made a few comments below for the parts which aren't fairly self-explanatory.
Rename `napi/oxlint2` to `napi/oxlint`. This is in preparation for migrating code from `napi/oxlint` into `apps/oxlint`. This PR is incidental, but renaming everything in this PR will reduce diff when moving code over to `apps/oxlint`.
Note: I haven't started to refactor the generator files yet. I just found that implementing `FormatWrite::write` for enum AST is unnecessary, so I made it clearer so that I can develop the remaining features.
First working prototype of JS linter plugins.
Rust now passes the AST to JS via buffer, and lint rule runner uses the JS rule's provided visitor to traverse the AST using lazy deserialization.
Only a single rule is supported at present.
Before anyone gets too excited, this is very much unfinished! It's incomplete, the AST is not compliant with ESTree, and it's not well tested. Really this is a proof of concept more than a usable feature at present. But we'll get there...
`RawTransferMetadata` is used to store basic info about the data in the buffer which is used on JS side to locate `Program` in the buffer etc.
This struct lives in `napi/parser` crate but we're going to need it in `napi/oxlint2` too. So make a 2nd copy of it in `napi/oxlint2` and use `oxc_ast_tools` to enforce that the 2 copies are identical.
This is a bit of a hack, but it doesn't seem worthwhile creating a new crate just to hold this one struct, and it doesn't really fit in `oxc_allocator`, or any of our other existing crates.
`FixedSizeAllocator`s allocate a block of memory larger than what the `Allocator` controls, and often with a different start pointer. So when `FixedSizeAllocator` is dropped, it needs to deallocate that block of memory using the original pointer.
That pointer way previously stored in `FixedSizeAllocator` itself. Instead store it in a `FixedSizeAllocatorMetadata` struct, which is written into the memory block itself, after the section controlled by the `Allocator`, and other metadata (`RawTransferMetadata`).
This allows getting the original `alloc_ptr` when you only have access to the `Allocator`, but not the `FixedSizeAllocator`.
Similar to #12268. Instead of hard-coding the size and alignment of raw transfer buffers in multiple places, codegen add these to generated files, and everywhere else import the values from those files. This is more robust.
Pure refactor. Instead of hard-coding constants for fixed offsets of data in the buffer, generate these constants in one place and import them from `constants.js` everywhere else.
Add lazy visitor to `oxc-parser`.
This API is experimental and not ready for use yet. AST is not ESTree-compatible at present (same as lazy deserializer).
Usage:
```js
import { parseSync, experimentalGetLazyVisitor } from 'oxc-parser';
// Get `Visitor` class.
// `Visitor` is not exported directly from `oxc-parser` as all code
// related to lazy deserialization is lazy-loaded.
const Visitor = experimentalGetLazyVisitor();
// Define visitor.
// This visitor counts how many identifiers in AST.
let identCount = 0;
const visitor = new Visitor({
BindingIdentifier(ident) {
identCount++;
},
IdentifierReference(ident) {
identCount++;
},
IdentifierName(ident) {
identCount++;
},
});
const code = 'let x, y; x = { z: 2 };';
const { visit, dispose } = parseSync('test.js', code, { experimentalLazy: true });
visit(visitor);
assert(identCount === 4);
// Dispose of AST now we're done with it.
// This returns buffer storing AST data to the pool for reuse.
dispose();
```
First iteration of lazy deserialization. It works (does not throw an error on any of Test262 or TS test cases), but has some major shortcomings at present:
* Does not match ESTree - shape of AST is more like our Rust types.
* Some missing fields.
* `Vec`s are not lazily deserialized.
* No caching of nodes, so you get a different object each time you get a node e.g. `assert(binaryExpression.left == binaryExpression.left)` fails!
Note: This is a lazy deserializer, which is first step towards a lazy visitor. But there's no visitor implemented yet.
I will build on this starting point in future PRs.
close: https://github.com/oxc-project/oxc/pull/11339 (Another approach is the memory address-based approach, but it is not ideal and has some problems.)
This PR introduces `AstNodes` and `AstNode` to replace the `ParentStack`, and changes that implement `Format` and `FormatWrite` traits from `AST`s to `AstNode` so that now we can access parent by `self.parent` during formatting. All other changes stem from the main one.
```rs
pub enum AstNodes<'a> {
Dummy(),
Program(&'a AstNode<'a, Program<'a>>),
IdentifierName(&'a AstNode<'a, IdentifierName<'a>>),
...
}
pub struct AstNode<'a, T> {
inner: &'a T,
pub parent: &'a AstNodes<'a>,
allocator: &'a Allocator,
}
```
### What is the purpose of `AstNode`?
The `AstNode` aims to get the parent node easily. Before we use `ParentStack` to get the parent node correctly, which has a significant problem, that is, we have to push ancestor nodes sequentially for a node, whereas the implementation of some ASTs can't meet this condition.
For example, in the `CallExpression`, we have special logic to print `Arguments` if the first argument is an `ArrowFunctionExpression`; if so, we will print `ArrowFunctionExpression` directly rather than printing a `Argument`, which causes us to lose a parent node `Argument`.
The above issue is exactly what we aim to address with `AstNode`.
### How does `AstNode` solve the above issue?
As you can see from the `AstNode` struct, which is shown above, every AstNode has `inner` and `parent` fields. The `inner` field stores the original AST, which would be used to print the child of itself, and the `parent`, which is `AstNodes` and points to the parent of the current AST, so that we can know what the parent is.
Let's take a look at how we print the children of the `Program`.
```rs
impl<'a> AstNode<'a, Program<'a>> {
...
#[inline]
pub fn hashbang(&self) -> Option<&AstNode<'a, Hashbang<'a>>> {
self.allocator
.alloc(self.inner.hashbang.as_ref().map(|inner| AstNode {
inner,
allocator: self.allocator,
parent: self.allocator.alloc(AstNodes::Program(transmute_self(self))),
}))
.as_ref()
}
#[inline]
pub fn directives(&self) -> &AstNode<'a, Vec<'a, Directive<'a>>> {
self.allocator.alloc(AstNode {
inner: &self.inner.directives,
allocator: self.allocator,
parent: self.allocator.alloc(AstNodes::Program(transmute_self(self))),
})
}
#[inline]
pub fn body(&self) -> &AstNode<'a, Vec<'a, Statement<'a>>> {
self.allocator.alloc(AstNode {
inner: &self.inner.body,
allocator: self.allocator,
parent: self.allocator.alloc(AstNodes::Program(transmute_self(self))),
})
}
}
// Printing
impl<'a> FormatWrite<'a> for AstNode<'a, Program<'a>> {
fn write(&self, f: &mut Formatter<'_, 'a>) -> FormatResult<()> {
write!(
f,
[
self.hashbang(),
self.directives(),
self.body(),
hard_line_break()
]
)
}
}
```
As you can see, now we implement `FormatWrite` trait for the `AstNode<'a, Prgoram<'a>>` rather than `Program<'a>`, and we need to call the `AstNode`'s methods to get the child `AstNode` to print, which converts `inner`'s child nodes to `AstNode` and binds the parent. That is to say that we handle `AstNode` all along.
NOTE: All of the field-to-AstNode methods are generated by the ast tools and generated in [PLACEHOLDER].
### Worth noting things
#### As little as possible to get child nodes by those methods
Since getting a child node from those methods has some overhead, as we need to wrap an `AstNode` for the original AST. It would be better to only use those methods to get the child nodes when they need to be printed.
#### Efficiently check certain fields
Same as the above, to avoid unnecessary overhead, it would be better to get the original AST to check if you want to check something. For example, if you want to check the first statement of the `body` of the `Program` if it's an `EmptyStatement`, you can check by `self.as_ref().body.first()` rather than `self.body().first()`.
```rs
impl<'a, T> AsRef<T> for AstNode<'a, T> {
fn as_ref(&self) -> &T {
self.inner
}
}
```
`as_ref()` is to get the original AST.
#### `Deref` trait for the `AstNode`
```rs
impl<'a, T> Deref for AstNode<'a, T> {
type Target = T;
fn deref(&self) -> &Self::Target {
self.inner
}
}
```
Since we implemented the `Deref` trait for the `AstNode`, now we can directly call the original AST methods by `self.method()` rather than `self.as_ref() method()` which can reduce a lot of `as_ref()` calls.
But there is a footgun, after `Deref` was implemented, both `self.body` and `self.body()` are work, but only `self.body()` can be printed because only `AstNode` implements `Format` trait, so this may make the new contributor who unfamiliar with the underly design to struggle. I will keep an eye on this to see whether it should be removed.
#### `as_ast_nodes` method for enum AST, such as `Statement`, `Expression`, and so on.
In general, we don't need to call `as_ast_nodes` because in most cases we only need to call `fmt` for the inner AST of the enum AST. But in some cases, we need to deal with the specific thing for a certain inner node of the enum AST.
For example, if the `Expression` is a `ClassExpression`, you need to print the decorators of that class first, then handle the class somewhere. In that time, you need to use `as_ast_nodes` to get the inner node which is converted to the `AstNode`, the code lis ike:
```rs
impl<'a> AstNode<'a, Expression<'a>> {
#[inline]
pub fn as_ast_nodes(&self) -> &AstNodes<'a> {
let parent = self.parent;
let node = match self.inner {
Expression::ClassExpression(s) => AstNodes::Class(self.allocator.alloc(AstNode {
inner: s.as_ref(),
parent,
allocator: self.allocator,
}))
...
};
self.allocator.alloc(node)
}
}
if let AstNodes::Class(class) = expr.as_ast_nodes() {
write!(f, class.decorators())
}
```
### Performance
The initial expectation was a performance improvement due to removing a large `Vec` (previously used for parent kinds) and eliminating indexed parent lookups. However, this has not been observed.
The performance regression might stem from the overhead of wrapping original ASTs into `AstNode` instances, which involves frequent `Allocator::alloc` calls. Additionally, the current implementation is incomplete, with limited parent-checking logic, so the benefits of `AstNode` may not be fully realized yet.
### Downsides
The extensive use of generic structs and the generation of a large file (over 8000 lines) have led to an increase in binary size and compilation time, though the increases are not excessive.
#### Binary Size
* Before: 4.8 MB
* After: 6.9 MB
#### Compilation Time (Release Profile)
* Before: 6.67s
* After: 7.41s
First step towards #11213.
Introduce `CommentNodeId`, which we'll add to AST structs.
Note: `CommentNodeId` wraps a `u32`, not a `NonMaxU32`. `CommentNodeId`s will be created in parser, every node will have one, and they will never change thereafter. So we don't need to wrap in `Cell<Option<CommentNodeId>>`, the way we do with e.g. `ScopeId`. So `NonMaxU32` offers no benefit here, and would be slightly more costly.
Add a method `get_comment_node_id` to `AstBuilder` which other `AstBuilder` methods will call to populate a field with a `CommentNodeId` field. This method is currently a dummy.
`CommentNodeId` is marked `#[clone_in(default)]`, `#[content_eq(skip)]`, `#[estree(skip)]`, so adding this field to AST types will not disrupt any other code.
All our AST structs are `#[repr(C)]`, which gives us guaranteed stable memory layouts. This is essential for raw transfer, and helpful for other purposes too.
However, the downside is that we lose out on the automatic field-reordering that default `#[repr(Rust)]` provides, where structs are packed to minimize padding bytes.
This PR implements a similar field-reordering scheme, to get the same effect.
* `oxc_ast_tools` determines optimal field order for compact memory layout.
* `oxc_ast_tools` generates a file (`structs.rs`) which details the desired field order for each type.
* `#[ast]` proc macro follows the instructions in `structs.rs` and re-orders the fields.
This reduces the size of various AST types by 8 bytes. No type now has more than 7 padding bytes (previously e.g. `Function` had 15 bytes of padding).
Possible future improvements:
1. Get Rust Analyser to display original field order.
2. Remove `syn` dependency from `oxc_ast_macros` crate, to improve compile times. It should be possible to do all the AST manipulation using just `proc_macro` crate, by generating more code in `oxc_ast_tools`, so the macro's logic can be simpler.
Note: This PR doesn't add `syn` dependency, it was already there. But it makes it easier to remove it.
It's unnecessary to have a separate `errors.rs` file for `SourceType`. Combine 2 files into 1.
Pure refactor. This PR doesn't alter any of the logic, just relocates it.
Pure refactor. `serialize.rs` had got too large. Split it up into multiple files.
Also make sure all custom serializers/converters have clear doc comments.
`Pattern` and other types in `oxc_regular_expression` are not part of ESTree AST. Don't implement `ESTree`, and don't generate raw transfer deserializers for these types. None of that code is actually used.
This reduces the scope of ESTree serialization.
overlookmotel