#18089 added support in parser for `commonjs` source type.
Extend this support to the NAPI packages - accept `sourceType: 'commonjs'` in options for `oxc-parser` and `oxc-transform`.
Refactor tests. The tests for `ImportDeclaration` and `ImportExpression` had somehow got nested within the tests for `TemplateExpression`. Move them to a better place.
## Summary
- Implements speculative/unambiguous parsing for `.js`, `.jsx`, `.ts`, and `.tsx` files
- Instead of assuming module or script upfront, parses with `ModuleKind::Unambiguous` and upgrades to module mode when ESM syntax is detected
- ESM indicators: `import`, `export`, `import.meta` (NOT top-level await alone)
- Defers top-level await errors until module type is resolved
- Vue loader upgrades unambiguous scripts to module mode
- Updates linter tests to use explicit ESM extensions (`.mjs`/`.mts`) where needed
## Test plan
- [x] All parser tests pass
- [x] All linter tests pass
- [x] All semantic tests pass
- [x] Coverage conformance tests updated
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Follow-on after #16383 and #16403.
While working on those I discovered an option NAPI-RS has to represent `Option::None` as `null`, instead of omitting the field entirely.
Use that option on the structs used for transferring errors and module record over to JS.
I think this is likely more performant because:
1. NAPI-RS can create all properties of the objects using only the faster `node_api_create_object_with_properties` API.
2. It produces consistent object shapes, so JS engine can better optimize code using these objects.
It also brings the shape of the data perfectly into line between standard transfer and raw transfer, and is consistent with how empty fields in the AST are represented as `null`.
I would have used this option before if I'd known it existed.
### Breaking change
I've marked this as a breaking change because code consuming these objects would now need to check for empty fields with `value === null` instead of `value === undefined`.
However in practice, most people probably use `!value` or `value ?? ...`, so it's unlikely to affect many users. This only affects module record and errors anyway, not the AST itself, as we transfer that as JSON, not via NAPI.
Use `const` where variables are not reassigned instead of `let`. Preparation for enabling ESLint's `prefer-const` rule (running as an Oxlint JS plugin) in our linting setup.
Now that `apps/oxlint` and `napi/parser` are both ESM packages (#13723 and #14042), we can use `.js` file extensions for all ESM files, rather than the mix of `.js` and `.mjs` file we had previously.
It's less confusing having to remember what's `.js` and what's `.mjs`, and avoids ugly workarounds like #14038 (`.d.mts`??).
Add an AST visitor to `oxc-parser`.
```js
import { parseSync, Visitor } from 'oxc-parser';
const { program } = parseSync('test.js', 'let x = { y: z };');
const idents = [];
const visitor = new Visitor({
Identifier(node) {
idents.push(node.name);
}
});
visitor.visit(program);
// Logs [ 'x', 'y', 'z' ]
console.log(idents);
```
Primarily we need a visitor for Oxlint plugins, but it's easier to develop and test in `oxc-parser`, so I thought I'd add it there first. Oxlint build process will copy the generated files for tree-walking from `napi/parser` dir, as it does with other files.
Notes:
* The `walk.mjs` file is large, so it's lazy-loaded only when user first constructs a `Visitor`. Adding `Visitor` to the package will not affect start-up time where user only uses `parseSync` etc.
* TS type definition for `Visitor` and the visitor object it takes are provided.
* Visitor supports enter and exit visitors e.g. `Program` and `Program:exit`.
* `src-js/visit/visitor.mjs` is copied from `apps/oxlint` with only minor modifications, and conversion from TS to JS. It'd be better if we only had this impl in one place, but that would require converting `oxc-parser` to TS and building with TSDown, like `oxlint` is. Leaving that for later - #13935.
Add visitor keys export to `oxc-parser` NPM package. This is useful in itself, but also the first step towards an ESTree walker.
Visitor keys are based on `@typescript-eslint/visitor-keys`.
It'd be ideal to generate visitor keys direct from Oxc's own types, but that's proving difficult due to all the custom serializers we use to translate to ESTree. So using the shortcut of borrowing from TS-ESLint for now.
Oxc's AST is slightly different from TS-ESTree (notably Oxc's AST adds `ParenthesizedExpression`). But apart from those few differences, TS-ESLint's AST and Oxc's are identical, and I made a PR to TS-ESLint earlier this year (https://github.com/typescript-eslint/typescript-eslint/pull/11279) to ensure visitor key order is correct, and matches Oxc.
To avoid adding a runtime dependency on `@typescript-eslint/visitor-keys`, extract the data from TS-ESLint in `oxc_ast_tools`, amend for Oxc/TS-ESTree differences, and re-output it as a simple object literal.
This approach will be faster at runtime, as the object's shape is consistent, and also avoids a dependency.
Pure refactor. Root of `napi/parser` was getting really crowded with tons of files, and I'm about to add a load more for the ESTree walker.
Move JS source files into a `src-js` directory, like in `apps/oxlint`, so that the root of the package is just config files and scripts.
Part of #10307. Add `range` option to `oxc-parser`, to add `range` property to nodes in ESTree AST.
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: overlookmotel <theoverlookmotel@gmail.com>
Related to #10978. Add `phase` property to `ImportExpression` in TS-ESTree AST.
This aligns with ESTree spec, but does *not* align with TS-ESLint - TS-ESLint outputs a `CallExpression` with a `MetaProperty` as its `callee` instead.
We don't have the necessary information in our AST to match TS-ESLint, because we don't have the spans of `import`, `defer`, or `import.defer`, to use for the `MetaProperty`, or the `IdentifierName`s it contains.
This is not completely ideal, but I figure it's preferable to represent this syntax *somehow* in the AST, rather than completely ignoring it, in which case there's no way to distinguish between `import("x")`, `import.defer("x")`, and `import.source("x")`.
Presumably TS-ESLint will change their AST shape to conform to ESTree spec once these reach stage 4, so I don't think we should expend effort changing our AST to align with them in the meantime. TypeScript doesn't even have test cases for these (we'd know if they did, as we'd be failing them) - so I doubt these are popular constructs in TS.
`acorn-test262` submodule is bumped to include https://github.com/oxc-project/acorn-test262/pull/36 which adds the `phase` field to `ImportExpression`s in test snapshots.
Related to #10978. Add `phase` property to `ImportExpression` in ESTree AST.
This property is *not* added in TS-ESTree, because TS-ESLint supports `import.defer(...)` and `import.source(...)`, but does not align with ESTree - it outputs a `CallExpression` with a `MetaProperty` as its `callee` instead.
We don't have the necessary information in our AST to match TS-ESLint, because we don't have the spans of `import`, `defer`, or `import.defer`, to use for the `MetaProperty`, or the `IdentifierName`s it contains.
Presumably TS-ESLint will change their AST shape to conform to ESTree spec once these reach stage 4, so I don't think we should expend and changing our AST to align with them in the meantime. TypeScript doesn't even have test cases for these (we'd know if they did, as we'd be failing them) - so I doubt these are popular constructs in TS.
`acorn-test262` submodule is bumped to include https://github.com/oxc-project/acorn-test262/pull/31.
Fixes#10978. Add `phase` property to `ImportDeclaration` in ESTree and TS-ESTree ASTs.
`ImportExpression` is not covered by this PR. That's more tricky as TS-ESLint already supports `import.defer(...)` and `import.source(...)`, but does not align with ESTree - it outputs a `MetaProperty` instead.
`acorn-test262` submodule is bumped to include https://github.com/oxc-project/acorn-test262/pull/30.
closes#10980
Previously final source type was only computed when `lang` is not set.
It is changed to:
* compute source type from `lang`, use filename extension if `lang` is
not set
* reset the computed source type with 'script` or `module` if provided
In ESTree (plain ESTree for JS i.e. Acorn), a hashbang is recorded as a line comment. Follow that behavior in our ESTree AST.
In TS-ESTree, hashbangs are ignored. That's not how it looks in AST explorer, because there's some post-processing going on, but `@typescript-eslint/parser` itself ignores hashbangs. So only add the comment in ESTree AST (not TS-ESTree AST).
More info on this rather confusing subtlety: https://github.com/typescript-eslint/typescript-eslint/issues/6500
Encode lone surrogates in `cooked` property of `TemplateElementValue` using same encoding scheme as for `StringLiteral`s.
In fact, they were already being encoded like this after #10041, but add a `lone_surrogates` flag to `TemplateLiteral` to decode them correctly in ESTree AST.
`oxc_codegen` ignores `cooked` and just prints `raw`, so needs no alteration.
Fix 2 edge cases when parsing lone surrogates in `StringLiteral`s:
* Lone surrogate followed by `\u{...}` escape e.g. `"\uD800\u{41}"`.
* Escaped lossy replacement character after lone surrogate e.g. `"\uD800 \u{FFFD}"`.
Add an `astType` option to `parseSync` and `parseAsync`.
When set to `'js'`, the returned AST does not include TypeScript-related properties. When set to `'ts'`, it does.
If not specified, the option defaults to same as source type. So when parsing JS / JSX files, the AST returned does not include TS properties by default.
The motivation for allowing the user to override that behavior, is in case someone is parsing a mix of JS and TS files, and want all the ASTs in the same format. In that case they'd set `astType: 'ts'` for all files.
This is a breaking change as it alters what properties appear in the AST on JS side.
Replace `serde` with a custom serializer specialized for writing JSON.
`impl Serialize` -> `impl ESTree`. I've called it `ESTree` in case we
want to add e.g. `impl Babel` later on.
The main motivation is that with control of the serializer, we can make
it stateful, so we can extend it to, for example, omit TS fields based
on a flag. This is pretty much impossible with `serde`'s interface.
But a side-effect of the simpler implementation is that it's also more
performant.
The usage of `ESTree` trait is very similar to `serde::Serialize`. Main
differences:
* `serialize` method is infallible. Does not return a `Result`.
* `state = serializer.serialize_struct();` instead of `map =
serializer.serialize_map();`.
* `state.serialize_field(...)` instead of `map.serialize_entry(...)`.
* Struct keys must be strings, and JSON-safe (not including `"` or other
characters that require escaping in JSON).
This is a breaking change, as it's possible downstream consumers are
using `serde` to serialize our AST to other formats. I'd guess unlikely,
but possible. We can always implement `serde::Serialize` as well as
`ESTree` further down the line if there's a demand.
I (and mostly copilot) wrote `VisitMutModuleRecord` to visit all spans as I wasn't sure how `VisitMut` is generated. I'm looking into visit generator now if it's something I can do. Please let me know if manually writing it down is fine for now or I should try macro (or maybe a completely different approach).
- closes https://github.com/oxc-project/oxc/issues/9079
I'm still leaving an existing TODO for `module_record` and `errors` span not being converted as they won't cause an immediate issue, but I'll work on that shortly.
Add a "reviver" function to `JSON.parse` in NAPI parser module, to correctly set `value` field of `Literal`s for `RegExp`s and `BigInt`s.
Surprisingly, judging by a local run of NAPI parser benchmarks (added in #9045), this does not seem to hurt performance. In fact the benchmarks were showing a ~2% speed-up. That's clearly nonsense - just noise - but it does suggest at least that this isn't hurting performance significantly.
Boshen