Introduce `ESTreeTokenConfig` trait and allow option for the "flavor" of tokens to be set at compile time.
When producing Espree-style tokens, this allows cutting out all the special-case logic related to `JSXIdentifier` tokens and removing 2 branches from `emit_identifier_at`.
Small effect on perf - only +0.3% on ESTree tokens benchmark.
Previously we used a 3-pass process for serializing tokens in ESTree format.
* Pass 1: Walk AST and produce a list of "overrides" - which tokens need their `Kind` changed.
* Pass 2: Loop through all tokens, apply any overrides, convert to `ESTreeToken`s, and save them in a `Vec`.
* Pass 3: Serialize the `Vec<ESTreeToken>` with `ESTree` trait (JSON serializer).
Compress this all to a single pass:
* Create an `ESTreeSequenceSerializer` which can have items sent to it to serialize one-by-one (streaming).
* Walk AST.
* When encounter a token which needs its `Kind` altered:
* Flush all previous tokens, serializing each in turn, using its default `Kind`.
* Serialize the current token with the amended `Kind`.
* Continue walking.
* At the end, flush any remaining tokens to the serializer.
This avoids multiple passes, and also removes the `Vec<ESTreeToken>` which was just intermediate data, and grew very large.
+18% perf improvement on ESTree tokens benchmark.
Refactor.
Previously we converted spans from UTF-8 to UTF-16 in AST first, then serialized tokens. This meant that when serializing tokens, we had to convert spans to UTF-16 before comparing offsets of overrides to the token's start offset (because the override offsets are taken from the AST nodes during the AST walk pass).
Instead, convert token spans first, *then* the AST. This removes span conversion from the critical path. On a CPU with instruction-level parallelism, other work can start without being blocked on waiting for the result of span conversion, and the two can happen in parallel.
This should be a slight perf improvement, though it likely won't show up on CodSpeed benchmarks as CodSpeed's CPU simulation to basic to emulate instruction-level parallelism.
But mainly, the motivation is that this makes the process simpler and easier to reason about.
## Summary
- When `#ident` appears as a property key in invalid positions (object literals `{#x: 1}`, destructuring `const {#x} = y`, TS interface members), the parser previously fell into `parse_identifier_name()` producing an unhelpful "Unexpected token" error
- Added a `Kind::PrivateIdentifier` arm in `parse_property_name()` that creates a proper `PropertyKey::PrivateIdentifier` AST node and emits "Private identifier '#x' is not allowed in property names"
- This works correctly both inside and outside class bodies, with no conformance regressions
closes#19669🤖 Generated with [Claude Code](https://claude.com/claude-code)
Oxc's AST on JS side has a non-standard `hashbang` property. Previously in ESTree conformance tests, we skipped files with a hashbang.
Bump `estree-conformance` submodule to include https://github.com/oxc-project/estree-conformance/pull/183 which adds `hashbang` property to AST in snapshots.
We can now enable the tests which we previously skipped.
Our convention is that object properties are in source code order. Reverse `pattern` and `flags` properties of `regex` objects in ESTree tokens to align with this pattern.
Bump `estree-conformance` submodule to include https://github.com/oxc-project/estree-conformance/pull/182 which makes the same change in snapshots for conformance tests.
Perf optimization to converting tokens to ESTree JSON. Previously we serialized to pretty-printed JSON. Instead use compact JSON - it's just passed to `JSON.parse` on JS side, so there's no need for whitespace.
Fix a bug where we didn't produce any tokens for files which start with a BOM.
As a side effect, re-use the existing `Utf8ToUtf16` span converter created for converting spans in the AST again for tokens, rather than creating another one.
Refactor. We had pointlessly complex logic where caller would call `collect_token_context` and then pass the result back into some other tokens code `to_estree_tokens_json`. Combine the 2 functions to make the call sites simpler.
Pure refactor of conformance tests for ESTree and ESTree tokens. Use the same order of statements, and same variable names in each test function, to make it easier to follow.
We were skipping some ESTree tokens conformance tests for files containing hashbangs. We have to skip these tests for ESTree AST tests due to our non-standard `hashbang` field on `Program`, but we don't have to skip them for tokens tests - hashbangs are handled correctly.
Fix tokens when parsing JSX (not TSX) so they match `acorn-jsx` test cases. TS-ESLint's tokens are clearly wrong. We continue to match TS-ESLint bug-for-bug, but process plain JSX differently, so both the TS and JSX tests pass.
The only 2 remaining failing JSX test cases are due to #9667. Both these cases are also failing in the ESTree AST conformance tests.
## Improve `undefined_export` and related errors with name suggestions
This PR adds helpful "did you mean?" suggestions to errors where names are not found, using Levenshtein distance (threshold=2).
### Example Output
**undefined_export with suggestion:**
```
Export 'myVarible' is not defined
help: Did you mean 'myVariable'?
```
**private_field_undeclared with suggestion:**
```
Private field '#privateFild' must be declared in an enclosing class
help: Did you mean '#privateField'?
```
**invalid_label_target with suggestion:**
```
Use of undefined label
help: Did you mean 'outerLoop'?
```
### Implementation Details
- Moved Levenshtein edit distance calculation from `oxc_linter` to `oxc_span` for reuse across crates
- Uses a threshold of 2 for edit distance to suggest similar names
- Collects available names from appropriate scopes only when errors are found (performance optimization)
<!-- START COPILOT CODING AGENT SUFFIX -->
<!-- START COPILOT ORIGINAL PROMPT -->
<details>
<summary>Original prompt</summary>
> Improve `undefined_export` and some other errors to include a hint that suggests similar names that are defined
</details>
<!-- START COPILOT CODING AGENT TIPS -->
---
💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more [Copilot coding agent tips](https://gh.io/copilot-coding-agent-tips) in the docs.
### What this PR does
Introduce `ParserConfig` trait (another try at #16785).
The aim is to remove the large performance regression in parser that #19497 created, by making whether the parser generates tokens or not a compile-time option.
`ParserConfig::tokens` method replaces `ParseOptions::collect_tokens` property. The former can be const-folded at compile time, where the latter couldn't.
### 3 options
This PR also introduces 3 different config types that users can pass to the parser:
* `NoTokensParserConfig` (default)
* `TokensParserConfig`
* `RuntimeParserConfig`
The first 2 set whether tokens are collected or not at compile time. The last sets it at runtime.
All 3 implement `ParserConfig`.
`NoTokensParserConfig` is the default, and is what's used in compiler pipeline. It switches tokens off in the parser, and makes all the tokens-related code dead code, which the compiler eliminates. This makes the ability of the parser to generate tokens zero cost when it's not used (in the compiler pipeline).
`TokensParserConfig` is the one to use where you *always* want tokens. This is probably the config that linter will use.
`RuntimeParserConfig` is the one to use when an application decides whether to generate tokens or not at runtime. This config avoids compiling the parser twice, at the cost of runtime checks. This is what NAPI parser package will use.
### Future extension
#### Supporting additional features
In future we intend to build the UTF-8 to UTF-16 offsets conversion table in the parser. This will be more performant than searching through the source text for unicode characters in a 2nd pass later on. But this feature is only required for uses of the parser where we're interacting with JS side (NAPI parser package, linter with JS plugins).
`ParserConfig` can be extended to toggle this feature on/off at compile time or runtime, in the same way as you toggle on/off tokens.
#### Options and configs
This PR introduces `ParserConfig` but leaves `ParseOptions` as it is. So we now have 2 sets of options, passed to `Parser` with `with_options(...)` and `with_config(...)`. This is confusing.
We could merge the 2 by making `ParseOptions` implement `ParserConfig`, so then you'd define all options with one `with_options` call.
This would have the side effect of making all other parser options (e.g. `preserve_parens`) able to be set at either runtime or compile time, depending on the use case.
For users consuming `oxc_parser` as a library, with specific needs, they could also configure `Parser` to their needs e.g. create a parser which only handles plain JS code with all the code paths for JSX and TS shaken out as dead code. This would likely improve parsing speed significantly for these use cases.
### Implementation details
Why a trait instead of a cargo feature?
IMO a trait is preferable for the following reasons:
1. We have 3 different use cases we need to support (the 3 provided configs). 3 different Cargo features would be unwieldy.
2. This situation would become far worse once we introduce more features e.g. UTF-8 -> UTF-16 conversion.
3. No problems around feature unification. We found Cargo features caused headaches when we used them in linter for toggling on/off JS plugins support.
4. No clippy errors which only appear when the feature is/isn't disabled, requiring complex `#[cfg_attr(feature = "whatever", expect(clippy::unused_async))]` etc.
The introduction of a trait does not seem to significantly affect compile time:
```
# Before this PR
cargo build -p oxc_parser 15.33s user 3.27s system 254% cpu 7.316 total
cargo build -p oxc_parser --release 17.36s user 2.26s system 231% cpu 8.477 total
cargo build -p oxc_parser --example parser 18.43s user 3.75s system 271% cpu 8.156 total
cargo build -p oxc_parser --example parser --release 32.52s user 2.59s system 180% cpu 19.454 total
# After this PR
cargo build -p oxc_parser 15.00s user 3.24s system 272% cpu 6.692 total
cargo build -p oxc_parser --release 16.71s user 2.12s system 287% cpu 6.539 total
cargo build -p oxc_parser --example parser 18.50s user 3.91s system 285% cpu 7.845 total
cargo build -p oxc_parser --example parser --release 33.48s user 2.63s system 169% cpu 21.263 total
```
Measured on Mac Mini M4 Pro, `cargo clean` run before each. The difference appears to be mostly within the noise threshold.
Implement a `collect_tokens` option for parser. When enabled, the parser stores all tokens produced during parsing in a `Vec`. Implement serializing the tokens to ESTree format.
Currently this is not hooked up to NAPI parser or linter. That happens in a later PR in this stack (#19498).
Serialization of tokens implementation is in new `oxc_estree_tokens` crate. Unfortunately it can't go in existing `oxc_estree` crate because `oxc_ast` depends on `oxc_estree`, and the tokens serialization code requires AST types - which would be a circular dependency. We may be able to move it into another crate (parser?) later on.
Move the removal of TS-only nodes earlier from `exit_statements` to `enter_statements` to avoid deep traversal into the TS-only child nodes, which are never used.
## Summary
- Replace `Ident<'a>` (transparent `&'a str` wrapper) with a `#[repr(C)]` struct storing `NonNull<u8>` + packed `len_and_hash: u64` on 64-bit (16 bytes, same as `&str`)
- Add ~1ns head-tail XOR hash function (`ident_hash`) that reads first 4 + last 4 bytes, discriminating identifiers like `useState`/`useEffect`
- Add `IdentBuildHasher`/`IdentHasher` custom hasher that handles both precomputed `Ident::hash()` and `str::hash()` lookup paths
- `PartialEq<Ident>` fast-rejects via single `u64` compare before byte comparison
- Parser creates `Ident` directly in `parse_identifier_kind` (hash computed once, not recomputed on Atom→Ident conversion)
- All `Ident`-keyed HashMaps use `IdentBuildHasher` (`IdentHashMap`, `ArenaIdentHashMap`, `IdentHashSet`)
- Generalize arena `HashMap::new_in` and `CloneIn` to work with any `S: Default + BuildHasher`
Ref: https://github.com/oxc-project/backlog/issues/193🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
- In JavaScript modules, `import { Foo } from './foo.js'; var Foo;` (and `let`, `const`, `class`, `function`) should report `"Identifier 'Foo' has already been declared"` per the [ES spec](https://tc39.es/ecma262/#sec-imports-static-semantics-early-errors)
- The existing excludes-based conflict detection in the binder doesn't catch this because `Import` is not in the `Value` bitset
- Added a checker function that runs at `Program` leave, scanning root scope bindings for symbols with both `Import` and a value flag, and reports a redeclaration error
- TypeScript is excluded since it has its own declaration merging rules
Closes#15961🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
- Add early return when current scope has no unresolved references
- Skip resolution entirely for scopes with no bindings (if-blocks, try-blocks, loop bodies without `let`) — merge unresolved refs to parent using O(1) swap when possible
- Use union-by-size merge (drain smaller map into larger) for non-empty parents
- Add `#[inline]` to hot per-reference functions: `declare_reference`, `reference_identifier`, `create_reference`
### Benchmark results (semantic)
```
group main pr
----- ---- --
semantic/RadixUIAdoptionSection.jsx 1.05 4.1±0.05µs ? ?/sec 1.00 3.9±0.03µs ? ?/sec
semantic/binder.ts 1.03 469.3±26.69µs ? ?/sec 1.00 456.3±17.59µs ? ?/sec
semantic/cal.com.tsx 1.00 3.6±0.03ms ? ?/sec 1.01 3.6±0.04ms ? ?/sec
semantic/react.development.js 1.01 171.9±3.86µs ? ?/sec 1.00 170.0±9.19µs ? ?/sec
```
### Benchmark results (pipeline)
```
group main pr
----- ---- --
pipeline/RadixUIAdoptionSection.jsx 1.03 29.8±0.35µs ? ?/sec 1.00 29.0±0.25µs ? ?/sec
pipeline/binder.ts 1.00 2.9±0.07ms ? ?/sec 1.00 2.9±0.09ms ? ?/sec
pipeline/cal.com.tsx 1.00 25.0±0.32ms ? ?/sec 1.01 25.2±0.36ms ? ?/sec
pipeline/react.development.js 1.03 1317.1±96.08µs ? ?/sec 1.00 1273.5±18.26µs ? ?/sec
```
### Analysis
Scope resolution (`resolve_references_for_current_scope`) is called on every scope exit. Many scopes (if-blocks, try-blocks, loop bodies) have zero bindings, yet the current code still drains all unresolved references one-by-one through hash maps. The swap optimization makes this O(1) for the common case where the parent has no accumulated refs yet (first/only child scope).
The `#[inline]` additions target functions called per-identifier-reference (~hundreds to thousands of times per file) that were not previously inlined.
Note: 3 linter snapshot ordering changes expected (diagnostics emitted in different order due to hash map iteration order change from swap).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
- Port TypeScript's binary file detection (`TS1490: File appears to be binary.`)
- When the scanner encounters U+FFFD (replacement character) as a standalone token, emit the error and stop parsing
- U+FFFD inside strings, comments, and templates is unaffected
- Reference: https://github.com/microsoft/TypeScript/blob/main/src/compiler/scanner.ts🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
- Ignore `usingDeclarations.14.ts` in TypeScript conformance tests
- TypeScript allows top-level `using` in scripts (no import/export), but oxc rejects it per TC39 spec — `using` is not allowed at the top level of scripts
- This brings parser_typescript conformance to 100.00%
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
- Exclude `asyncWithVarShadowing_es6.ts` from TypeScript conformance tests
- TSC allows `catch({x}) { var x; }` (destructured catch param + var redeclaration), but per Annex B §B.3.4 only simple identifier catch params are exempt from the redeclaration error
- OXC correctly follows the spec, so this is a known behavioral difference
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
- Don't report "Function implementation is missing" (TS2391) for computed property name methods like `[e]()` that aren't part of an overload group, matching TypeScript's behavior
- TypeScript only checks TS2391 when it can resolve the method to a symbol, which isn't possible for dynamic computed names
- Computed-name overload groups like `[Symbol.iterator]` still correctly report the error
**TypeScript**: Positive passed 9839 → 9841 (+2)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
- Replace `check_for_duplicate_modifiers` with `check_modifier` that
handles both duplicate detection and ordering validation in a single
match statement, following TypeScript's `checkGrammarModifiers` logic
- Correctly reports TS1029 (`'x' modifier must precede 'y' modifier`)
for out-of-order class member modifiers
- Remove redundant static/readonly ordering check for index signatures
in `class.rs` (now handled centrally)
- Remove unused `contains_all_flags` method from `Modifiers`
**Babel**: Negative passed 1660 → 1665 (+5)
**TypeScript**: Negative passed 1515 → 1529 (+14)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Implement package.json `"type"` field lookup for `moduleDetection: auto` in TypeScript conformance tests, matching TypeScript's `getPackageScopeForPath` behavior
- When `moduleDetection` is unset and `module` is `node16`..`nodenext`, default to `force` (matching TypeScript's `_computedOptions.moduleDetection`)
- Refactor stringly-typed `get_module_detection_mode` → `is_forced_module` returning `bool`
The estree_typescript snapshot regresses because the baselines in the estree-conformance submodule were generated with the same old incorrect logic. The JS port (`typescript-make-units-from-test.cjs`) needs the same fix and baselines need regeneration.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
- Add `decorators` field to `FormalParameterRest` to support decorators on rest parameters in TypeScript
- Parse decorators before checking for rest parameters in formal parameter lists
- Update semantic builder to visit decorators on rest parameters
Fixes parsing of `typescript/tests/cases/compiler/sourceMapValidationDecorators.ts`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
- Extract `path.to_string_lossy()` before error loops to avoid repeated conversions
- Use `as_str()` instead of `clone()` in BaselineFile::print()
- Remove unnecessary clone in BaselineFile::snapshot()
- Use `map_or_else` to avoid allocating "Panicked" string in success paths
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
## Summary
- Allow conditional types inside function type parameter annotations, even when the function type itself is being parsed in a `DisallowConditionalTypes` context
- Wrap `parse_formal_parameters` in `parse_function_or_constructor_type` with `context_remove(DisallowConditionalTypes, ...)`
This fixes parsing of nested conditional types like:
```typescript
type Equals = A extends (x: B extends C ? D : E) => 0 ? F : G;
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
- Add TypeScript error TS1477: "An instantiation expression cannot be
followed by a property access"
- Check for `TSInstantiationExpression` before parsing property access
(static `.` or computed `[]`)
This fixes parsing of invalid syntax like:
- `a<b>.c` (static member access)
- `a<b>?.c` (optional static member access)
- `a<b>?.[c]` (optional computed member access)
- `a?.b<c>.d` (chained)
Valid syntax like `a<b>?.(c)` (optional call) and `(a<b>).c`
(parenthesized) are not affected.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
overlookmotel