`oxc_lexer` has 2 implementations:
1\. SIMD-based on x86_64 with AVX2.
2\. Scalar/SWAR fallback which works on all platforms.
Both produce the same output.
This PR enables running the lexer conformance on either implementation. Which implementation is tested depends on what build flags are passed to `cargo coverage`.
It turns out that it's possible to test the SIMD implementation on Mac by compiling as x86_64, and running it under Rosetta 2.
This PR also adds `just` recipes to run tests and conformance on either implementation:
- `just test-lexer`
- `just test-lexer-simd`
- `just conformance-lexer`
- `just conformance-lexer-simd`
These commands should work correctly on any of Linux x86, Windows x86, Mac aarch64.
Closes the migration tracked in #23043.
The new builder API (methods on the AST types themselves) has been in place for a while, all Oxc crates are migrated, and every crate which uses `AstBuilder` already enables `disable_old_builder` in its tests. The module docs said "after a few weeks, we will remove `AstBuilder`'s own methods entirely" — this does that.
## What's removed
- `crates/oxc_ast/src/generated/ast_builder.rs` and `builder/methods.rs` — `AstBuilder`'s own node and primitive constructors.
- `AstBuilderGenerator`, which produced the former.
- The `disable_old_builder` Cargo feature, plus the dev-dependency hacks it required across 33 manifests. Eight crates depended on `oxc_ast` *purely* to turn the feature on; those deps and their `cargo-shear` ignore entries are gone too.
## One thing worth reviewing
`AstBuilderGenerator` owned the `#[builder(default)]` / `#[builder(skip)]` attr registration that `BuilderMethodsGenerator` only *reads* — its doc comment said as much. Deleting it naively would have silently dropped `skip`/`default` handling for the 12 AST types using those attrs, so `attrs()` and `parse_attr()` move to `BuilderMethodsGenerator`.
The evidence this was faithful: after regenerating, `builder_methods.rs` and every other generated file are **byte-identical**. The only generated change in the diff is the deletion.
## Breaking changes
- `AstBuilder`'s own methods are gone. Use the AST type methods: `Statement::new_expression_statement(span, expr, &builder)` instead of `builder.statement_expression(span, expr)`.
- `AstBuilder` is no longer `Copy`/`Clone`, and its `allocator` field is private — use the `allocator` method from `GetAllocator`.
- `AstBuilder` and `NONE` are no longer re-exported from the crate root; import them from `oxc_ast::builder`. (The `disable_old_builder` feature already gated these off, so nothing in-tree used the root path.)
- The `disable_old_builder` feature no longer exists — remove it from any manifest that sets it.
A migration note covering all of the above is in `crates/oxc_ast/src/builder/mod.rs`.
---
Implemented with AI assistance (Claude Code); design, review, and verification driven by the author per the repo AI-usage policy.
Adds the x86_64 version of my Research `SIMD For Context Sensitive
Languages`.
(CPB = Cycles per byte)
- Adds new create crates/oxc_lexer; Which eventually lead into an
unfused parser design inside oxc_parser
- Fully test262 compliant JS standalone Lexer; SIMD used has
counterparts on every platform.
- Performs at 2.5CPB on checker.js on my ZEN 3 CPU; or around 1.5-1.75
gbs (3.8GHz).
- Across aarch64, + other platforms performance will hold which I've
seen in my POCs but those targets will come later.
- Supports JSX/TS options, Which don't have much impact on overall perf
results.
------
The expected results for seeing this work through [including on parser
side on oxc_parser]
1. P95 improvements of 80-100% over current `oxc_parser`; 12-18% of this
will come from the unfused contract where JS grammar specifically loves
being able to see 'ahead' into `kinds` in various places. The rest comes
from the perf improvements to the lexer itself and some wins that will
come inside the parser from work now done in the lexer.
Part of #23043.
We don't enable `oxc_ast`'s `disable_old_builder` Cargo feature in any of our published crates, or crates that are depended on by published crates.
Tasks are private to the repo, so enable this feature on them, to ensure no usage of legacy `AstBuilder` methods sneak in during the transition period before we delete these methods.
Split `oxc_formatter` into:
- `oxc_formatter(_js)`: JS/TS specific formatter, comment handling, sortImports, jsdoc, etc
- `oxc_formatter_core`: Language agnostic IR, printer, etc
The naming convention is different from Biome or Ruff (they do not have `_core`, and `_formatter` is equivalent to the core), but this is because in Oxc, items without `_js` are the default.
As a basic task, I have added `Context` as a generic to all relevant parts to ensure support for multiple languages.
There might still be other components that could be refactored into the core, but I would like to finish them as I implement support for other languages.
Also moved some shared utilities to the core in advance.
Implement a `collect_tokens` option for parser. When enabled, the parser stores all tokens produced during parsing in a `Vec`. Implement serializing the tokens to ESTree format.
Currently this is not hooked up to NAPI parser or linter. That happens in a later PR in this stack (#19498).
Serialization of tokens implementation is in new `oxc_estree_tokens` crate. Unfortunately it can't go in existing `oxc_estree` crate because `oxc_ast` depends on `oxc_estree`, and the tokens serialization code requires AST types - which would be a circular dependency. We may be able to move it into another crate (parser?) later on.
- Extract `path.to_string_lossy()` before error loops to avoid repeated conversions
- Use `as_str()` instead of `clone()` in BaselineFile::print()
- Remove unnecessary clone in BaselineFile::snapshot()
- Use `map_or_else` to avoid allocating "Panicked" string in success paths
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
## Summary
- Disable `test = false` and `doctest = false` for crates that have no tests
- Reduces CI noise from "0 passed; 0 failed" output
- Saves compilation time for empty test targets
**Affected crates:**
- `oxc_ast_macros` - lib test
- `oxc_language_server` - bin test, lib doctest
- `oxc_napi` - lib test
- `oxc_syntax` - lib doctest
- `oxc_compat_data` - bin test
- `oxc_coverage` - lib test
- `oxc_linter_codegen` - bin test
- `oxc_prettier_conformance` - lib test
- `oxc_transform_conformance` - lib test
- `vscode_docs` - bin test
- `website_common` - lib test, lib doctest
## Test plan
- [x] `cargo test` passes with no empty test suites
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Improved the error file collection logic to pick up files like `importMeta(module=system,target=es5).errors.txt`. Also added support for `preserveconstenums` and `usedefineforclassfields`.
Add a Cargo feature `conformance` to `oxc_ast_visit` crate.
When this feature is enabled, it exposes a method `Utf8ToUtf16::convert_program_with_ascending_order_checks` which panics if span offsets are converted out of order.
Use this feature and method in ESTree conformance tests. This ensures:
1. `Utf8ToUtf16Converter` visitor is functioning as intended.
2. All AST types have their fields ordered in order they appear in source code.
We intentionally add a `hashbang` field to `Program`, which isn't present in ESTree or TS-ESTree AST. This causes some tests to erroneously fail. So skip them.
#10669 adds hashbangs to `comments` in ESTree AST to align with Acorn, and adds tests relating to hashbangs to `napi/parser`, so our test coverage for hashbangs is maintained, despite skipping these tests.
Part of #5068, suggested by @ematipico and @MichaReiser, where @leaysgur
and I explored using biome formatter as the base and made a success. All
credits to the old Rome and the new Biome team.
This PR is in a boilerplate state where some of the functions are marked
as `todo!()`.
I also made the following changes:
* changed the base to be less generic, removed the `Language` and
`Context` generics
* expanded the formatter lifetime to include an `'ast` lifetime
* added access to the source text and comments
* added a mechanism for tracking and accessing parent ast nodes (done by
our codegen)
* removed most of the unit tests because they no longer compile
* removed the `oxc_prettier` crate
Use the `#[estree(field_order(...))]` attribute introduced in #9127 to re-order fields in the ESTree AST so they match Acorn exactly.
Make the ESTree conformance tester stricter - it now errors on incorrect field order.
All conformance tests still passing after this PR.
Fix ESTree conformance tester, to handle deeply nested JSON. Previously some tests were failing only because the AST's depth exceeded `serde_json`'s default recursion limit.
Move post-transform checker into a `tasks` crate. It doesn't feel like it belongs in `oxc_semantic`. It also feels like too heavy a lump of code to put in `tasks/common`.
closes https://github.com/oxc-project/oxc/issues/4790
@overlookmotel enjoy ... take a look at the snapshots and probably nothing else.
The snapshots are minimal right now, but it's already showing symbols from import specifiers are not being removed. We can iterate on the snapshot representation to aid debugging later.
I'll extend this to `transformer_conformance` and `oxc-monitor` in an up coming PR.
The sourcemap implement port from
[rust-sourcemap](https://github.com/getsentry/rust-sourcemap), but has
some different with it.
- Encode sourcemap at parallel, including quote `sourceContent` and
encode token to `vlq` mappings.
- Avoid `Sourcemap` some methods overhead, like `SourceMap::tokens()`
caused extra overhead at common cases. Here using `SourceViewToken` to
instead of it.
This PR merges the previous confusing features `serde` and `wasm` into a
single `serialize` feature.
We'll eventually do serialize + type information for both wasm and napi
targets.
`oxc_macros` is removed from `oxc_ast`'s dependency because it requires
`syn` and friends, which goes against our policy ["Third-party
dependencies should be
minimal."](https://oxc-project.github.io/docs/contribute/rules.html#development-policy)
The runtime performance gains does not out weight the compilation speed from
building the custom allocators, which takes about a minute to build on
slower machines.
overlookmotel