Fix tokens produced when `<<` opens a type argument list. e.g.:
```typescript
type Bar = ReturnType<<T>(x: T) => number>;
```
Previously only 1 `<` token was produced between `ReturnType` and `T`. This PR corrects that so 2 `<` tokens are produced.
TS-ESLint get this wrong too - hence why this wasn't noticed before, as our conformance tests compare to TS-ESLint.
PR submitted to fix TS-ESLint: https://github.com/typescript-eslint/typescript-eslint/pull/12821
In meantime, this PR skips the erroneously failing ESTree tokens conformance tests. We can can re-enable those tests once the fix lands in TS-ESLint.
Pin the Codegen benchmark fixture to an immutable `benchmark-files` commit. This makes benchmark inputs reproducible and prevents `main` updates from silently changing results.
A little breakdown of what was going on:
- the failing test case was testing that raw transfer vs non raw transfer was the same. This was parsing checker.ts (a 3.2mb file)
- this test was running concurrently (via vitest's describe.concurrently) with a memory exhaustion test that was running parse with raw transfer 10,000x concurrently
- as a result of these concurrent tests, it resulted in a large number of JS objects being created (and hence needing to be GC-ed)
- since CI is not deterministic, it caused a flakiness roughly 2% of the time
- this can clearly be seen from the attached flamegraph - the readFile call (reading checker.ts from the disk) was sometimes taking up to 3 seconds to read the file from the disk
This seems to be fixed by no longer using describe.concurrent, as it reduces the amount of concurrent work and reduces the chance of the timeout.
<img width="4096" height="1724" alt="image" src="https://github.com/user-attachments/assets/0fd5bc59-8d28-4453-adda-bf33834aafa8" />
## Summary
Follow-up to #22590. That PR replaced the synthetic `cal.com.tsx` fixture — a concatenation of Next.js pages that declares top-level names 70–113× and emits thousands of redeclaration diagnostics, inflating semantic `sys_allocs` — with excalidraw's real, single-file `App.tsx`. But it only touched `TestFiles::complicated()` (used by `track_memory_allocations`).
This applies the same swap to every remaining benchmark that still used `cal.com.tsx`, and unifies **all** excalidraw `App.tsx` references on one pin (`f6d85bc8`):
| File | Change |
| --- | --- |
| `tasks/common/src/test_file.rs` | `minimal()` (feeds every `tasks/benchmark` criterion bench): `cal.com.tsx` → `App.tsx@f6d85bc8`. `formatter()` re-pinned `@v0.18.0` → `@f6d85bc8` |
| `napi/parser/bench.bench.js` | parser benchmark fixture → `App.tsx@f6d85bc8` |
| `napi/parser/test/parse-raw.test.ts` | bench-fixture list + size comment → `App.tsx@f6d85bc8` |
| `tasks/benchmark/benches/semantic.rs` | rewrote the now-false "`cal.com.tsx` has many errors" comment |
| `crates/oxc_ast/src/serialize/mod.rs` | removed the stale `cal.com.tsx` row from the capacity-ratio doc table |
## Why a single pin
`get_source_text` caches downloads by the last URL segment only (`App.tsx`). With two different pins live, a single `cargo bench` run would download both into the same `target/App.tsx` and silently bench whichever landed first. Re-pinning `formatter()` to `f6d85bc8` removes that collision — and the pre-existing latent one between `cargo bench` (`@v0.18.0`) and `cargo allocs` (`@f6d85bc8`).
## Notes
No committed snapshots change: these fixtures feed only timing benches and a `programRaw.toEqual(programStandard)` self-consistency test; the snapshotted `complicated()` / `minifier()` sets are untouched.
---
This PR was prepared with AI assistance; I reviewed, tested, and verified the changes.
Implement the beginnings of support for tokens in `napi/parser`.
This PR only adds support for tokens via raw transfer, and behind an undocumented `experimentalTokens` option.
Add tests checking that tokens received on JS side via raw transfer match snapshots for all Test262, AcornJSX, and TypeScript test cases. They do!
The tests are the main purpose of this PR, making sure it works before we switch over to tokens via raw transfer in Oxlint.
We can add full tokens support to `napi/parser` (including via JSON transfer) later on.
`napi/parser` tests utilize hidden `experimentalRawTransfer` and `experimentalParent` options. Patch the types of `parse` and `parseSync` in the tests so using these options is not a type error. This replaces a bunch of `// @ts-ignore` and `// @ts-expect-error` comments.
Use `const` where variables are not reassigned instead of `let`. Preparation for enabling ESLint's `prefer-const` rule (running as an Oxlint JS plugin) in our linting setup.
The raw transfer test on `antd.js` fixture is timing out sometimes on CI. We already disabled the "range & parent" test on this fixture. Disable the other test on it too. But only skip these tests in CI.
(alternative to #14438)
The raw transfer range + parent test on `antd.js` fixture is still hitting 5 second timeout on CI. For some reason the `timeout` option isn't increasing the timeout. Just skip this test.
Add an option to `oxc-parser` to add `parent` to all AST nodes. This option is only available with raw transfer (not really feasible with JSON transfer, as JSON doesn't support circular references).
This is really for the purposes of JS plugins in Oxlint, so have not included the new deserializers which add `parent` in the `oxc-parser` NPM package, to avoid bloating download size. But it's ideal home is in `napi/parser` so can add it to the conformance tests.
`parent` is correct for all nodes in all Test262 and TypeScript test cases.
#13465 aligned the version of `antd.js` used in different Rust tasks to v4.16.1. But it missed a couple of places in `napi/parser` tests/benchmarks. Align these to same version too.
Misalignment was problematic as the tasks using these fixtures all download the file to same location (`target` dir) and don't re-download if the file is already present. This lead to unpredictable results depending on which task you ran first, and so which version was downloaded.
Now that `apps/oxlint` and `napi/parser` are both ESM packages (#13723 and #14042), we can use `.js` file extensions for all ESM files, rather than the mix of `.js` and `.mjs` file we had previously.
It's less confusing having to remember what's `.js` and what's `.mjs`, and avoids ugly workarounds like #14038 (`.d.mts`??).
the latest version of `oxc-parser` (on npm) does not have resolvable
types because the entry is `.mjs` and the types are `.d.ts` (these need
to match - `.d.mts`). you can see this in the failing CI for [this
renovate
PR](https://github.com/nuxt/nuxt/actions/runs/17940186619/job/51014702361?pr=33290)
but I'm happy to provide a more minimal reproduction if it helps
I also spotted that `napi/parser/test/visit.test.ts` was importing
`visitorKeys` which was not exported from the corresponding declaration,
so took the liberty to add that export as well.
let me know if this needs any changes 🙏
I hope it helps
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: overlookmotel <theoverlookmotel@gmail.com>
Partial fix for #13851.
Raw transfer `preserveParens` option remove `TSParenthesizedType`s from AST, same as #13855 removed `ParenthesizedExpression`s.
This is not used in `oxc-parser`, but the raw transfer `preserveParens` option is required in `oxlint` JS plugins, to align with TS-ESLint, which has no `TSParenthesizedType`s in its AST.
Pure refactor. Root of `napi/parser` was getting really crowded with tons of files, and I'm about to add a load more for the ESTree walker.
Move JS source files into a `src-js` directory, like in `apps/oxlint`, so that the root of the package is just config files and scripts.
Partial fix for #13851.
Add a `preserveParens` argument to raw transfer `deserialize` functions. This is not useful in `oxc-parser`, but these generated files will be duplicated in `oxlint`, where this option is required to remove `ParenthesizedExpression`s from AST, to align with ESLint and TS-ESLint.
Same as #13184.
#13173 enabled Oxlint for all the repo, but we currently skip `napi` directory.
Fix all errors that Oxlint flags in `napi/parser`.
We currently can't turn on Oxlint for the directory, because `oxlint-disable` comments don't work for type-aware rules, but we should be able to once that's fixed.
In tests for `napi/parser`, don't assume `target` dir exists. It generally should, because you'll have built the `.node` binary. But it `cargo clean` will remove it, and it can also not exist on CI if running build and test in separate jobs.
Fix#10519.
Run raw transfer tests across multiple threads. Vitest can run multiple test *files* in parallel on separate threads, but it doesn't have ability to run test *cases* in the same file in parallel.
So create a worker pool and send details of each test case to the workers, which then run them in parallel.
On my machine, this cuts down the time to run `napi/parser` tests by a factor of 3 - from 19s to 6s.
Also: Fix `parseAsync` tests: `Promise`s weren't being `await`-ed, and cut down number of iterations on `parseAsync` test from 100,000 to 10,000 so it completes in a reasonable amount of time.
First iteration of lazy deserialization. It works (does not throw an error on any of Test262 or TS test cases), but has some major shortcomings at present:
* Does not match ESTree - shape of AST is more like our Rust types.
* Some missing fields.
* `Vec`s are not lazily deserialized.
* No caching of nodes, so you get a different object each time you get a node e.g. `assert(binaryExpression.left == binaryExpression.left)` fails!
Note: This is a lazy deserializer, which is first step towards a lazy visitor. But there's no visitor implemented yet.
I will build on this starting point in future PRs.
Add `codeframe` property to errors returned by `parseSync` / `parseAsync` when using raw transfer. This matches the shape of error objects without raw transfer.
There were some test cases we needed to skip because they'd cause parser to panic. That bug is now fixed (#11473), so no need to skip these tests any more.
Add NAPI parser tests to check that TS-ESTree AST via raw transfer matches snapshots.
Previous PRs have fixed all remaining issues, so all the tests pass!
Previously, only `parseSync` supported raw transfer. Extend support to `parseAsync` too.
This requires:
1. A queue to limit the number of files being parsed simultaneously, to avoid the possibility of memory exhaustion.
2. A more complicated mechanism for recycling buffers, since now multiple buffers can be in use at the same time.
Related to #10978. Add `phase` property to `ImportExpression` in ESTree AST.
This property is *not* added in TS-ESTree, because TS-ESLint supports `import.defer(...)` and `import.source(...)`, but does not align with ESTree - it outputs a `CallExpression` with a `MetaProperty` as its `callee` instead.
We don't have the necessary information in our AST to match TS-ESLint, because we don't have the spans of `import`, `defer`, or `import.defer`, to use for the `MetaProperty`, or the `IdentifierName`s it contains.
Presumably TS-ESLint will change their AST shape to conform to ESTree spec once these reach stage 4, so I don't think we should expend and changing our AST to align with them in the meantime. TypeScript doesn't even have test cases for these (we'd know if they did, as we'd be failing them) - so I doubt these are popular constructs in TS.
`acorn-test262` submodule is bumped to include https://github.com/oxc-project/acorn-test262/pull/31.
Fixes#10978. Add `phase` property to `ImportDeclaration` in ESTree and TS-ESTree ASTs.
`ImportExpression` is not covered by this PR. That's more tricky as TS-ESLint already supports `import.defer(...)` and `import.source(...)`, but does not align with ESTree - it outputs a `MetaProperty` instead.
`acorn-test262` submodule is bumped to include https://github.com/oxc-project/acorn-test262/pull/30.
In ESTree (plain ESTree for JS i.e. Acorn), a hashbang is recorded as a line comment. Follow that behavior in our ESTree AST.
In TS-ESTree, hashbangs are ignored. That's not how it looks in AST explorer, because there's some post-processing going on, but `@typescript-eslint/parser` itself ignores hashbangs. So only add the comment in ESTree AST (not TS-ESTree AST).
More info on this rather confusing subtlety: https://github.com/typescript-eslint/typescript-eslint/issues/6500
The tests for ensuring field order is the same between JSON-based transfer and raw transfer were not working correctly. Fix them. Thankfully they all still pass!
Encode lone surrogates in `cooked` property of `TemplateElementValue` using same encoding scheme as for `StringLiteral`s.
In fact, they were already being encoded like this after #10041, but add a `lone_surrogates` flag to `TemplateLiteral` to decode them correctly in ESTree AST.
`oxc_codegen` ignores `cooked` and just prints `raw`, so needs no alteration.
Fix 2 edge cases when parsing lone surrogates in `StringLiteral`s:
* Lone surrogate followed by `\u{...}` escape e.g. `"\uD800\u{41}"`.
* Escaped lossy replacement character after lone surrogate e.g. `"\uD800 \u{FFFD}"`.
overlookmotel