Use `int32` (`Int32Array`) instead of `uint32`(`Uint32Array`) for getting data from buffer in raw transfer deserializer. The buffer is 2 GiB in size, so all offsets within the buffer can be represented as a `u31` which can be stored in an `i32` without any loss, and with no risk of being interpreted as negative numbers.
Same as in #21129, the advantage of fetching offsets from an `Int32Array` is that V8 statically knows the value can be stored in an SMI, its native integer type, without any range checks.
`>>` is cheaper than `>>>` because `>>` produces a 32-bit _signed_ integer which is V8's native number type (SMI). `>>>` produces a 32-bit _unsigned_ integer, which needs to be boxed and stored on the heap.
Source text in raw transfer is limited to 1 GiB, and therefore source offsets, number of lines, number of tokens, and number of comments after all less than `1 << 30`. Therefore even the sum of 2 of them cannot reach `1 << 31` (the maximum positive integer which can be stored as a positive signed 31-bit int. Therefore it's safe to use `>>` in these binary loops.
Refacfor. In a strange pattern, we were injecting `getNodeLoc` into `deserialize.js` module on every call to `deserializeProgramOnly`. It's a static function, so just import it.
Optimize raw transfer string decoding.
Replace `TextDecoder("utf8")` with `Buffer.prototype.utf8Slice` to decode Rust UTF-8 strings to JS strings.
Benchmarks show this is an average 5% speed-up in `deserializeStr`, though the reason why is unclear. It skips various checks, and avoids creating a temporary `Uint8Array` (which is required with `TextDecoder.decode`) which trims 60 bytes of the size of the assembly created for `deserializeStr`. But that's not enough to explain the speed-up 🤷. Anyway, take the win.
Add an object pool for `Location` objects, and recycle them across files.
Usually, only a minority of AST nodes, tokens, and comments will have their `loc` property accessed, so the pool is not pre-populated - `Location` objects are only created on demand.
Like the other object pools, once warmed up over the first batch of files, the fast path for "use an existing object from the pool" will be consistently taken and no further objects will be allocated. All `Location` objects will soon graduate to "old space" where they don't impede or trigger minor GC.
Performance improvement to tokens and comments APIs.
## The problem
Previously, all tokens and comments methods would deserialize *all* tokens/comments into an array of `Token` / `Comment` / `Token | Comment` objects, and then binary search through those arrays to find the token(s) / comment(s) they're looking for.
This has 2 major disadvantages:
1. Files typically contain *a lot* of tokens (even more than the number of AST nodes). Deserializing them all is very costly (up to 30% of total Oxlint runtime when run with only a JS rule which just calls a tokens-related method).
2. The binary searches these methods do are quite expensive. Even in TurboFan-optimized code, accessing `token.start` involves getting pointer to the `Token` object from the `tokens` array, an "is this object a `Token`?" safety check, then reading the `start` field from the `Token` - all just to access a single `u32`, and that happens over and over.
## This PR's solution
Solve both these problems by making tokens and comments methods read `start` / `end` offsets directly from the buffers which contain the tokens/comments data.
This data is tightly packed in memory, and strongly typed (read from `Uint32Array`s), so getting `start` / `end` of a token requires no indirection and no type checks.
More importantly, it removes the need to deserialize all tokens / comments upfront. The desired token(s) are located, touching only the buffer, and then *only* the ones which need to be returned to rule code are deserialized into JS objects.
If a rule accesses `ast.tokens`, `ast.comments`, or `sourceCode.tokensAndComments` then all tokens / comments need to be deserialized, as they're all returned to the rule as an array - but that's unavoidable. This PR doesn't make that any cheaper, but it doesn't make it measurably more costly either.
But where no rule requires the full array of tokens / comments, and they only use token/comment search methods (e.g. `getFirstToken`, `getCommentsBefore`), a great deal of work will be saved. This covers the vast majority of rules.
## Implementation details
The main complication is the `includeComments` option to tokens methods. When `true`, search needs to be over a combined set of both tokens and comments.
When `includeComments: true` option is passed to a tokens method, a buffer is created containing data about all tokens and comments, interleaved in source code order. This buffer can then be used for binary search in tokens methods.
Whether each token / comment has been deserialized already or not is tracked by a "deserialized" flag in the tokens/comments buffers. Each token / comment in the buffer is 16 bytes. This flag lives in byte 15. For tokens, this byte is always already 0 in the buffer when it arrives from Rust side. For comments, we manually set `comment.content = CommentContent::None;` for every comment on Rust side. `comment.content` is positioned at byte 15 in the `Comment` struct, and `CommentContent::None` is stored as 0.
## Possible future improvements
### SoA storage
Binary search operates only on `start` field of tokens / comments, which are 16 bytes apart in the buffer. It would be more efficient if tokens were stored in struct-of-arrays (SoA) style so all `start` values were tightly packed together. This would reduce CPU cache misses in the hot loops of binary searches.
### Pre-compute tokens-and-comments buffer on Rust side
The buffer containing tokens and comments, required to support `includeComments: true`, is currently generated on JS side (but lazily). We could move that to Rust side, which would be faster. However, it might be redundant work in many cases because the buffer is only required if a rule uses `includeComments: true`.
We could alternatively keep the laziness optimization, by calling back into Rust to build the buffer on demand - but JS-Rust calls have a cost too. Maybe communicating via `Atomics` would be faster than an actual function call?
If we had a way to share buffers with WASM, optimal solution might be to generate the buffer lazily (as now) but in WASM, which would be faster for this kind of pure number-crunching, but without the overhead of calling into Rust.
Pure refactor. Move `sourceCode` methods related to comments into a separate file `comments_methods.ts`. Does not alter any of the code, just moves it.
Apply the same optimization as #19978 to comments - hold a pool of `Comment` objects, and re-use those objects rather than creating new objects each time.
Same as with `Token`s, `loc` property is a getter which calculates `loc` lazily, and caches it in a private property.
Pure refactor. `tokens.ts` was getting too large. Split it into 2 files:
* `tokens.ts`: Tokens lifecycle methods - initialization, deserialization, and reset.
* `tokens_methods.ts`: Public methods on `SourceCode` e.g. `sourceCode.getFirstToken`.
Transfer tokens via raw transfer in Oxlint JS plugins.
Deserializer is written by hand. `Token` is not a normal struct, so deserializer cannot be generated.
Refactor. Logic related to tokens was spread across 2 files. Move it all into `tokens.ts`.
Also, fix one small bug where there'd be an error when linting an empty file.
Builds on #19497. Use the tokens generated by Oxc's parser in linter plugins, instead of running TS parser to tokenize source.
This is a sizeable perf gain, and also allows us to remove TS parser from `oxlint`'s bundle (#19531).
Additionally, our parser is more accurate than TypeScript - we handle HTML comments and space before slash in closing JSX elements (`< /div>`) - so this fixes a few conformance tests.
Pure refactor. No point passing an object around through functions, when `parserServices` is always an empty object, as Oxlint provides no parser services. Just define it as an empty object inline.
In ES3, directives didn't exist yet, and therefore `"use strict"` is treated as a standard string literal, rather than a directive.
In conformance tests, when test case specifies ES3, walk the AST and set `directive` property of all `ExpressionStatement`s to `null`. This transforms any directives in AST into plain string literals.
Combined with #18632 which fixed `"use strict"` processing in TS-ESLint, this means there are 8 ESLint test cases we no longer have to skip. More importantly, we're less likely to run into erroneous test failures when testing other plugins, which are laborious to diagnose.
Previous raw transfer required that the source text start exactly at the start of the buffer. In linter, relax this restriction and handle when source text is stored elsewhere in the buffer.
This is necessary for stripping BOM from source (#18376), as then the source text used on JS side starts 3 bytes after the start of the buffer.
When parsing tokens for JS plugins, the TypeScript parser was always using `ScriptKind.TSX`, regardless of file extension. This caused TypeScript to incorrectly parse generic arrow functions like `<T>() => {}` in `.ts` files, creating bogus `JsxText` tokens that overlap with comments.
This PR adds an `is_jsx` flag to `RawTransferMetadata`. Rust side sets the flag depending on source type, and JS side uses that information to pass the correct `ScriptKind` for the file to TypeScript parser.
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: overlookmotel <theoverlookmotel@gmail.com>
ESLint has undocumented `getRange` and `getLoc` methods on `SourceCode`. Add them.
These are the last 2 missing methods on `SourceCode` (according to Claude).
Also add more tests for correct functioning of `range` and `loc` properties.
Previously we had a pattern of destructuring properties of global objects into top-level vars. e.g.:
```js
const { keys: ObjectKeys, hasOwn } = Object,
{ isArray } = Array;
```
Doing this is more performant, but it makes the code harder to read.
Instead, write code normally, and use a TSDown plugin to convert:
```js
const keys = Object.keys(obj);
```
to:
```js
import { ObjectKeys } from "../utils/globals.ts";
const keys = ObjectKeys(obj);
```
Defining all the destructured globals in a single file (`utils/globals.ts`) is advantageous for bundle size. If the TSDown plugin created top level `const ObjectKeys = Object.keys;` statements in each file, then you can end up with this in bundle as minifier doesn't de-duplicate:
```js
const ObjectKeys = Object.keys;
const ObjectKeys$1 = Object.keys;
const ObjectKeys$2 = Object.keys;
```
The globals we use have to be manually added to `utils/globals.ts`, but TSDown plugin warns when this needs to be done, and doesn't generate a broken build in the meantime.
This plugin is run in debug builds too, so that the tests will catch any bugs.
Previously we used `@typescript-eslint/typescript-estree` parser to tokenize a source text. `@typescript-eslint/typescript-estree` uses TypeScript parser internally, and then builds the tokens array by traversing TypeScript's AST.
Instead, use TypeScript parser directly, and copy the code from TS-ESLint for converting tokens from TypeScript's format to TS-ESTree format.
### Why?
This has a few advantages:
1. Remove a dependency.
2. TS-ESLint's tokens did not have `start` or `end` properties, which we want for consistency with our AST. By reimplementing ourselves, we can add those properties to tokens.
3. TS-ESLint does a bunch of other work besides compiling the tokens array - it converts the AST too. We discard the AST that TS-ESLint produces, so this work is pointless. By cutting out TS-ESLint, we also cut out that unnecessary work, and do only what we need to - tokenizing.
4. TS-ESLint would throw an error if TypeScript produces any errors (even non-fatal ones), and there is no option to tell TS-ESLint not to do that. This caused a bunch of conformance tests to fail, because ESLint's tests include invalid code.
5. Can add workarounds for TypeScript bugs - e.g. not being able to parse JSX closing fragments containing line breaks (`<>hello<\n/>`).
### Implementation detail
We no longer need to get comments from TypeScript. We use the comments produced by Oxc parser instead.
### `CommentToken` type
A side effect is that there's no longer any need for separate `Comment` and `CommentToken` types - as we use our own comments now. So this PR removes `CommentToken` type.
Have marked this as a breaking change due to removal of this type.
Only alters comments.
* Make sure all `// @ts-expect-error` comments include "TODO" where it's an issue we need to fix.
* Format all the "TODO" comments consistently.
Add a getter `lineStartIndices` to `SourceCode`. ESLint has this property, but it's undocumented. Possible that some ESLint plugins rely on it (if it's there, people will use it!).
Add a getter `tokensAndComments` to `SourceCode`. ESLint has this property, but it's undocumented, but many of ESLint's built-in rules rely on it. Probably other ESLint plugins rely on it too.
Visitor keys object used by `oxlint` and `napi/parser` is frozen, but the individual arrays of keys were not. Deep freeze the whole structure to prevent user modifying it.
Implement `SourceCode#getNodeByRangeIndex` method.
Behaves the same as ESLint's version, but this implementation should be more performant as it's much simpler, avoiding ESLint's `Traverser` abstraction, and exiting loops earlier.
Rename assertion functions to clarify that they're not "real" assertions which we can rely on for validating input, and are only for compile-time testing or debug assertions.
`typeAssertIs` is not a great name, but I couldn't come up with anything better. `debugAssertIs` would be not be accurate, as it'd imply that in debug builds in performs an actual assertion, which it doesn't.
In any case, we should replace `typeAssertIs` with specialized assertion functions which do actually perform runtime assertions in debug builds (e.g. `typeAssertIs<Fix>()` -> `debugAssertIsFix()`).
- "Slow but working" implementation for
https://github.com/oxc-project/oxc/issues/14829#issuecomment-3512679496
- Uses the parser from `@typescript-eslint/typescript-estree` for
tokens. The source code is fully parsed and everything but the tokens
and comments is discarded.
- Tests are directly adapted from `eslint`.
- Direct commits from maintainers welcome.
### Tasks
- [x] `getTokens`
- [x] Move token types to `src-js/plugins/tokens.ts`
- [x] Bundle typescript, handling `_filename` usage in ES modules.
- [x] ~[Prevent class name mangling by
`tsdown`](https://github.com/oxc-project/oxc/pull/15861#discussion_r2542816947).
Could be deferred.~
https://github.com/oxc-project/oxc/pull/15906#pullrequestreview-3487271352
- [x] [Test regex-divide ambiguous
syntax](https://github.com/oxc-project/oxc/pull/15861#discussion_r2542919409).
- [x] Test for conformance directly against `eslint` tests.
- [ ] [Parse as JSX only if source text is
JSX](https://github.com/oxc-project/oxc/pull/15861#discussion_r2542850640).
Could be deferred.
- [ ] Lazy load the parser on demand. Could be deferred.
### Future work
- [ ] `getFirstToken`
- [ ] `getFirstTokens`
- [ ] `getLastToken`
- [ ] `getLastTokens`
- [ ] `getTokenBefore`
- [ ] `getTokenOrCommentBefore`
- [ ] `getTokensBefore`
- [ ] `getTokenAfter`
- [ ] `getTokenOrCommentAfter`
- [ ] `getTokensAfter`
- [ ] `getTokensBetween`
- [ ] `getFirstTokenBetween`
- [ ] `getFirstTokensBetween`
- [ ] `getLastTokenBetween`
- [ ] `getLastTokensBetween`
- [ ] `getTokenByRangeStart`
- [ ] `isSpaceBetween`
- [ ] `isSpaceBetweenTokens`
### Decisions
- [x] `@typescript-eslint/typescript-estree` peer-depends on
`typescript`. How should we package it? Currently, `typescript` is being
made an optional dependency.
- **overlookmotel (on discord): ideally bundled, direct runtime
dependency otherwise.**
- [x] Deprecated methods are being removed altogether in ESLint 10:
`getTokenOrCommentBefore`, `getTokenOrCommentAfter`, and
`isSpaceBetweenTokens`. These are surface level deprecations: the
functionality was merged with other methods (the `includeComments: true`
option) and plugins can migrate with a one line change. I'm guessing we
are targeting fairly modern, actively developed projects. Should we
expose them?
- **[overlookmotel and
camc314](https://github.com/oxc-project/oxc/pull/15861#issuecomment-3552651295):
yes.**
- [x] `Program` range is different between ESLint and TS-ESLint. Which
one should we follow?
-
**[overlookmotel](https://github.com/oxc-project/oxc/pull/15861#discussion_r2547915407):
they are going to both change and align soon, we should update our range
calculation to match when they do.**
---------
Co-authored-by: overlookmotel <theoverlookmotel@gmail.com>
Pure refactor. Split up `plugins/utils.ts` into 3 files in `utils` directory.
Most importantly, assertion functions are by themselves in an `asserts.ts` file. This paves the way for the next PR (#15946).
Adapted from #15868. Enable `strictNullChecks` in TSConfig for `apps/oxlint`.
Rather than using `!` all over the place, I've added a function `assertIsNonNull`, which performs the same function. Idea is that we can later on produce a debug build where `assertIsNonNull` does an actual runtime assertion (rather than just a TS `asserts`) that the value is not `null` or `undefined`. We can use the debug build in tests, and catch any logic errors.
Unfortunately, TypeScript doesn't seem able to track side effects, so there seems to be no way to tell it "after calling `initAst()`, `ast` is always non-null". Don't want to switch to a having a `getAst` function which returns a non-null value, as that'd introduce unnecessary temp vars, and function calls, on hot paths. So this seems to be the best we can do without incurring runtime cost.
Bundle the deserializer. Originally it wasn't bundled at my request, so that source maps wouldn't be required for debugging. But it's turning out that we have no problems in the deserializer - our problems are elsewhere!
Add `SourceCode#getJSDocComment` method. It currently throws an "unimplemented" error.
This concludes the `Context` API additions. Every method, getter, and property listed in ESLint's [custom rules docs](https://eslint.org/docs/latest/extend/custom-rules) are now present in our API too.
Note: `Plugin` and `Rule` types may still be missing properties.
Implement the following deprecated `SourceCode` methods, related to tokens:
* `getTokenOrCommentBefore`
* `getTokenOrCommentAfter`
* `isSpaceBetweenTokens`
The 1st 2 are currently unimplemented, but can be implemented when we add tokens APIs.
`isSpaceBetweenTokens` implementation is flawed at present, in the same way as `isSpaceBetween` is, due to lack of tokens. But probably close enough to be useful in many cases. Added tests which demonstrate the flaw.
Add `Context#markVariableAsUsed` method. It's unimplemented at present, so just throws an error saying "unimplemented". But that's better for user than "TypeError: context.markVariableAsUsed is not a function", and having the stub in codebase reminds us we need to implement it.
Implement `sourceCode.isSpaceBetween`. Implementation is not quite correct, because we don't have tokens yet. But it's better than no implementation at all. We can correct it once we have tokens.
The reason it's not correct is explained in comment on `isSpaceBetween` implementation, and the incorrect behavior is demonstrated in the test fixture.
Follow-on after #15364. Re-use a single object for `parserServices`, instead of creating a new object each time. This is better for perf, and also matches ESLint where each access to `sourceCode.parserServices` in a `create` invocation results in the same object.
## Summary
- Changed `sourceCode.parserServices` from throwing an error to
returning an empty object `{}`
- Aligns with ESLint specification for parsers that don't implement
parser services
- Prevents crashes in ESLint plugins such as
`eslint-plugin-better-tailwindcss`.
## Motivation
According to the [ESLint custom parser
documentation](https://eslint.org/docs/latest/extend/custom-parsers#parseforeslint-return-object):
> services can contain any parser-dependent services (such as type
checkers for nodes). The value of the services property is available to
rules as context.sourceCode.parserServices. **Default is an empty
object.**
The current implementation throws an error when `parserServices` is
accessed, which breaks plugins that use defensive checks like:
```typescript
if(typeof ctx.sourceCode.parserServices?.defineTemplateBodyVisitor === "function"){
// ...
}
```
Plugin authors defensively handle:
- ✅ `parserServices` being undefined
- ✅ `defineTemplateBodyVisitor` not existing
- ❌ `parserServices` getter throwing an error (unexpected)
## Solution
Return an empty object `{}` for unimplemented parser services, which:
1. Matches ESLint's documented default behavior
2. Allows plugins' defensive checks to work correctly
3. Provides correct semantics for "not implemented"
## Test Plan
- Existing tests should continue to pass
- Plugins using defensive checks like the example above will no longer
crash
## Related
Fixes the compatibility issue with plugins like
[eslint-plugin-better-tailwindcss](https://github.com/schoero/eslint-plugin-better-tailwindcss/blob/a90cc3f547098a4293d7b0b7998bf6774f717cd2/src/utils/rule.ts#L140)
---------
Signed-off-by: magic-akari <akari.ccino@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: overlookmotel <theoverlookmotel@gmail.com>
Follow-on after #14890.
That PR implemented `ScopeManager` class. Instead of a class, which an instance of is created for each file, use a singleton object, with the underlying TS-ESLint scope manager stored in a top-level var. Methods/getters of `SCOPE_MANAGER` lazily initialize the TS-ESLint scope manager as required, and cache it.
This pattern is the same as we use for `SourceCode`, enabled by the fact that only 1 file is linted at a time. This has several advantages:
1. Reduce object creation - less GC pressure.
2. Property accesses don't need to go up prototype chain, as they would for instances of a class.
3. No need for private properties, which are somewhat expensive to access.
Nicolas Le Cam