Oxlint's JS-side code is processed by a TSDown plugin that transforms global property accesses to `const`s imported from `globals.ts`. e.g.:
```js
function foo(obj) {
return Object.keys(obj);
}
```
is transformed to:
```js
import { ObjectKeys } from "./utils/globals.ts";
function foo(obj) {
return ObjectKeys(obj);
}
```
The purpose of this transform is:
1\. Better perf - avoid a property lookup/check on each call.
2\. More robust in the face of user code (plugins) altering globals e.g. `Object.keys = function myWeirdFunction() {}`.
While delving into the assembly that V8 produces while reviewing #22238, I discovered that in fact the transform was having a _negative_ effect on perf, because TSDown puts the global `const`s in a separate chunk `global.js`. Because the consts are in a different module, this actually makes them _more_ costly to access, rather than less.
In fact, it turns out that TurboFan doesn't insert a check at all in the original code - it instead has machinery to trigger de-opt if the global is reassigned. So the original perf rationale for this transform was completely misguided - it doesn't in fact produce a perf gain at all! We still want to keep it for the robustness advantage, but we don't want it to be perf _loss_, as it was before this PR.
This PR fixes the perf by instead defining consts inline in each file. The example above is now transformed to:
```js
const ObjectKeys = Object.keys;
function foo(obj) {
return ObjectKeys(obj);
}
```
This also removes the need to manually maintain the `globals.ts` file. It's now automatic.
Additionally, improve the transform in a few minor ways:
1. Support deeper nested global props e.g. `Object.prototype.toString`.
2. Skip converting e.g. `Array.prototype.slice.call(arguments)` (because `.call` requires `this`, so `ArrayPrototypeSliceCall(arguments)` is incorrect).
3. Do replace methods which take `this` (e.g. `Promise.resolve`) when they're not used in a method call.
Note: The robustness advantage of this transform may appear to be marginal, and not worth the bother of having this transform at all. Personally, I think it's worthwhile. Oxlint JS-side code has access to a slice of Rust's memory as a `Uint8Array`. Writing to that buffer could cause UB, reading/writing out of bounds, or other serious problems on Rust side, so IMO it's critical to protect against that in every way we can.
Performance improvement to tokens and comments APIs.
## The problem
Previously, all tokens and comments methods would deserialize *all* tokens/comments into an array of `Token` / `Comment` / `Token | Comment` objects, and then binary search through those arrays to find the token(s) / comment(s) they're looking for.
This has 2 major disadvantages:
1. Files typically contain *a lot* of tokens (even more than the number of AST nodes). Deserializing them all is very costly (up to 30% of total Oxlint runtime when run with only a JS rule which just calls a tokens-related method).
2. The binary searches these methods do are quite expensive. Even in TurboFan-optimized code, accessing `token.start` involves getting pointer to the `Token` object from the `tokens` array, an "is this object a `Token`?" safety check, then reading the `start` field from the `Token` - all just to access a single `u32`, and that happens over and over.
## This PR's solution
Solve both these problems by making tokens and comments methods read `start` / `end` offsets directly from the buffers which contain the tokens/comments data.
This data is tightly packed in memory, and strongly typed (read from `Uint32Array`s), so getting `start` / `end` of a token requires no indirection and no type checks.
More importantly, it removes the need to deserialize all tokens / comments upfront. The desired token(s) are located, touching only the buffer, and then *only* the ones which need to be returned to rule code are deserialized into JS objects.
If a rule accesses `ast.tokens`, `ast.comments`, or `sourceCode.tokensAndComments` then all tokens / comments need to be deserialized, as they're all returned to the rule as an array - but that's unavoidable. This PR doesn't make that any cheaper, but it doesn't make it measurably more costly either.
But where no rule requires the full array of tokens / comments, and they only use token/comment search methods (e.g. `getFirstToken`, `getCommentsBefore`), a great deal of work will be saved. This covers the vast majority of rules.
## Implementation details
The main complication is the `includeComments` option to tokens methods. When `true`, search needs to be over a combined set of both tokens and comments.
When `includeComments: true` option is passed to a tokens method, a buffer is created containing data about all tokens and comments, interleaved in source code order. This buffer can then be used for binary search in tokens methods.
Whether each token / comment has been deserialized already or not is tracked by a "deserialized" flag in the tokens/comments buffers. Each token / comment in the buffer is 16 bytes. This flag lives in byte 15. For tokens, this byte is always already 0 in the buffer when it arrives from Rust side. For comments, we manually set `comment.content = CommentContent::None;` for every comment on Rust side. `comment.content` is positioned at byte 15 in the `Comment` struct, and `CommentContent::None` is stored as 0.
## Possible future improvements
### SoA storage
Binary search operates only on `start` field of tokens / comments, which are 16 bytes apart in the buffer. It would be more efficient if tokens were stored in struct-of-arrays (SoA) style so all `start` values were tightly packed together. This would reduce CPU cache misses in the hot loops of binary searches.
### Pre-compute tokens-and-comments buffer on Rust side
The buffer containing tokens and comments, required to support `includeComments: true`, is currently generated on JS side (but lazily). We could move that to Rust side, which would be faster. However, it might be redundant work in many cases because the buffer is only required if a rule uses `includeComments: true`.
We could alternatively keep the laziness optimization, by calling back into Rust to build the buffer on demand - but JS-Rust calls have a cost too. Maybe communicating via `Atomics` would be faster than an actual function call?
If we had a way to share buffers with WASM, optimal solution might be to generate the buffer lazily (as now) but in WASM, which would be faster for this kind of pure number-crunching, but without the overhead of calling into Rust.
Pure refactor. Move `sourceCode` methods related to comments into a separate file `comments_methods.ts`. Does not alter any of the code, just moves it.
#20310 and #20311 reduce code size and complexity by moving binary search routine which is used in all tokens and comments methods into a shared function `firstTokenAtOrAfter`.
However, as noted in #20310, the downside of that change is a perf hit due to the cost of function calls.
This PR fixes that regression by adding a TSDown plugin to the build which inlines the body of `firstTokenAtOrAfter` into all the call sites in tokens and comments methods. The net result is that the final code in bundled output is almost identical to before #20310, but the source code is much shorter and easier to understand.
This unlocks a large optimization which will follow in a future PR (see #20310 for more details).
TSDown build uses a custom plugin to replace global property lookups e.g. `Object.assign` with top level vars e.g. `const ObjectAssign = Object.assign;`, and then replace the usages with those top-level vars.
The plugin was only applied to `.ts` files. Broaden the filter to include `.js` files too, so it applies to the large generated files `deserialize.js` and `walk.js`.
Pure refactor. TSDown config had got very large with the custom plugins to replace `debugAssert` calls, and hoist globals vars to top level. A PR to follow will make the situation worse, as it adds another plugin.
Move the plugins into separate files, to bring the TSDown config back to a manageable size.
This PR does not alter the plugin code, just moves it.
Boshen