Add more debug assertions to debug build-only code which checks the validity of comments and tokens.
Make sure all comments and tokens have spans which are wihin bounds of source text. In particular, this will surface any problems where the source text contains a BOM, which causes us to adjust spans on Rust side when trimming off the BOM.
Tiny perf optimization to linter plugins.
`if (x === false)` is longer than `if (!x)`, but is more performant. The 1st compiles down to a simple equality check between `<value x>` and `kFalse` constant, whereas the 2nd is more like `coerceToBoolean(<value x>) === kFalse` - a more expensive operation.
This follows our convention in all JS plugins code of always using explicit `=== false` / `=== true` comparisons.
Pure refactor. Use debug asserts instead of type assertions in code for deserializing comments and tokns. This gives better error messages if we break the invariants in tests. Makes no difference in release build.
## Summary: Fixing unicorn/expiring-todo-comments via jsPlugin
## Problem
The `unicorn/expiring-todo-comments` rule failed when run through
oxlint's jsPlugin system with error:
```TypeError: Either `node` or `loc` is required```
The rule spreads Comment instances (`{...comment}`) to process each line of multi-line comments, but lost the `loc` property because:
- Comment's `loc` was a prototype getter (not an own enumerable property)
- Spread only copies own enumerable properties
## Solution
Define `loc` as an own accessor property on every `Comment` instance using
`__defineGetter__`. Own properties are visible to spread, `JSON.stringify,` and
`for…in` iteration, so all three work without any extra workarounds.
The getter function is defined in the class static block and captured in a `const`
(`getCommentLoc`). Using a `const` rather than a `let` lets V8 skip reassignment checks
at the `defineGetter(this, "loc", getCommentLoc)` call site — a pattern already
established in the codebase for `resetCommentLoc`.
As a result:
- `toJSON()` is removed — `JSON.stringify` now works via the own property directly.
- `Object.defineProperty(Comment.prototype, "loc", { enumerable: true })` is removed —
no longer needed.
## Test Results
- ✅ unicorn/expiring-todo-comments rule works (detects expired TODOs)
- ✅ updated comments fixture pass
- ✅ All untouched existing tests pass
---------
Co-authored-by: overlookmotel <theoverlookmotel@gmail.com>
Micro-optimization.`Token` and `Comment` classes create functions defined in static blocks to reset the value of `#loc` private property on class instances. Copy these functions into `const` vars, to avoid checks at these functions' call sites. Previously they were stored in `let` bindings, which incurs a check at each call site (because the value might have been re-assigned). The difference in Turbofan-optimized code is just an 8-byte load and a comparison, but why keep that when we can lose it?
Discovered the perf difference while digging into #22238.
`loc` property of `Token`s and `Comment`s is a getter on these classes' prototypes. Add a `toJSON` method to both classes so `JSON.stringify` includes `loc` in the generated JSON.
Similar to #20479. `tokensWithLoc` and `commentsWithLoc` contain tokens/comments on which `loc` property has been accessed.
Previously these arrays were grown with `.push(token)`, and shrunk at end of linting file with `tokensWithLoc.length = 0;`. Problem with that is that setting length to 0 frees the array's backing allocation, which means it has to reallocate again on next file when a token's `loc` is accessed.
Instead, never shrink these arrays, and track the active length in separate variables. After warm-up over first batch of files, these arrays will graduate to "old space" and just sit there without further allocations.
Unlike #20479, we don't use `Uint32Array`s as `loc` is calculated lazily, and `Token`'s / `Comment`s don't know what their index is in the `cachedTokens` / `cachedComments` arrays. Adding an `#index` field to `Token` and `Comment` would bloat every instance of these classes by 8 bytes.
Fix a JS-side memory leak from tokens and comments.
## The problem
V8 uses "SlicedString" optimization to avoid copying string data. `sourceText.slice(10, 20)` does not copy bytes 10..20 to a new string, but references the original string data of `sourceText` (much like `&str` in Rust).
#19978 and #20362 introduced caching and recycling of `Token` and `Comment` objects. This was a large performance gain.
However, it contained a flaw. The problem is that `Token` objects now live forever, and they contain little slices of the source text in their `value` property. This means they prevent the source text which they reference from being garbage-collected, and therefore it lives forever too (or at least until another file with an equal or greater number of tokens overwrites the `value` property of all the `Token` objects).
## Solution in this PR
Fix this by setting the `value` property of all `Token`s and `Comment`s to `null` after linting a file in `resetTokens` and `resetComments`.
## Implementation details
This requires some additional book-keeping to track which tokens have been deserialized and which haven't, so that `resetTokens` / `resetComments` do the minimum amount of work necessary - only resetting tokens / comments which actually had their `value` property set in the first place.
The indexes of tokens are stored in 2 x `Uint32Array`s `deserializedTokenIndexes` and `deserializedCommentIndexes`. A simpler solution would have been to store the `Token` / `Comment` objects themselves in a plain `Array`, but `Uint32Array` has 3 advantages:
* 4 bytes to store index (`u32`) per token/comment, instead of 8 bytes to store a pointer to the `Token` / `Comment` itself.
* Faster to write to `Uint32Array` - no write barrier checks.
* On GC runs, a `Uint32Array` is known not to contain references to any other heap objects, so GC tracing doesn't need to walk it, which it would with an `Array`.
`Uint32Array` does have one disadvantage over `Array`:
* 1 level of indirection in `resetTokens` / `resetComments`. You have to read the index, then index in to `cachedTokens` array to get the `Token` object, in order to reset it.
The latter will only have a significant effect when there's a large number of tokens/comments to reset. We avoid this in the worst case where *all* tokens have been deserialized (e.g. by accessing `ast.tokens`). When `allTokensDeserialized === true`, `resetTokens` just loops through `cachedTokens` itself, and ignores the `Uint32Array` entirely.
## Future potential optimizations
It's probably not necessary to reset all tokens quite so aggressively when all tokens have been deserialized. If a rule has done that on 1 file, chances are it'll do it again on the next file, at which point most `Token`s get their `value` property overwritten with slices of *that* source text anyway.
It's acceptable for source texts to hang around in memory for a bit, just not forever. We could choose to do token resetting only e.g. once every 10 files, to reduce the cost.
Possibly clean-up operations could also run in next tick, so they happen in parallel while Rust side is prepping the next file to send over to JS, rather than blocking.
Performance improvement to tokens and comments APIs.
## The problem
Previously, all tokens and comments methods would deserialize *all* tokens/comments into an array of `Token` / `Comment` / `Token | Comment` objects, and then binary search through those arrays to find the token(s) / comment(s) they're looking for.
This has 2 major disadvantages:
1. Files typically contain *a lot* of tokens (even more than the number of AST nodes). Deserializing them all is very costly (up to 30% of total Oxlint runtime when run with only a JS rule which just calls a tokens-related method).
2. The binary searches these methods do are quite expensive. Even in TurboFan-optimized code, accessing `token.start` involves getting pointer to the `Token` object from the `tokens` array, an "is this object a `Token`?" safety check, then reading the `start` field from the `Token` - all just to access a single `u32`, and that happens over and over.
## This PR's solution
Solve both these problems by making tokens and comments methods read `start` / `end` offsets directly from the buffers which contain the tokens/comments data.
This data is tightly packed in memory, and strongly typed (read from `Uint32Array`s), so getting `start` / `end` of a token requires no indirection and no type checks.
More importantly, it removes the need to deserialize all tokens / comments upfront. The desired token(s) are located, touching only the buffer, and then *only* the ones which need to be returned to rule code are deserialized into JS objects.
If a rule accesses `ast.tokens`, `ast.comments`, or `sourceCode.tokensAndComments` then all tokens / comments need to be deserialized, as they're all returned to the rule as an array - but that's unavoidable. This PR doesn't make that any cheaper, but it doesn't make it measurably more costly either.
But where no rule requires the full array of tokens / comments, and they only use token/comment search methods (e.g. `getFirstToken`, `getCommentsBefore`), a great deal of work will be saved. This covers the vast majority of rules.
## Implementation details
The main complication is the `includeComments` option to tokens methods. When `true`, search needs to be over a combined set of both tokens and comments.
When `includeComments: true` option is passed to a tokens method, a buffer is created containing data about all tokens and comments, interleaved in source code order. This buffer can then be used for binary search in tokens methods.
Whether each token / comment has been deserialized already or not is tracked by a "deserialized" flag in the tokens/comments buffers. Each token / comment in the buffer is 16 bytes. This flag lives in byte 15. For tokens, this byte is always already 0 in the buffer when it arrives from Rust side. For comments, we manually set `comment.content = CommentContent::None;` for every comment on Rust side. `comment.content` is positioned at byte 15 in the `Comment` struct, and `CommentContent::None` is stored as 0.
## Possible future improvements
### SoA storage
Binary search operates only on `start` field of tokens / comments, which are 16 bytes apart in the buffer. It would be more efficient if tokens were stored in struct-of-arrays (SoA) style so all `start` values were tightly packed together. This would reduce CPU cache misses in the hot loops of binary searches.
### Pre-compute tokens-and-comments buffer on Rust side
The buffer containing tokens and comments, required to support `includeComments: true`, is currently generated on JS side (but lazily). We could move that to Rust side, which would be faster. However, it might be redundant work in many cases because the buffer is only required if a rule uses `includeComments: true`.
We could alternatively keep the laziness optimization, by calling back into Rust to build the buffer on demand - but JS-Rust calls have a cost too. Maybe communicating via `Atomics` would be faster than an actual function call?
If we had a way to share buffers with WASM, optimal solution might be to generate the buffer lazily (as now) but in WASM, which would be faster for this kind of pure number-crunching, but without the overhead of calling into Rust.
Optimization to comments deserialization. Files which have no comments are probably fairly common. Add a fast path for this case. We have to branch on `length === 0` later on anyway, so might as well branch earlier and exit.
Previously the entire AST had to be deserialized when calling any comments method. This was purely to get the `hashbang` property of `Program`, which has to be added to start of comments array as a `Shebang` comment.
Usually the AST has to be deserialized anyway in order to visit it, but some rules might early exit and return an empty visitor after checking if a certain comment is present. In those cases, it was extremely wasteful to deserialize the whole AST just to check for `hashbang`.
Instead, add a `Line` comment to start of `Vec<Comment>` on Rust side if a hashbang is present. If there are a lot of comments, this involves shifting them all up which could be expensive, but files with hashbangs are rare, and this change enables other optimizations coming in future PRs that rely on all comments being present in the buffer.
Pure refactor. Move `sourceCode` methods related to comments into a separate file `comments_methods.ts`. Does not alter any of the code, just moves it.
Apply the same optimization as #19978 to comments - hold a pool of `Comment` objects, and re-use those objects rather than creating new objects each time.
Same as with `Token`s, `loc` property is a getter which calculates `loc` lazily, and caches it in a private property.
Rename assertion functions to clarify that they're not "real" assertions which we can rely on for validating input, and are only for compile-time testing or debug assertions.
`typeAssertIs` is not a great name, but I couldn't come up with anything better. `debugAssertIs` would be not be accurate, as it'd imply that in debug builds in performs an actual assertion, which it doesn't.
In any case, we should replace `typeAssertIs` with specialized assertion functions which do actually perform runtime assertions in debug builds (e.g. `typeAssertIs<Fix>()` -> `debugAssertIsFix()`).
Pure refactor. Split up `plugins/utils.ts` into 3 files in `utils` directory.
Most importantly, assertion functions are by themselves in an `asserts.ts` file. This paves the way for the next PR (#15946).
Adapted from #15868. Enable `strictNullChecks` in TSConfig for `apps/oxlint`.
Rather than using `!` all over the place, I've added a function `assertIsNonNull`, which performs the same function. Idea is that we can later on produce a debug build where `assertIsNonNull` does an actual runtime assertion (rather than just a TS `asserts`) that the value is not `null` or `undefined`. We can use the debug build in tests, and catch any logic errors.
Unfortunately, TypeScript doesn't seem able to track side effects, so there seems to be no way to tell it "after calling `initAst()`, `ast` is always non-null". Don't want to switch to a having a `getAst` function which returns a non-null value, as that'd introduce unnecessary temp vars, and function calls, on hot paths. So this seems to be the best we can do without incurring runtime cost.
Add `SourceCode#getJSDocComment` method. It currently throws an "unimplemented" error.
This concludes the `Context` API additions. Every method, getter, and property listed in ESLint's [custom rules docs](https://eslint.org/docs/latest/extend/custom-rules) are now present in our API too.
Note: `Plugin` and `Rule` types may still be missing properties.
Part of #14564. Binary search is used to traverse the comments array
wherever possible.
- [x] `getCommentsBefore`
- [x] `getCommentsAfter`
- [x] `getCommentsInside`
- [x] `commentsExistBetween`
Pure refactor. Import `SourceCode` methods related to comments with a namespace import, continuing the pattern used for token and scope-related methods.
Follow-on after #14715. I for one had completely misunderstood what the comments APIs are meant to do. Expand the docs for these methods to clarify their behavior.
Follow-on after #14715. Pure refactor. Move code for `SourceCode`'s comment-related methods into a separate file. Code is completely unchanged - it's just moved.
overlookmotel