Closes out remaining spec compliance gaps in the lexer, Regex vs
Division cases, `>>` cases, + many more.
Fully passes on Test262, All Typescript tests, Oxc tests, SWC tests,
VSCode (codebase), And 1_000 machine hours of differential fuzzing;
Confidence is high that it's either 'correct' or very close.
This PR specifically needs AI Disclosure in both finding and fixing the
cases now because the bitwise logic is complex; Something that I'll
start addressing once 'correctness' is locked in (and performance).
---------
Co-authored-by: overlookmotel <theoverlookmotel@gmail.com>
#26263 added the ability to run tests and conformance for both versions of `oxc_lexer` (SIMD and scalar fallback) on any platform.
Run tests and conformance on CI (both SIMD and scalar) on any PR which touches `oxc_lexer` crate or other files which affect conformance runner.
Additionally, run Clippy on the SIMD version (main Lint CI task doesn't lint it, due to the `target_feature = "avx2"` gate on the code).
All of this sets up `oxc_lexer` crate for further development, catching any regressions in CI.
`@__KEY__` and `#__KEY__` are currently treated as normal comments. Codegen can drop them before a later property-mangling pass. The property may then be renamed while the annotated string keeps its old name.
This classifies the marker as a property-key annotation. Codegen keeps it next to string and no-substitution template literals when annotation comments are enabled, and the property mangler reads that classification instead of scanning source text. Final output still follows the normal comment options, so whitespace-minified or annotation-disabled output removes the marker.
## Example
Input:
```js
const object = { _field: 1 };
const key = /* #__KEY__ */ "_field";
object[key];
```
Before, an intermediate codegen pass could remove the marker:
```js
const object = { _field: 1 };
const key = "_field";
object[key];
```
After, the marker survives for the later property-mangling pass.
Verified with the affected crate suites, `just minsize`, workspace lint, docs, AST generation, and independent review. `just ready` is only blocked locally by existing Node-version and terminal-color snapshot differences.
Implemented with AI assistance (OpenAI Codex). I reviewed and understand the changes and remain responsible for this contribution under the repository AI usage policy.
`prev_is_regex` walks back one token to decide whether `/` opens a
regex. When a newline forces ASI but the previous token still looks
expression-final, it answered division. The JSX `<` decision goes
through the same function, so TSX was wrong in the same places.
```ts
let x: T
/re/.exec(s) // was lexed as division
let x
/re/.test(s)
import y from 'm'
/re/.exec(s)
break outer
/re/.test(x)
let x: 1
/re/.exec(s)
let x!: T
/re/.exec(s)
```
Also `type A = T`, `declare function f(): T`, `function f(): void {}`,
template literal types, and `continue label`.
Free on the hot path: ASI needs a newline before the `/`, so the
backward walks sit behind an `lt_in_range` gate that straight-line `a /
b` never passes. No measurable change over the fixture corpus.
18 tests added.
Review feedback was that the token kinds should be a real enum rather than a module of u8 constants. The values were listed twice, once as the constants and again in the token_kind_name match, so they now come from one declaration, and tok_kinds and the new Lexer::kinds hand out &[TokenKind] instead of raw bytes.
The discriminants do not move. The pipeline computes kinds arithmetically and blends them in SIMD registers, so it keeps working on u8. Only the aliases in pipeline and the compile-time tables in opmap are re-expressed in terms of the enum, and those fold away at compile time. Reading the bytes back as TokenKind assumes every kind written is a declared discriminant, which a debug assertion checks on every accessor. Output is identical across both corpora and an interleaved A/B against the branch below lands inside the noise floor.
Two shaves on the comment path in carve. push_comment_record decided preceded_by_newline by walking back over the whole run of whitespace before the comment and then re-scanning that same run forward looking for a line terminator, so it touched those bytes twice. It now walks back once and stops at the first newline, which is the answer it was looking for anyway. Separately, scan_block_comment recomputed the newline mask and the at-sign mask on every 32-byte block of the comment body even after saw_nl was already true and the license marker had already been found, so both are now skipped once they can no longer change the result, which on a long JSDoc block is nearly the whole body.
Comment records are field-exact across both fixture corpora and the token stream and every value lane are unchanged. This is worth about 8% of the per-comment cost and 12 to 15% on an all-comment file. Corpus-weighted it comes out neutral, since comments are a small share of the work almost everywhere, so it is aimed at comment-heavy real source rather than at the average.
A template body's line terminator sequences are normalized in both the TRV and the TV, so a raw CRLF or a lone CR in the source becomes a single LF in the cooked value. push_template was cooking with normalization switched off, on the reasoning that a consumer could redo it from the raw text, so on a CRLF checkout every multi-line template cooked to bytes the grammar says it should not have. oxc_parser does the normalization itself in template_literal_carriage_return, so the two disagreed.
Redoing it afterwards is not possible, which is the part worth spelling out. Once escapes are decoded a CR in the cooked bytes is ambiguous: it may have come from a raw CRLF, which has to collapse, or from a \r escape, which has to survive, and a template of \r\n must cook to CR LF while any blanket pass over the cooked bytes would flatten it to LF. The decision has to be made on the raw text before escapes are decoded, which is inside cook, so this flips the existing const-generic on rather than adding machinery. Only push_template changes, since identifier atoms cannot contain a raw CR and a raw CR in a string literal is already an error. The new tests go through push_template rather than cook_decode, because the decoder was always correct and already had a passing test while the wiring was what was wrong. No measurable cost, and across both corpora exactly one file's output changes, nanoplotx_components.tsx, by one byte, the single CRLF it has inside a template collapsing to LF.
A HexEscapeSequence denotes a code point, so \x80 through \xff are U+0080 through U+00FF, and the cooked lane, which consumers read back as a &str, has to carry them as two UTF-8 bytes. The b'x' arm of cook_decode wrote the raw byte instead, so a lone continuation byte landed in the lane and "\xa9" disagreed with both "\251" and the literal character, which already went through the two-byte paths in emit_oct and the code-point emitter. That makes it an unsound &str rather than merely a wrong string, and it surfaces as a confusing panic somewhere downstream instead of a clean mismatch.
The fix is the same branch those two paths already have. Escapes below 0x80 are unchanged, and the reservation still covers it either way since the shortest escape of this kind is four raw bytes producing two cooked ones. There are tests for the byte-exact output and for all three escape forms agreeing across the whole 0x00 to 0xff range, which is the invariant that was broken, and the \u path was already correct and is now covered by the same test. No measurable cost, since none of the fixtures contain a hex escape at or above 0x80 and an interleaved A/B sits inside the run-to-run noise. This one is independent of the perf work underneath it and only sits on that branch to keep the stack linear, so it can be retargeted at main if that is easier to land.
The token tape carried a u32 start per entry and kept whitespace and comments on it as ordinary tokens, so anything reconstructing a token's end had to reach for the next entry and every consumer had to step over trivia. Because whitespace was itself a token, the end of each significant token was already in the array as the start of the following entry, just stored under a different name and then discarded. This publishes one oxc_span::Span per significant token instead, start in the low half and end in the high half, bit-identical to the AST's own Span, with trivia gone from the tape entirely. A span turns out to be a misaligned eight-byte window onto positions the pipeline already had, so build_spans widens and compacts them with a qword permute and a byte shuffle rather than recomputing anything, and compress now fills a fixed staging buffer a chunk of blocks at a time instead of a full-length array, carrying one entry across chunks so the last token of a chunk can still see its end. Eight EOF sentinels spanning (n, n) follow the last token so a consumer can peek a short way past the end without a bounds check, and the hashbang stays on the tape, since its kind sits inside the trivia range and the drop mask excludes it explicitly.
Value lanes, cooked bytes, comment records and diagnostics are byte-identical across both fixture corpora, and the span stream was checked against a scalar reference on every fixture. It costs about 0.22 cyc/byte, roughly 1.0 to 1.5 cycles per significant token, because the tape writes nine bytes per token where it used to write five per entry across about a third more entries than there are real tokens. Worth saying plainly that this one costs throughput rather than winning it. It is a contract change that moves work out of the consumer, and it only pays off once the consumer stops reconstructing ends, skipping trivia and scanning for token boundaries.
compress compacts the st/kind bitmaps into the dense starts/kinds streams. It emitted each 8-bit chunk as an interleaved pair of stores, one into starts and one into kinds. On Zen 3 a store that partially overlaps the previous one serializes to commit whenever another stream's store sits between them, so interleaving the two streams paid that penalty on every block.
This composes two 8-bit chunks into a single 16-lane shuffle and groups the stores so the overlapping starts writes land back to back, dropping the serialized seams from 16 to 4 per block. The control comes from a new pair-composed lookup table replacing the old per-chunk one, and compress stays inline(never) so the store order survives LLVM's aliasing analysis. Output is identical and a new exhaustive test checks it against a scalar reference over every lookup-pair combination. This is the larger of the two wins, moving the big files by roughly 0.25 to 0.3 cyc/byte (checker.js and all the minified bundles), since it is per-block work rather than a once-per-file cost.
The AVX2 classify stage finished each file with a byte-at-a-time scalar
loop over the last n%64 bytes. On small inputs that loop is most of the
cost, up to around 1150 cycles when the leftover sits close to a full
64-byte block.
It now runs whole blocks only. The final block reads a little past the
end into the zero padding the caller already guarantees, and the
out-of-range lanes are masked off afterwards, so the token output is
bit-for-bit identical. Small files are what improve: jsx_test and
tsx_test come down about 0.6 cyc/byte and expressions.js about 0.3,
while larger files do not move, since the tail was only ever paid once
per file.
File-level V8 and Istanbul coverage directives attach to the first AST statement. When TypeScript import removal or another upstream pass deletes that statement, codegen never visits the comment attachment position and silently drops the directive. Coverage tooling then instruments a file that was explicitly excluded.
The lexer now classifies `v8 ignore file` and `istanbul ignore file` separately, including line-comment forms and explanatory suffixes. Codegen lets those directives share the existing legal-comment orphan flush, preserving source order without scanning source text during printing. Position-sensitive directives such as `ignore next`, `start`, and `stop` remain unchanged.
The recovery path is named and documented around orphaned attachments. Hashbang ordering remains intact, and disabling annotation comments still suppresses these directives. The transformer regression stays in the Rust integration harness because transform conformance deliberately clears comments before codegen.
## Example
Input:
```ts
/* v8 ignore file */
import type { Foo } from "./types";
export default { typed: "stuff" } satisfies Foo;
```
Before:
```js
export default { typed: "stuff" };
```
After:
```js
/* v8 ignore file */
export default { typed: "stuff" };
```
Covered by direct codegen regressions for removed anchors, hashbang ordering, disabled annotation comments, and position-sensitive negative cases, plus a TypeScript transformer integration test. Verified with `env -u NO_COLOR just ready`, including all-feature tests, linting, documentation, generated-code checks, formatting, and a clean-tree check.
closes#23667
This PR was assisted by OpenAI Codex.
- Adds scalar fallback to oxc_lexer.
- Closes the Regex vs Division mislex gaps (without parser feedback).
- Adds a swathe of aggressive regex vs division test cases.
- Adds `replay.rs` which kicks in for cases that will not appear in real
source but closes out the remaining spec compliance gaps for `await` and
`yield` and regex.
- All regex vs division gaps filled now falls into `zero cost` or `pay
when occurs` permafrost cold path.
---------
Co-authored-by: overlookmotel <theoverlookmotel@gmail.com>
Adds the x86_64 version of my Research `SIMD For Context Sensitive
Languages`.
(CPB = Cycles per byte)
- Adds new create crates/oxc_lexer; Which eventually lead into an
unfused parser design inside oxc_parser
- Fully test262 compliant JS standalone Lexer; SIMD used has
counterparts on every platform.
- Performs at 2.5CPB on checker.js on my ZEN 3 CPU; or around 1.5-1.75
gbs (3.8GHz).
- Across aarch64, + other platforms performance will hold which I've
seen in my POCs but those targets will come later.
- Supports JSX/TS options, Which don't have much impact on overall perf
results.
------
The expected results for seeing this work through [including on parser
side on oxc_parser]
1. P95 improvements of 80-100% over current `oxc_parser`; 12-18% of this
will come from the unfused contract where JS grammar specifically loves
being able to see 'ahead' into `kinds` in various places. The rest comes
from the perf improvements to the lexer itself and some wins that will
come inside the parser from work now done in the lexer.
Shanon Jackson