mirror of
https://github.com/oxc-project/oxc.git
synced 2026-09-14 19:36:11 +08:00
ShanonJackson
256369a8da
The token tape carried a u32 start per entry and kept whitespace and comments on it as ordinary tokens, so anything reconstructing a token's end had to reach for the next entry and every consumer had to step over trivia. Because whitespace was itself a token, the end of each significant token was already in the array as the start of the following entry, just stored under a different name and then discarded. This publishes one oxc_span::Span per significant token instead, start in the low half and end in the high half, bit-identical to the AST's own Span, with trivia gone from the tape entirely. A span turns out to be a misaligned eight-byte window onto positions the pipeline already had, so build_spans widens and compacts them with a qword permute and a byte shuffle rather than recomputing anything, and compress now fills a fixed staging buffer a chunk of blocks at a time instead of a full-length array, carrying one entry across chunks so the last token of a chunk can still see its end. Eight EOF sentinels spanning (n, n) follow the last token so a consumer can peek a short way past the end without a bounds check, and the hashbang stays on the tape, since its kind sits inside the trivia range and the drop mask excludes it explicitly. Value lanes, cooked bytes, comment records and diagnostics are byte-identical across both fixture corpora, and the span stream was checked against a scalar reference on every fixture. It costs about 0.22 cyc/byte, roughly 1.0 to 1.5 cycles per significant token, because the tape writes nine bytes per token where it used to write five per entry across about a third more entries than there are real tokens. Worth saying plainly that this one costs throughput rather than winning it. It is a contract change that moves work out of the consumer, and it only pays off once the consumer stops reconstructing ends, skipping trivia and scanning for token boundaries.
32 lines
629 B
TOML
32 lines
629 B
TOML
[package]
|
|
name = "oxc_lexer"
|
|
version = "0.139.0"
|
|
authors.workspace = true
|
|
categories.workspace = true
|
|
edition.workspace = true
|
|
homepage.workspace = true
|
|
include = ["/src"]
|
|
keywords.workspace = true
|
|
license.workspace = true
|
|
publish = false
|
|
repository.workspace = true
|
|
rust-version.workspace = true
|
|
description.workspace = true
|
|
|
|
[lints]
|
|
workspace = true
|
|
|
|
[lib]
|
|
doctest = false
|
|
|
|
[dependencies]
|
|
oxc_ast = { workspace = true }
|
|
oxc_span = { workspace = true }
|
|
oxc_syntax = { workspace = true }
|
|
memchr = { workspace = true }
|
|
|
|
oxc_diagnostics = { workspace = true, optional = true }
|
|
|
|
[features]
|
|
oxc_diagnostics = ["dep:oxc_diagnostics"]
|