ShanonJackson 7d24ddb927 refactor(lexer): replace token kind constants with a TokenKind enum (#24931) ...

Review feedback was that the token kinds should be a real enum rather than a module of u8 constants. The values were listed twice, once as the constants and again in the token_kind_name match, so they now come from one declaration, and tok_kinds and the new Lexer::kinds hand out &[TokenKind] instead of raw bytes.

The discriminants do not move. The pipeline computes kinds arithmetically and blends them in SIMD registers, so it keeps working on u8. Only the aliases in pipeline and the compile-time tables in opmap are re-expressed in terms of the enum, and those fold away at compile time. Reading the bytes back as TokenKind assumes every kind written is a declared discriminant, which a debug assertion checks on every accessor. Output is identical across both corpora and an interleaved A/B against the branch below lands inside the noise floor.

2026-07-29 00:15:12 +00:00

..

diagnostics.rs

feat(lexer): add scalar lexer and close the remaining pathological regex vs division cases. (#24376)

2026-07-12 00:08:57 +01:00

module_goal.rs

refactor(lexer): replace token kind constants with a TokenKind enum (#24931)

2026-07-29 00:15:12 +00:00

regex_div.rs

refactor(lexer): replace token kind constants with a TokenKind enum (#24931)

2026-07-29 00:15:12 +00:00

ts_keywords.rs

refactor(lexer): replace token kind constants with a TokenKind enum (#24931)

2026-07-29 00:15:12 +00:00