Part of an effort to move non symbol/scope checks from semantic ->
parser.
> I intend to only keep checks associated with symbols and scopes in the
checker, and move everything back to the parser.
> We need to stop the formatter from formatting code when a syntax error
occurs. The formatter don't run semantic.
Add `value` field to `BigIntLiteral`, same as the other literal types.
Once upon a time, we stored the value as a `BigInt` which normalized the representation (removing `_`s, and converting e.g. `0xFF` to `255`). But we had to remove it because `BigInt` owns a heap allocation, and this caused a memory leak.
https://github.com/oxc-project/oxc/commit/50fe0fef73027f10740e68a65a78216933b5d8a0
This PR brings back the `value` field, but avoids memory leak by storing the normalized value as a string (`Atom`).
This has effects in various places in the codebase:
* Brings our JS-side AST into line with ESTree (fixing the test cases broken by #11563).
* Fixes edge cases in a couple of lint rules.
* Codegen now prints bigints in normalized form e.g. `0xFFn` is printed as `255n`.
* Changes behavior of 1 function in `oxc_ecmascript`.
I've made comments below on the parts I'm not so sure about. Review would be very welcome!
Bump `acorn-test262` submodule to include https://github.com/oxc-project/acorn-test262/pull/47, which bumps Acorn dependency.
Acorn v8.15.0 fixes the formatting of the `bigint` field of `Literal`s to align with ESTree spec. Currently we are failing those test cases as our behavior is incorrect. Will fix it in following PR.
ESTree conformance tests check field order is correct in TS-ESTree ASTs.
This revealed some mismatches, but they were fixed by #11449.
Need to skip 1 test which is only failing due to rounding errors in floating point numbers in the `serde_json` round-trip conversion JSON -> value -> JSON.
- towards https://github.com/oxc-project/oxc/issues/11334
Rewrites the parsing for import declarations to not require any lookahead. I've diverged from the TS reference implementation here because it uses a fair amount of `tryParse` and lookahead. The main trick used here is simply parsing as we go and then doing "lookback" to see if the identifier we just parsed was actually a keyword like `defer` or `type`. This allows us to not need to use rewind at all.
Also improves diagnostics slightly by emitting a new diagnostic message for `import ... =`. In the end, this didn't improve conformance at all, but it's a perf win of about 1% on some files.
Re-order fields in ESTree and TS-ESTree ASTs to be in visitation order (which is the order in which they're defined in Rust types).
This does not align with either Acorn or TS-ESLint, but for the reasons discussed in https://github.com/oxc-project/oxc/issues/9705#issuecomment-2859250834, I feel this isn't a concern. The field order doesn't matter much per se, only that it's consistent, and this change has other benefits - it will make it simpler to build a visitor which visits fields in the correct order, and allows adding other parsers to `acorn-test262` in order to produce test cases for decorators.
Visitation order aligns with TS-ESLint except for the following types:
* `ExportSpecifier`
* `TSImportType`
* `TSIndexedAccessType`
* `TSMethodSignature`
* `TSPropertySignature`
* `TSTypePredicate`
In these 6 cases, TS-ESLint's `visitorKeys` has the wrong visitation order, because it does not visit fields in source order. I will open an issue and hopefully they can fix.
This PR also bumps `acorn-test262` to include https://github.com/oxc-project/acorn-test262/pull/37 which re-orders the fields in fixture snapshots to match the new field order here.
- part of https://github.com/oxc-project/oxc/issues/11334
Rewrites the import/export specifier parsing to be much closer to the TS compiler reference version. The diagnostics should be nearly the same as before, but with a slight improvement for a few cases where we had "unexpected token" before and now we know it should be "identified expected." In theory this might restore the performance somewhat since we don't do any lookahead in this version. However, the initial version of this has some extra allocations there weren't there previously I think, so it may not be much faster.
Out of curiosity, I've just deleted the peeking in `parse_delimited_list`.
Then, coverage errors show me there are only a few places missing error report for invalid trailing comma after rest element, etc.
So I added those checks manually.
In TSC, these checks seem to be done at checker, not parser. But we need this in parser for JS.
---
(Sorry, I have PASSED the test, but I am not very confident that this change is intrinsically sound.)
My original plan was to rework the JSX parser from the entry point, as I had found in #11232.
However, when I read through all the TSC JSX parser code, I found that what we are doing is not much different.
---
As for perf perspective, the OXC currently requires rewind in 3 places for JSX:
- 1: `parse_jsx_expression()` - `lookahead`
- To determine if it is a `JSXElement`(`<C`) OR `JSXFragment`(`<>`) at the entry point
- 2: `parse_jsx_child()` - `checkpoint+rewind`
- The same above OR breaking list(`</`)
- 3: `parse_jsx_child()` - `checkpoint+rewind`
- To determine if it is `JSXChild::Spread`(`{...Expression}`) OR `JSXChild::ExpressionContainer`(`{JSXExpression}`)
These don't exist in TSC, they just use union type or loose ASTs.
But we have different and explicit ASTs in Rust, it seems they can't be easily eliminated.
And unfortunately, I tried, but there was no remarkable improvement.
- For 1, 3045952596c7201348b26ef347bdd99e110384e4 => Minifier +3%, but no effect for parser
- For 2, I wish we had `Kind::LessThanSlash`...
- For 3, a12ff218e6d93be84a90ba1f8bcff25181b4b384 => No effect for parser
As a result, I think it would be better to keep the current clean and explicit code.
Just for aligning with `TypeScript`, the object key should be converted to a static property key when possible
```ts
const ObjectKeys = {
a: 0,
["b"]: 1,
[`c`]: 2,
[3]: 3,
[-3]: 4,
[4n]: 5,
};
```
```ts
declare const ObjectKeys: {
a: number
b: number
c: number
3: number
[-3]: number
};
```
`BigInt` is invalid as a property key in TypeScript, so `4n` doesn't emit.
- towards https://github.com/oxc-project/oxc/issues/11194
Replaces all usage of the lexer lookahead functions like `nth_kind`, so that we only use checkpoints and rewinding, which is aligned with the TypeScript compiler implementation. This also will allow us to remove the lookahead field from the lexer in the future.
Don Isaac