- Console health badge: reverse-video grade chip, eighth-block gauges per
dimension, one row per sub-score with what it was measured from.
- markdown/html reporters for --dashboard and --health, alongside the
existing console/json/badge; the health SVG badge moves into badge.rs.
- --dashboard's Duplication section lists a breakdown by format instead of
by file, sourced from the same jscpd statistics as everywhere else.
- Duplication no longer counts markup, stylesheet and template clones
(HTML, CSS, Handlebars, ...): a repeating style rule is not the
maintenance problem repeating programming logic is. The exclusion
follows the clone's own format, not the file it lives in, so a .svelte
or .vue file's style/template block is left out even though the
component itself is full weight.
- The duplication line and format-breakdown table now name what they
actually measured and, only for categories the project has files in,
what was left out (e.g. "5.4% in typescript (no text)") instead of a
fixed disclaimer.
One 0-100 score with an A-E grade, shown on top of --dashboard and alone
with --health.
- Dimensions are shares of the code lines, so size cancels out:
duplication (jscpd's percentage over code files), dead code, and the
share of code in complex files (complexity 50+). Prose and data files
are not counted.
- Each share is scored on a half-life curve (100 at zero, 50 at one
half-life), calibrated on 42 open-source projects so the median project
scores 75 per dimension; shares are mixed with a 2000-line prior so one
finding does not sink a small project.
- Sub-scores combine as a weighted geometric mean. A dimension that
cannot be measured is left out and named, and dead code weighs as much
as the share of the code it could read.
- --health-input FILE and the health config object add metrics from
other tools (a ready score, or a value with a half-life) and tune the
built-in half-lives and weights.
- Reporters: console badge, ai one-liner, json (jscpd-health.json,
jscpd-dashboard.json with every dashboard section) and badge
(jscpd-health-badge.svg). Exit gates apply as in the dashboard.
CpdClone::matched_lines moves to cpd-core so statistics and health count
duplicated lines the same way.
--dashboard prints one screen: project size and largest formats,
duplication with clones per kind and the most duplicated files, total and
mean complexity with the most complex files, and dead code by category
with the largest findings (JavaScript, TypeScript, Python). The dead-code
scan runs beside clone detection; --summary-top sets the rows per list.
Data formats (JSON, YAML, TOML, CSV, ...) join markdown at complexity 0,
so a lock file's version ranges no longer top the complexity ranking.
Docs for --kind, --complexity and --dashboard; fixtures/dashboard-demo,
plus --kind and --complexity sections in the type3 and summary demos.
Copilot's review of #1062 raised seven points; each checked out and each is
fixed with a test that fails without it.
- A glob keeps its pattern. `import.meta.glob('./locales/*.js')` and
import(`./pages/${name}.vue`) used to keep every file under the directory
alive; they now reach what the pattern matches, `*` staying within one
segment as it does for the bundler. On Firefly an image glob,
'../../**/*.{png,jpg}', had been keeping all of src/ alive.
- Every pattern of an `import.meta.glob([...])` array is kept, not the first.
- A declared alias is not second-guessed: when `@` points at src/ and the file
is missing there, the `@/` project-root fallback no longer finds a
different file. A catch-all `*` path claims no prefix and does not count.
- Astro client <script> blocks are read as the modules Astro bundles them
into, so what they import is used. A script with any attribute but `src` is
left to the browser, as Astro leaves it.
- import('./x') in markup is found only inside expressions, not in comments
or plain text.
- A quoted attribute value is an expression only on a directive (`:prop`,
`@click`, `#slot`, `v-*`); elsewhere only its `{...}` interpolations are,
so `class="card"` no longer credits a declaration named `card`.
- An object key in markup, `{ label: value }`, is not a read.
Re-measuring on the thirty trending projects also exposed `?raw` imports, as
in `import script from './panzoom.js?raw'`: a bundler query now is stripped
before resolving. On one snapshot of those projects the previous release
reports 1093 unused files and this branch 481, with the share some import
names falling from 83% to 62%; the CHANGELOG now quotes those numbers
instead of ones measured against a development baseline. Components used
only from .mdx content are still reported, and the docs say so.
The CX column was one path per file plus every branch keyword the token
stream showed, and on the generic tokenizer that list was partly
unreachable: it splits punctuation into single characters, so the && and ||
entries never matched outside JavaScript and ?? counted twice. Measured
against lizard on ten GitHub-trending projects in nine languages, CX ranked
files with a median Spearman correlation of 0.83 and put their size at 0.75
of lizard's.
- Adjacent punctuation is joined before matching, so short-circuit
operators count in every language.
- Per-language rules: Rust match arms count by their =>, less one per match
since N arms are N paths; Swift adds guard and Go select. Where ? also
marks an optional (TypeScript, JavaScript, Swift, Kotlin, Groovy, C#) it
counts only as a ternary, told apart from String? and x?.y by being
written in the open; ?: is Elvis only in Kotlin and Groovy.
- One path per function: by declaring keyword (def, fn, func, function, fun,
and => for arrow functions), and in C, C++, Java, Objective-C and C# by the
head(args) { shape told apart from if (...) {. Languages with neither keep
the per-file baseline.
- The body of a """ or ''' string no longer counts. The line-based tokenizer
leaves its quotes in adjacent literals, and an odd number of triple quotes
in such a run opens or closes the string, which handles the PEP 257
opener """Summary. that a pair-of-tokens check read backwards. This is
limited to languages where """ is a delimiter: where a doubled quote is an
escape (C#, VB.NET, SQL, Pascal) it swallowed a 413-branch C# file.
- A keyword-shaped word the file binds as a name - case = 3, x.case,
f(case, when) - is not a branch in that file.
On the same ten projects the median correlation rose to 0.92 and the ratio
to 1.00, and every project improved. Where CX and lizard still differ they
differ on purpose: lizard counts a match once whatever its arms, and does
not count ??.
Detection, tokenization and clone output are untouched: the fixtures corpus
reports the same 248 clones. fixtures/summary-demo has one file per language
whose complexity can be counted by hand, and every CX value matches it.
basta reported most of a real Vue, Svelte or Astro project as dead: on
thirty GitHub-trending projects in those three ecosystems it flagged 1572
unused files, and some import in the tree contradicted 78% of them. The
misses were almost never in the analysis itself; they were imports only the
project's own build knows how to resolve. This reads them the way the build
does.
- Single-file components are read whole. The script of a .vue, .svelte or
.astro file is masked in place so positions stay real, and the markup is
scanned for the names it reads: <Foo />, kebab-case spellings, {{ }}
interpolations, Svelte actions, attribute expressions, and a literal
import() written in markup.
- Aliases come from vite.config.* resolve.alias and svelte.config.* kit.alias
as well as tsconfig paths, and SvelteKit's $lib is known without the
generated .svelte-kit/tsconfig.json no repository commits. The project's
own declaration now wins over the ~/ @/ project-root fallback.
- A computed import whose static head names a directory,
import(`./pages/${name}.vue`), and import.meta.glob('./locales/*.js') are
expanded into one namespace edge per file there, the way a bundler hands
back whole module objects. Such an import no longer marks its module as
dynamic access, so the module's own findings keep their confidence.
- In a monorepo a package name resolves through that package's package.json,
exports subpath by subpath with source conditions preferred over ./dist,
anywhere in the workspace pnpm-workspace.yaml, lerna.json, turbo.json or a
workspaces field marks out.
- A path literal in a build config resolves against the project root, and
Nitro's api/, routes/ and middleware/ are file-system routes.
On the same thirty projects reported unused files fell from 1572 to 468.
fixtures/dead-code-demo gains components/, bundler/ and monorepo/. On master
the bundler demo reports five files as certainly dead, four of them wrongly;
the monorepo demo reports badge.ts, which the app imports by package name.
Both now report only what is dead. The quote-aware byte scanning the markup
and config readers share lives in one place, lang::Quotes and
lang::outside_strings, so rust/crates keeps master's 25 clones.
Adds a second question about the same tree: not "what is duplicated" but
"what does nothing reach". A new `basta` crate builds the import graph from
the project's entry points and walks it, reporting unused files, unused
exports, unused module-private declarations, unused imports and — opt-in —
unused class members, across JavaScript, TypeScript, JSX, TSX and Python.
It ships two ways: `jscpd --dead-code` inside the existing binary, and a
standalone `basta` command. Both reuse the walker, the filters, all fifteen
reporters and the `--threshold` / `--exit-code` gates, so nothing new has to
be learned to run it.
Because it is a graph traversal rather than a reference count, dead code
cascades: a helper whose only caller is itself dead is reported too. Every
finding carries a confidence score and, below 100, the reasons it might be
wrong — `eval`, an unrecognised decorator, a wildcard re-export, a name seen
only in a string, a file in the scan that did not parse. `--min-confidence`
sets the floor at 60.
Entry points come from `package.json`, `pyproject.toml`, framework and
file-name conventions, and anything a shell script, CI workflow, Makefile or
Dockerfile names by path. CommonJS is read alongside ESM. A project that
renames its own import paths is read on its own terms: `paths` and `baseUrl`
from `tsconfig.json` and `jsconfig.json` are resolved, through a relative
`extends` chain and per package in a monorepo. A file named only by a path
string — a worker spawned through `new URL`, a build entry, a vitest setup
file — counts as used.
Languages plug in through one trait, `basta::lang::Analyzer`, which owns
parsing, specifier resolution, entry-point conventions, manifests, alias
configs and path traits. Nothing outside `lang/` is language-specific;
`docs/basta-extending.md` walks through adding one.
Validated against knip, fallow, vulture, skylos, fossil-mcp and repowise on
trending repositories. On a corpus of ten of them — 11k files, 3.3M lines —
every one of basta's 1,806 findings was checked against an import graph
rebuilt independently of it: no unused-file, unused-import or unused-symbol
claim survived as wrong.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
With `--follow-symlinks` the source id was the canonical path, so a file
reached through a link showed up as an absolute path outside the scan
root, `--ignore` (matched on the walked path) and the report disagreed
about its name, and a file reachable through two paths was two sources
(a file symlink next to its target even became a clone of itself).
The walked path, anchored at the canonical scan root, is now the id:
what reports show, what `--ignore` matches, what `--absolute` prints.
The canonical path travels alongside (`DiscoveredFile::real_path`,
`PreparedSource::real_path`) for reading the file and for the
`--skip-local` / `--skip-isolated` prefix checks. When links are
followed, files whose canonical path was already seen are dropped; the
entry that is the real file wins, else the first walked path.
Docs: `--follow-symlinks` row in the option table, the v4 default in the
migration table and README. Fixture `fixtures/follow-symlinks-demo`.
Closes#1059
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#375.
A snippet containing a NUL, an ANSI escape, a form feed or any other
control byte made the xml report unparseable: XML 1.0 has no way to
represent those characters, not even inside CDATA, so xmllint stopped at
"PCDATA invalid Char value 27" and every parser refused the file. The
reporter now replaces every character outside XML's Char production
(tab, LF, CR and U+0020 and up, minus U+FFFE/U+FFFF) with U+FFFD in
snippets and in path attributes. The existing "]]>" split is kept.
Path attributes were also escaped twice: quick-xml escapes attribute
values on its own, so the reporter's escape_xml() turned "r&d" into
"r&amp;d". The manual escape is gone and the sanitized path goes in
raw.
Tests parse the generated report to the end with quick-xml and check:
control characters become U+FFFD and none survive, a "]]>" round-trips
through the split sections, a path with & < " > is escaped exactly once
and reads back unchanged, and legal text is left borrowed.
Fixtures: fixtures/xml-report-demo with a clone whose body holds real
escape bytes, a form feed and a "]]>" literal; the README shows the
xmllint and Python checks. Verified the published 5.2.0 fails xmllint
on it and the fixed binary passes.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
The one-line sparkline was too small to read a trend from. The console
block now draws an eight-row bar chart: a percentage axis on the left
spanning the series' own min and max (a series moving between 2.0% and
2.4% would be a flat line on a 0-based axis), one column per point built
from eighth-block characters so every point keeps at least a sliver, an
x-axis, and point numbers that match a new `#` column in the table.
Past 20 points the columns narrow to one character so 30 points still
fit an 80-column terminal. The ai reporter keeps the compact sparkline.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#1002 (the --history half; --ratchet is deliberately not
implemented, the report shows the headroom instead).
`jscpd src --history v5.0.0..HEAD` (or `--history-since 2026-01-01`)
lists the commits with `git log`, checks each one out into a temporary
detached worktree (the --baseline-from-ref machinery, now shared through
baseline_ref::map_scan_paths), scans it with the run's own configuration
and keeps the totals; the current run is the series' last point.
`--history-every N` keeps every Nth commit counted from the newest,
`--history-limit N` (default 30) samples a long series evenly keeping
both ends. Nothing runs unless one of the flags is given.
Output: the console reporter appends a sparkline of the duplication
percentage, a table (commit, date, files, lines, clones, dup lines,
dup%, change, subject) with the change column red when duplication rose
and green when it fell, the overall trend line, and, when --threshold is
set and the latest value is under it, the headroom with the value the
threshold could be tightened to. The json reporter adds a `history`
object; the ai reporter prints one compact line per point. Other
reporters are unchanged.
New cpd_core::history (model, sparkline, change/headroom helpers with
unit tests), cpd::history (git log parsing, sampling, worktree scans),
cpd_reporter::history_render. Config keys history / historySince /
historyEvery / historyLimit; action input `history`. Integration tests
build a four-commit repo with fixed dates and check the JSON series,
the console block, every/limit thinning and the error paths. 969
workspace tests pass, clippy clean.
Docs: History section and option rows in docs/rust.md, README bullet,
action input row, fixtures/history-demo/README.md with a script that
builds the demo repository and the expected output.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#1047.
Four situations used to write an empty report and exit 0, so a CI job
that checks only the exit code saw a green run:
- `--format` naming a format that is not supported (a warning since #964)
is now `Error: --format: 'x' is not a supported format`, exit 1. Names
introduced through --formats-exts or --formats-names stay accepted.
- A scan path that does not exist is `Error: path does not exist: <p>`,
exit 1, before anything runs.
- A reporter that fails to write its output still prints the existing
`Reporter '<name>' error: ...` line; the run now ends with
`ERROR: a reporter failed to write its output` and exit 1, after the
other reporters have run.
- A scan that analyzes no files (paths exist, but nothing matched the
--format/--ignore/--pattern filters or every file was below
--min-tokens) prints `Warning: jscpd analyzed no files` and still
exits 0, so an intentionally empty tree passes. The new
`--fail-on-empty` flag (config key `failOnEmpty`, action input
`fail-on-empty`) makes it exit 1. Reports are written before the check.
Docs: an "Exit codes" section in docs/rust.md, option rows in
docs/rust.md and docs/ci-and-hooks.md, a README feature bullet.
Fixtures: fixtures/fail-on-empty-demo with a `code/` clone pair, a
`nothing/` comment-only file, and a README quoting every command's
output and exit code. Tests: the #964 warning test becomes an error
test, plus integration tests for each new exit path and for the config
key; 531 tests pass, clippy clean.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
An open clone was enlarged whenever the next window matched any stored
occurrence, without checking that the clone's anchor still matched. When
a third file had stored a window that continues the same text, the
anchored fragment was stretched past what the anchor file contains: the
clone was dropped by flush_clone when the anchor ran out of tokens, or
reported with mismatched ends when it did not (fixtures/haxe had such a
pair, [1:1 - 62:76] against [1:1 - 62:2]).
The match now asks first whether the anchor continues at the same offset;
if it does, the clone grows regardless of what the store holds for that
hash, and if it does not, the clone is flushed and a new one starts on the
occurrence the store matched. On the fixtures corpus this recovers 14
pairs among identical files and splits the misaligned haxe pair into two
token-identical ones; every *-demo directory's output is unchanged.
Regression test with the three-file shape from #1033 (a: X; b: X' T X'';
c: X T Z) asserting a↔c has exactly |X| tokens and every reported pair
holds identical token runs. Demo: fixtures/nway-demo/lost-pair.
Closes#1033.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Copilot review: gating on a <template> block let a script-only Vue
component keep <script>/<style> wrapper tags in its html stream. A Vue
file is blocks-only whatever blocks it has, so gate on the format instead.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Sorting the html tokens (previous commit) restores source order, but the
wrapper tags of a Vue file — <template>, </template>, <script ...>,
</script>, <style ...>, </style> — are identical in every Vue file and
sit at both ends of it. In source order they stretch every template clone
to the last </style>: on fixtures/vue the html clone went from
[1:1 - 10:9] to [1:1 - 262:10] and duplicated lines from 289 to 541,
counting the script and style bodies as duplicated html.
A file with a <template> block keeps all of its markup inside that block,
so the skeleton outside the blocks is only wrapper tags and is now left
out of the html stream. The clone becomes the template body itself:
file1.vue:html [2:3 - 10:9], 75 tokens. Svelte and Astro have no template
wrapper; their top-level markup is the skeleton and is kept, so their
output is unchanged. The sort stays, for custom blocks that map to html.
Fixture: fixtures/sfc-demo/template-range with a README showing the
template-only range; docs/rust.md links it.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Public-output items from the #1020 review, done while nothing is released:
- SARIF and Code Climate rule names now match the clone vocabulary:
renamed -> jscpd/renamed-code, similar -> jscpd/similar-code
(was jscpd/similar-code for renamed and jscpd/near-miss-code for similar).
The similar-code rule text covers both the gap and the ast method.
- Duplicated-line statistics and the per-file summary count only the
matched lines of a merged clone. CpdClone gains a non-serialized
unmatched_lines field holding each fragment's gap lines, so enabling
the merge no longer moves --threshold.
- A merge whose similarity would fall below 0.5 (more unmatched than
matched tokens, e.g. one very long inserted line) is refused and the
halves stay separate; MIN_GAP_SIMILARITY documents the floor.
- Precedence is decided and documented: merged renamed halves are
reported as similar.
merge_gapped_clones is restructured around merge_into(); the running
matched-token counter was redundant with last.token_count and is gone.
Fixtures: type3-demo/long-line (refused merge) and type3-demo/renamed-halves
(precedence); README numbers re-measured for the whole directory.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
A JavaScript/TypeScript file with any parse diagnostic (a redeclaration,
a recoverable syntax error) was handed to the word-split fallback
tokenizer. Its tokens then never matched an oxc-tokenized file, so every
cross-file clone involving it vanished while its self-clones survived —
the symptom reported in #1023, where a second copy of a function in one
file made all a↔b clones disappear.
Tokens come from the lexer, not the parser: only `panicked` (or an empty
token stream) now triggers the fallback. Function extraction for
--similarity uses the partial AST on recoverable errors the same way.
Closes#1023.
`CpdClone.similarity_method` (JSON "method": "gap" | "ast") records which
near-miss mechanism produced a `similar` clone. The two scores are not on
one scale (gap = matched tokens over the merged span, ast = structural
Jaccard of whole functions), so the console prints
`Clone found (javascript, similar (gap) ~0.91)` /
`similar (ast) ~0.75`, the ai reporter `[~0.91 gap]` / `[~0.75 ast]`,
JSON carries "method" and SARIF a `similarity_method` property. Exact and
renamed clones are unchanged.
Off by default. `--similarity RATIO` (config `similarity`, Action input
`similarity`, MCP `check_duplication` argument) summarizes every function,
method and arrow function of a JavaScript/TypeScript/JSX/TSX file by the
bag of 4-grams over the pre-order sequence of its AST node types (oxc),
indexes the bags with MinHash + LSH banding, and reports pairs whose
weighted Jaccard index reaches RATIO as `similar` clones with a
`similarity` value. Names and literal values are not part of the summary:
a renamed copy scores 1.0, one inserted line ~0.9, two inserted statements
plus renames ~0.75.
Functions must clear --min-tokens/--min-lines, nested functions are never
paired with their parent, and pairs already covered by an exact, renamed
or merged clone are skipped. Values outside (0, 1] warn and disable the
option. Reporting reuses the Type-3 vocabulary (console `similar ~0.75`,
JSON kind/similarity, SARIF jscpd/near-miss-code).
New: cpd-core::similarity (FunctionSig, SimilarityIndex, bag_jaccard,
minhash), cpd-tokenizer::functions (oxc visitor extractor), MCP function
index built lazily per scan. fixtures/type3-demo/similar-functions with
README numbers, docs, unit + integration + MCP stdio tests. Also adds the
missing max_gap_lines/similarity fields to the --debug config output.
Off by default (0). When set to N, clones of the same file pair whose
fragments follow each other in both files with at most N unmatched lines
between them are merged into one clone of kind `similar`, with `tokens` =
matched tokens and `similarity` = matched tokens over the tokens of the
longer merged span. Adjacent exact windows that share a boundary token are
counted once; chains merge transitively; overlapping or reordered fragments
never merge. The pass runs after every other filter and returns its input
untouched at 0, so default output is unchanged.
Reporting: console `Clone found (fmt, similar ~0.91)`, ai `[~0.91]`, JSON
"kind"/"similarity" (rounded to three decimals), SARIF rule
jscpd/near-miss-code with a similarity property (declared only when
referenced), Code Climate check_name.
Config key maxGapLines (kebab alias), GitHub Action input max-gap-lines,
docs section, fixtures/type3-demo with a README of runnable commands, unit
tests for the merge pass and reporters, and a binary-level integration test.
Closes#998.
Three opt-in flags (config keys ignoreIdentifiers, ignoreLiterals,
ignoreAnnotations) normalize token classes before hashing so that blocks
which differ only in names, values or annotations are detected:
- identifiers hash as `$id`; keywords are kept (the JS/TS tokenizer
classifies them, other languages use a built-in common-keyword list)
- string literals hash as `$str`, numeric literals as `$num`
- `@Name`, `@a.b.Name` and `@Name(...)` runs are dropped in Java, Kotlin,
Scala, Groovy, Python, Dart, Swift, JavaScript and TypeScript
Each clone now carries `kind: exact | renamed`. DetectionToken gains a
raw_hash (equal to hash unless normalization rewrote the token); the
PreparedSource materializes raw hashes only when a token was normalized,
and the detector compares raw hashes over the matched run to classify a
pair. A dropped annotation salts the raw hash of the token before it, so a
pair differing only in annotations inside the matched run is `renamed`.
Reporters: console prints `Clone found (fmt, renamed)`, ai appends
`(renamed)`, json adds "kind", sarif uses rule jscpd/similar-code (declared
only when referenced), codeclimate the same check_name. Default runs are
unchanged apart from the additive "kind": "exact" JSON field.
Also: GitHub Action inputs, docs section, fixtures/type2-demo with a README
of runnable commands, unit tests in tokenizer/core/reporter, CLI flag
tests, and a binary-level integration test.
Runnable demonstrations of the four ways to exclude code from detection,
each with a clone that the mechanism removes at default thresholds:
- glob/ --ignore file globs (vendor/ and generated/ copies)
- regex/ --ignore-pattern / ignorePattern on a shared C# license header,
via CLI flag, --config, and cwd discovery of .jscpd.json
- markers/ jscpd:ignore-start / ignore-end around a generated Python block
- gitignore/ .gitignore honored by default, --no-gitignore to see the clone
README.md lists every command with the expected `Found N clones.` output,
including the comma-split and invalid-regex warning cases from this PR.
Detect duplicates across related formats via format equivalence groups
sharing one comparison pool, e.g. --cross-formats "javascript,typescript"
or the js-ts preset (javascript,jsx,typescript,tsx).
When a group mixes TypeScript with JavaScript, TS files are compared with
erasable type-only syntax stripped from the detection token stream (type
annotations, generics, interface/type declarations, as/satisfies, ?/!
markers, access modifiers, implements clauses, type-only imports/exports,
overload signatures, declare statements) so that
`function f(a: number): void` matches `function f(a)`. Stripping is
AST-driven (oxc_ast_visit over the already-parsed program) and only drops
tokens, so reported clone positions always reference the original source.
- cpd-tokenizer: new ts_strip module, tokenize_js_stripped,
TokenizeOptions.strip_types_formats (empty by default)
- cpd-finder: RunConfig.cross_formats, build_pools() pool grouping,
strip-set computation for TS-family formats grouped with JS
- cpd: --cross-formats flag, crossFormats config field (string,
array-of-strings, or array-of-arrays), js-ts preset, overlap merging,
unknown-format warnings, --debug output
- docs: rust.md option + Cross-Format Groups section, CHANGELOG entry
- tests: tokenizer strip tests, pool unit tests, cross-pool core test,
cpd-finder integration tests, CLI parse/config tests, e2e binary tests
with fixtures; default behavior unchanged (golden parity intact)
Non-erasable TS syntax with runtime semantics (enum, non-declare
namespace, parameter properties, import =, export =) is left intact and
will not cross-match. Cross-format clones are attributed to one member
format in per-format statistics.
Closes#810
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add fixtures/razor/ with two .razor (Blazor component) and two .cshtml
(MVC view) pairs containing deliberate duplications for the razor tokenizer.
Verified with cpd 5.0.11: 3 clones detected across the 4 files.
Replace the nested-quantifier string pattern /"(?:[^"\\]*|\\.)*"/
with /"(?:[^"\\]|\\[\s\S])*"/ in the Lisp/Elisp grammar.
The original pattern could catastrophically backtrack (O(2^n)) when a
string contained a backslash-newline line continuation, because JS dot
does not match newlines, leaving both alternatives unable to advance.
The fixed pattern eliminates nested quantifiers and uses [\s\S] so
line continuations are consumed correctly.
Adds a regression test with a 5s timeout guard and populates the
previously-empty fixtures/commonlisp/*.el fixture files with realistic
Elisp code that includes the backslash-newline pattern.
- Detect format from shebang line (#!) for executable extensionless files (chmod +x)
as fallback in skipNotSupportedFormats; symlinks excluded via lstatSync
- Add SHEBANG_TO_FORMAT map and readShebangFormat() in packages/finder/src/files.ts
- Introduce IEntry interface extending fast-glob Entry with detectedFormat? field
- Add --formats-names CLI option to map specific filenames (e.g. Makefile, Dockerfile)
to formats, independent of --formats-exts so neither disables built-in EXT_TO_FORMAT
- Wire formatsNames through IOptions, getDefaultOptions, CLI parser, and both
call sites (skipNotSupportedFormats, in-files-detector)
- Add fixtures/shebang/ with 4 executable extensionless fixtures (bash + python)
- 360 tests pass (277 tokenizer + 83 finder); README updated with new sections
Extract YAML front matter and fenced code blocks from .md files and
tokenize each segment with its resolved sub-language format (yaml,
typescript, python, javascript, etc.), enabling jscpd to detect
duplicates across languages embedded in Markdown documents.
- Implement tokenizeMarkdown() with three-tier resolveFormat() lookup
- Split markdown grammar into markdown-format.ts to break circular dep
- Dispatch markdown format through createTokenMapBasedOnCode()
- Separate .txt into its own format (removed from markdown.exts)
- Export EXT_TO_FORMAT from formats.ts for sub-language resolution
- Add 25-test suite covering all acceptance criteria (271 total)
- Add fixtures/markdown/file3.md + file4.md for integration testing
Adds Svelte Single-File Component tokenization to @jscpd/tokenizer.
Script, style, and markup template blocks are each tokenized with the
correct sub-format (javascript, typescript, css, scss, less, markup)
with accurate absolute line and column offset correction applied.
Dual-script (context="module" + instance) and all style lang aliases
are supported. Fixtures in fixtures/svelte/ confirm end-to-end clone
detection works across all three block types.
- Register 'astro' format with .astro extension in src/formats.ts
- Add tokenizeAstro() in src/languages/astro.ts: extracts and tokenizes
frontmatter (typescript), <script> blocks (js/ts by lang attr), <style>
blocks (css/scss/less by lang attr), and template (markup) independently
- Dispatch to tokenizeAstro() in createTokenMapBasedOnCode() for format 'astro'
- Add 31 tests in __tests__/astro.test.ts covering all extraction scenarios,
line offset correctness, CRLF normalization, ReDoS guard, empty frontmatter
- Add 6 test fixture files under __tests__/fixtures/astro/
- Update formats.test.ts and tokenize.test.ts with astro coverage entries
- Add end-to-end integration fixtures in fixtures/astro/ (component1/2.astro)
- javascript clonesCount 10→9: reprism merges one pair through enlarged clone
- rust clonesCount 4→6: reprism correctly detects two additional self-clones in file2.rs
- diff fixtures extended to 60+ shared lines so 50-token windows fit within the
shared section, restoring the expected clonesCount: 1 for diff format
modes.test.ts and options.test.ts referenced the deleted fixtures/modes
directory. Replace all fixture-based paths with os.tmpdir() temp files
created in beforeAll/cleaned in afterAll.
Also update formats.test.ts vue clonesCount 4→5: after the detector.ts
namespace-per-tokenMap fix, cross-file file.js↔file2.vue detection is
enabled, producing one additional clone. Add the vue fixture files
(file.js, app1.vue, app2.vue) that drive the integration test count.
Apply getModeHandler in detectClones so a string mode (e.g. 'weak') is
converted to the handler function before detection runs, matching the
behaviour of the CLI path. Adds tests for Vue SFC weak/mild mode
(issue #788) and for the detectClones API accepting a string mode.
Bumps all packages to 4.1.0.
Adds 92 new tests (178 → 270 total) with zero source-file modifications:
- packages/finder: 13 new test files covering all reporters, subscribers,
validators, hooks, utils, options, cloneFound, getFilesToDetect, and
InFilesDetector; shared buildClone() factory helper
- apps/jscpd: 3 new test files (options-unit, print, store) plus extensions
to formats, modes, and options integration tests
- vitest.config.ts added to both packages to restrict coverage.include
to src/** only, eliminating bin/ and prism.js noise
- Fixes 3 tokenizer bugs (trailing space in format lookup, tokenization
edge cases) uncovered by the new format fixture suite
- Adds fixture files for 40+ language formats to support formats.test.ts
Add support for detecting duplicate code in GDScript (.gd) files used
by the Godot game engine. GDScript is a Python-like language with its
own keywords and syntax.
Changes:
- Add gdscript format with .gd extension to formats.ts
- Create custom grammar for GDScript tokenization including:
- Comments (#)
- Strings (single/double/triple quotes)
- Annotations (@onready, @export, etc.)
- GDScript keywords (func, signal, extends, class_name, var, const, etc.)
- Built-in types (Vector2, Vector3, Dictionary, Array, etc.)
- Operators (:=, ->, etc.)
- Register grammar in grammar-loader.ts
- Add test fixtures
Fixes#758🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
apk