Copilot's review of #1062 raised seven points; each checked out and each is
fixed with a test that fails without it.
- A glob keeps its pattern. `import.meta.glob('./locales/*.js')` and
import(`./pages/${name}.vue`) used to keep every file under the directory
alive; they now reach what the pattern matches, `*` staying within one
segment as it does for the bundler. On Firefly an image glob,
'../../**/*.{png,jpg}', had been keeping all of src/ alive.
- Every pattern of an `import.meta.glob([...])` array is kept, not the first.
- A declared alias is not second-guessed: when `@` points at src/ and the file
is missing there, the `@/` project-root fallback no longer finds a
different file. A catch-all `*` path claims no prefix and does not count.
- Astro client <script> blocks are read as the modules Astro bundles them
into, so what they import is used. A script with any attribute but `src` is
left to the browser, as Astro leaves it.
- import('./x') in markup is found only inside expressions, not in comments
or plain text.
- A quoted attribute value is an expression only on a directive (`:prop`,
`@click`, `#slot`, `v-*`); elsewhere only its `{...}` interpolations are,
so `class="card"` no longer credits a declaration named `card`.
- An object key in markup, `{ label: value }`, is not a read.
Re-measuring on the thirty trending projects also exposed `?raw` imports, as
in `import script from './panzoom.js?raw'`: a bundler query now is stripped
before resolving. On one snapshot of those projects the previous release
reports 1093 unused files and this branch 481, with the share some import
names falling from 83% to 62%; the CHANGELOG now quotes those numbers
instead of ones measured against a development baseline. Components used
only from .mdx content are still reported, and the docs say so.
The CX column was one path per file plus every branch keyword the token
stream showed, and on the generic tokenizer that list was partly
unreachable: it splits punctuation into single characters, so the && and ||
entries never matched outside JavaScript and ?? counted twice. Measured
against lizard on ten GitHub-trending projects in nine languages, CX ranked
files with a median Spearman correlation of 0.83 and put their size at 0.75
of lizard's.
- Adjacent punctuation is joined before matching, so short-circuit
operators count in every language.
- Per-language rules: Rust match arms count by their =>, less one per match
since N arms are N paths; Swift adds guard and Go select. Where ? also
marks an optional (TypeScript, JavaScript, Swift, Kotlin, Groovy, C#) it
counts only as a ternary, told apart from String? and x?.y by being
written in the open; ?: is Elvis only in Kotlin and Groovy.
- One path per function: by declaring keyword (def, fn, func, function, fun,
and => for arrow functions), and in C, C++, Java, Objective-C and C# by the
head(args) { shape told apart from if (...) {. Languages with neither keep
the per-file baseline.
- The body of a """ or ''' string no longer counts. The line-based tokenizer
leaves its quotes in adjacent literals, and an odd number of triple quotes
in such a run opens or closes the string, which handles the PEP 257
opener """Summary. that a pair-of-tokens check read backwards. This is
limited to languages where """ is a delimiter: where a doubled quote is an
escape (C#, VB.NET, SQL, Pascal) it swallowed a 413-branch C# file.
- A keyword-shaped word the file binds as a name - case = 3, x.case,
f(case, when) - is not a branch in that file.
On the same ten projects the median correlation rose to 0.92 and the ratio
to 1.00, and every project improved. Where CX and lizard still differ they
differ on purpose: lizard counts a match once whatever its arms, and does
not count ??.
Detection, tokenization and clone output are untouched: the fixtures corpus
reports the same 248 clones. fixtures/summary-demo has one file per language
whose complexity can be counted by hand, and every CX value matches it.
basta reported most of a real Vue, Svelte or Astro project as dead: on
thirty GitHub-trending projects in those three ecosystems it flagged 1572
unused files, and some import in the tree contradicted 78% of them. The
misses were almost never in the analysis itself; they were imports only the
project's own build knows how to resolve. This reads them the way the build
does.
- Single-file components are read whole. The script of a .vue, .svelte or
.astro file is masked in place so positions stay real, and the markup is
scanned for the names it reads: <Foo />, kebab-case spellings, {{ }}
interpolations, Svelte actions, attribute expressions, and a literal
import() written in markup.
- Aliases come from vite.config.* resolve.alias and svelte.config.* kit.alias
as well as tsconfig paths, and SvelteKit's $lib is known without the
generated .svelte-kit/tsconfig.json no repository commits. The project's
own declaration now wins over the ~/ @/ project-root fallback.
- A computed import whose static head names a directory,
import(`./pages/${name}.vue`), and import.meta.glob('./locales/*.js') are
expanded into one namespace edge per file there, the way a bundler hands
back whole module objects. Such an import no longer marks its module as
dynamic access, so the module's own findings keep their confidence.
- In a monorepo a package name resolves through that package's package.json,
exports subpath by subpath with source conditions preferred over ./dist,
anywhere in the workspace pnpm-workspace.yaml, lerna.json, turbo.json or a
workspaces field marks out.
- A path literal in a build config resolves against the project root, and
Nitro's api/, routes/ and middleware/ are file-system routes.
On the same thirty projects reported unused files fell from 1572 to 468.
fixtures/dead-code-demo gains components/, bundler/ and monorepo/. On master
the bundler demo reports five files as certainly dead, four of them wrongly;
the monorepo demo reports badge.ts, which the app imports by package name.
Both now report only what is dead. The quote-aware byte scanning the markup
and config readers share lives in one place, lang::Quotes and
lang::outside_strings, so rust/crates keeps master's 25 clones.
Two defects found by checking every finding on the ten trending Python
repositories against an import graph rebuilt independently of the tool.
They accounted for 280 wrong findings out of 1,915; 4 remain, both in
cases where the checker is the generous one.
A quoted type annotation is a forward reference, not data. `def f(m:
"torch.nn.Conv1d")` reads `torch` exactly as surely as the unquoted form
does, and the `if TYPE_CHECKING:` idiom exists so that it *is* quoted —
the name may not be importable at run time. Without parsing them the
import that supplies the name looks unused, which is what made 275 of
834 unused-import findings wrong: 36% of the category, every one of them
reported at full confidence. The walk now tracks whether it is inside an
annotation and parses string literals found there with ruff's own
`parse_string_annotation`, which resolves in the source file's
coordinates so the references keep real offsets. Outside an annotation a
string stays what it was: weak evidence, never a read.
Absolute imports now also resolve against the roots the tree implies,
through a new optional `Analyzer::import_roots`. Python derives the
parent of every top-level package, which is what makes a `src/` layout
work from a file that belongs to no package: `utils/check_repo.py`
importing `transformers.testing_utils` matched no scan root and no
package of its own. Derived roots are tried last — a scan root is what
the user asked for, this is an inference.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Adds a second question about the same tree: not "what is duplicated" but
"what does nothing reach". A new `basta` crate builds the import graph from
the project's entry points and walks it, reporting unused files, unused
exports, unused module-private declarations, unused imports and — opt-in —
unused class members, across JavaScript, TypeScript, JSX, TSX and Python.
It ships two ways: `jscpd --dead-code` inside the existing binary, and a
standalone `basta` command. Both reuse the walker, the filters, all fifteen
reporters and the `--threshold` / `--exit-code` gates, so nothing new has to
be learned to run it.
Because it is a graph traversal rather than a reference count, dead code
cascades: a helper whose only caller is itself dead is reported too. Every
finding carries a confidence score and, below 100, the reasons it might be
wrong — `eval`, an unrecognised decorator, a wildcard re-export, a name seen
only in a string, a file in the scan that did not parse. `--min-confidence`
sets the floor at 60.
Entry points come from `package.json`, `pyproject.toml`, framework and
file-name conventions, and anything a shell script, CI workflow, Makefile or
Dockerfile names by path. CommonJS is read alongside ESM. A project that
renames its own import paths is read on its own terms: `paths` and `baseUrl`
from `tsconfig.json` and `jsconfig.json` are resolved, through a relative
`extends` chain and per package in a monorepo. A file named only by a path
string — a worker spawned through `new URL`, a build entry, a vitest setup
file — counts as used.
Languages plug in through one trait, `basta::lang::Analyzer`, which owns
parsing, specifier resolution, entry-point conventions, manifests, alias
configs and path traits. Nothing outside `lang/` is language-specific;
`docs/basta-extending.md` walks through adding one.
Validated against knip, fallow, vulture, skylos, fossil-mcp and repowise on
trending repositories. On a corpus of ten of them — 11k files, 3.3M lines —
every one of basta's 1,806 findings was checked against an import graph
rebuilt independently of it: no unused-file, unused-import or unused-symbol
claim survived as wrong.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
With `--follow-symlinks` the source id was the canonical path, so a file
reached through a link showed up as an absolute path outside the scan
root, `--ignore` (matched on the walked path) and the report disagreed
about its name, and a file reachable through two paths was two sources
(a file symlink next to its target even became a clone of itself).
The walked path, anchored at the canonical scan root, is now the id:
what reports show, what `--ignore` matches, what `--absolute` prints.
The canonical path travels alongside (`DiscoveredFile::real_path`,
`PreparedSource::real_path`) for reading the file and for the
`--skip-local` / `--skip-isolated` prefix checks. When links are
followed, files whose canonical path was already seen are dropped; the
entry that is the real file wins, else the first walked path.
Docs: `--follow-symlinks` row in the option table, the v4 default in the
migration table and README. Fixture `fixtures/follow-symlinks-demo`.
Closes#1059
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#375.
A snippet containing a NUL, an ANSI escape, a form feed or any other
control byte made the xml report unparseable: XML 1.0 has no way to
represent those characters, not even inside CDATA, so xmllint stopped at
"PCDATA invalid Char value 27" and every parser refused the file. The
reporter now replaces every character outside XML's Char production
(tab, LF, CR and U+0020 and up, minus U+FFFE/U+FFFF) with U+FFFD in
snippets and in path attributes. The existing "]]>" split is kept.
Path attributes were also escaped twice: quick-xml escapes attribute
values on its own, so the reporter's escape_xml() turned "r&d" into
"r&amp;d". The manual escape is gone and the sanitized path goes in
raw.
Tests parse the generated report to the end with quick-xml and check:
control characters become U+FFFD and none survive, a "]]>" round-trips
through the split sections, a path with & < " > is escaped exactly once
and reads back unchanged, and legal text is left borrowed.
Fixtures: fixtures/xml-report-demo with a clone whose body holds real
escape bytes, a form feed and a "]]>" literal; the README shows the
xmllint and Python checks. Verified the published 5.2.0 fails xmllint
on it and the fixed binary passes.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
print_appendices gave console-full the history block it lacked; the
History section now names both console reporters and an integration
test checks console-full renders it.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
The one-line sparkline was too small to read a trend from. The console
block now draws an eight-row bar chart: a percentage axis on the left
spanning the series' own min and max (a series moving between 2.0% and
2.4% would be a flat line on a 0-based axis), one column per point built
from eighth-block characters so every point keeps at least a sliver, an
x-axis, and point numbers that match a new `#` column in the table.
Past 20 points the columns narrow to one character so 30 points still
fit an 80-column terminal. The ai reporter keeps the compact sparkline.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#1002 (the --history half; --ratchet is deliberately not
implemented, the report shows the headroom instead).
`jscpd src --history v5.0.0..HEAD` (or `--history-since 2026-01-01`)
lists the commits with `git log`, checks each one out into a temporary
detached worktree (the --baseline-from-ref machinery, now shared through
baseline_ref::map_scan_paths), scans it with the run's own configuration
and keeps the totals; the current run is the series' last point.
`--history-every N` keeps every Nth commit counted from the newest,
`--history-limit N` (default 30) samples a long series evenly keeping
both ends. Nothing runs unless one of the flags is given.
Output: the console reporter appends a sparkline of the duplication
percentage, a table (commit, date, files, lines, clones, dup lines,
dup%, change, subject) with the change column red when duplication rose
and green when it fell, the overall trend line, and, when --threshold is
set and the latest value is under it, the headroom with the value the
threshold could be tightened to. The json reporter adds a `history`
object; the ai reporter prints one compact line per point. Other
reporters are unchanged.
New cpd_core::history (model, sparkline, change/headroom helpers with
unit tests), cpd::history (git log parsing, sampling, worktree scans),
cpd_reporter::history_render. Config keys history / historySince /
historyEvery / historyLimit; action input `history`. Integration tests
build a four-commit repo with fixed dates and check the JSON series,
the console block, every/limit thinning and the error paths. 969
workspace tests pass, clippy clean.
Docs: History section and option rows in docs/rust.md, README bullet,
action input row, fixtures/history-demo/README.md with a script that
builds the demo repository and the expected output.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#1047.
Four situations used to write an empty report and exit 0, so a CI job
that checks only the exit code saw a green run:
- `--format` naming a format that is not supported (a warning since #964)
is now `Error: --format: 'x' is not a supported format`, exit 1. Names
introduced through --formats-exts or --formats-names stay accepted.
- A scan path that does not exist is `Error: path does not exist: <p>`,
exit 1, before anything runs.
- A reporter that fails to write its output still prints the existing
`Reporter '<name>' error: ...` line; the run now ends with
`ERROR: a reporter failed to write its output` and exit 1, after the
other reporters have run.
- A scan that analyzes no files (paths exist, but nothing matched the
--format/--ignore/--pattern filters or every file was below
--min-tokens) prints `Warning: jscpd analyzed no files` and still
exits 0, so an intentionally empty tree passes. The new
`--fail-on-empty` flag (config key `failOnEmpty`, action input
`fail-on-empty`) makes it exit 1. Reports are written before the check.
Docs: an "Exit codes" section in docs/rust.md, option rows in
docs/rust.md and docs/ci-and-hooks.md, a README feature bullet.
Fixtures: fixtures/fail-on-empty-demo with a `code/` clone pair, a
`nothing/` comment-only file, and a README quoting every command's
output and exit code. Tests: the #964 warning test becomes an error
test, plus integration tests for each new exit path and for the config
key; 531 tests pass, clippy clean.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Reviews compare jscpd with detectors that "use language-specific
normalization" as if jscpd matched raw text. It does not: every format
has its own comment and string rules, JS/TS/JSX/TSX go through the oxc
parser (with TypeScript erasure for cross-format pools), Vue/Svelte/
Astro/Markdown/Razor are split into embedded languages, and tokens are
classified so the Type-2 pass keeps keywords while renaming identifiers.
The README intro and a new first Features bullet say so, and docs/rust.md
gains a "How detection works" section walking through format detection,
tokenization, modes and ignores, normalization, Rabin-Karp matching and
the near-miss passes, plus what stays out of scope (Type-4). Every claim
is taken from cpd-tokenizer (generic.rs comment styles, javascript.rs,
ts_strip.rs, sfc.rs, markdown.rs, razor.rs, tokenizer.rs mode filter,
strips_annotations, is_common_keyword).
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#1037.
Adds rust/scripts/build-pypi-wheels.py, which repacks the eight
jscpd-<platform>.tar.gz release tarballs into platform wheels: the same
binary under <pkg>.data/scripts/ as both `jscpd` and `cpd`, one wheel
per target, stdlib only, no maturin and no recompilation. glibc wheels
take their manylinux floor from the GLIBC_x.y symbols in the binary
(manylinux_2_34 today), musl builds are static so they tag as
musllinux_1_2, macOS and Windows use the Rust minimums. Semver
pre-releases map to PEP 440 (5.3.0-beta.1 -> 5.3.0b1); other labels
fail rather than guess.
release.yml gains a rust-publish-pypi job after the npm publish: it
takes the binary artifacts from this run, refills any missing target
from the published npm platform package (resumed releases, or a version
already on npm), builds the wheels, pip-installs the matching one and
checks `jscpd --version` and `cpd --version`, then uploads with PyPI
trusted publishing (environment `pypi`, no token, skip-existing). The
detect job treats a version missing from PyPI as pending, so the wheels
for the current release publish on the next push to master without a
new release commit. The GitHub release lists the PyPI package and waits
on the upload the same way it waits on crates.io and npm.
Docs: PyPI in the README install table and packages table, rust/README
and docs/rust.md install sections, and a `language: python` pre-commit
option in docs/ci-and-hooks.md so the hook no longer needs Node.js.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
NO_COLOR asks for plain output and nothing else, and many people export
it globally while still working in an interactive terminal. Treating it
as a tips switch would hide the tips from them for good. The non-TTY
check at the print site already covers pipes, files and CI logs, so the
environment switches stay CI and JSCPD_NO_TIPS.
Help text, docs and the skill table updated; the options test asserts
that NO_COLOR does not enable no_tips.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Sorting the html tokens (previous commit) restores source order, but the
wrapper tags of a Vue file — <template>, </template>, <script ...>,
</script>, <style ...>, </style> — are identical in every Vue file and
sit at both ends of it. In source order they stretch every template clone
to the last </style>: on fixtures/vue the html clone went from
[1:1 - 10:9] to [1:1 - 262:10] and duplicated lines from 289 to 541,
counting the script and style bodies as duplicated html.
A file with a <template> block keeps all of its markup inside that block,
so the skeleton outside the blocks is only wrapper tags and is now left
out of the html stream. The clone becomes the template body itself:
file1.vue:html [2:3 - 10:9], 75 tokens. Svelte and Astro have no template
wrapper; their top-level markup is the skeleton and is kept, so their
output is unchanged. The sort stays, for custom blocks that map to html.
Fixture: fixtures/sfc-demo/template-range with a README showing the
template-only range; docs/rust.md links it.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Public-output items from the #1020 review, done while nothing is released:
- SARIF and Code Climate rule names now match the clone vocabulary:
renamed -> jscpd/renamed-code, similar -> jscpd/similar-code
(was jscpd/similar-code for renamed and jscpd/near-miss-code for similar).
The similar-code rule text covers both the gap and the ast method.
- Duplicated-line statistics and the per-file summary count only the
matched lines of a merged clone. CpdClone gains a non-serialized
unmatched_lines field holding each fragment's gap lines, so enabling
the merge no longer moves --threshold.
- A merge whose similarity would fall below 0.5 (more unmatched than
matched tokens, e.g. one very long inserted line) is refused and the
halves stay separate; MIN_GAP_SIMILARITY documents the floor.
- Precedence is decided and documented: merged renamed halves are
reported as similar.
merge_gapped_clones is restructured around merge_into(); the running
matched-token counter was redundant with last.token_count and is gone.
Fixtures: type3-demo/long-line (refused merge) and type3-demo/renamed-halves
(precedence); README numbers re-measured for the whole directory.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Print the tips/sponsor lines only when stdout is a terminal, so CI logs,
pipes and agent hooks no longer get them without --no-tips. Also honor
NO_COLOR and JSCPD_NO_TIPS next to the existing CI env check.
Fixes#1008
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A JavaScript/TypeScript file with any parse diagnostic (a redeclaration,
a recoverable syntax error) was handed to the word-split fallback
tokenizer. Its tokens then never matched an oxc-tokenized file, so every
cross-file clone involving it vanished while its self-clones survived —
the symptom reported in #1023, where a second copy of a function in one
file made all a↔b clones disappear.
Tokens come from the lexer, not the parser: only `panicked` (or an empty
token stream) now triggers the fallback. Function extraction for
--similarity uses the partial AST on recoverable errors the same way.
Closes#1023.
`CpdClone.similarity_method` (JSON "method": "gap" | "ast") records which
near-miss mechanism produced a `similar` clone. The two scores are not on
one scale (gap = matched tokens over the merged span, ast = structural
Jaccard of whole functions), so the console prints
`Clone found (javascript, similar (gap) ~0.91)` /
`similar (ast) ~0.75`, the ai reporter `[~0.91 gap]` / `[~0.75 ast]`,
JSON carries "method" and SARIF a `similarity_method` property. Exact and
renamed clones are unchanged.
Function extraction is now a language plug-in: `FunctionExtractor`
(grammar id, served formats, extract) with a static `EXTRACTORS` registry
consulted by `extract_functions` / `supports_functions` /
`supported_function_formats`. `OxcExtractor` is the first entry (JS/TS/
JSX/TSX). Every RawFunction/FunctionSig records its grammar and the scorer
refuses pairs across grammars, so a tree-sitter-backed extractor for
another language is a self-contained addition with no change to scoring,
CLI, MCP or reporters. Module docs describe the two steps; the MCP note
lists supported formats from the registry.
Off by default. `--similarity RATIO` (config `similarity`, Action input
`similarity`, MCP `check_duplication` argument) summarizes every function,
method and arrow function of a JavaScript/TypeScript/JSX/TSX file by the
bag of 4-grams over the pre-order sequence of its AST node types (oxc),
indexes the bags with MinHash + LSH banding, and reports pairs whose
weighted Jaccard index reaches RATIO as `similar` clones with a
`similarity` value. Names and literal values are not part of the summary:
a renamed copy scores 1.0, one inserted line ~0.9, two inserted statements
plus renames ~0.75.
Functions must clear --min-tokens/--min-lines, nested functions are never
paired with their parent, and pairs already covered by an exact, renamed
or merged clone are skipped. Values outside (0, 1] warn and disable the
option. Reporting reuses the Type-3 vocabulary (console `similar ~0.75`,
JSON kind/similarity, SARIF jscpd/near-miss-code).
New: cpd-core::similarity (FunctionSig, SimilarityIndex, bag_jaccard,
minhash), cpd-tokenizer::functions (oxc visitor extractor), MCP function
index built lazily per scan. fixtures/type3-demo/similar-functions with
README numbers, docs, unit + integration + MCP stdio tests. Also adds the
missing max_gap_lines/similarity fields to the --debug config output.
Off by default (0). When set to N, clones of the same file pair whose
fragments follow each other in both files with at most N unmatched lines
between them are merged into one clone of kind `similar`, with `tokens` =
matched tokens and `similarity` = matched tokens over the tokens of the
longer merged span. Adjacent exact windows that share a boundary token are
counted once; chains merge transitively; overlapping or reordered fragments
never merge. The pass runs after every other filter and returns its input
untouched at 0, so default output is unchanged.
Reporting: console `Clone found (fmt, similar ~0.91)`, ai `[~0.91]`, JSON
"kind"/"similarity" (rounded to three decimals), SARIF rule
jscpd/near-miss-code with a similarity property (declared only when
referenced), Code Climate check_name.
Config key maxGapLines (kebab alias), GitHub Action input max-gap-lines,
docs section, fixtures/type3-demo with a README of runnable commands, unit
tests for the merge pass and reporters, and a binary-level integration test.
Closes#998.
Three opt-in flags (config keys ignoreIdentifiers, ignoreLiterals,
ignoreAnnotations) normalize token classes before hashing so that blocks
which differ only in names, values or annotations are detected:
- identifiers hash as `$id`; keywords are kept (the JS/TS tokenizer
classifies them, other languages use a built-in common-keyword list)
- string literals hash as `$str`, numeric literals as `$num`
- `@Name`, `@a.b.Name` and `@Name(...)` runs are dropped in Java, Kotlin,
Scala, Groovy, Python, Dart, Swift, JavaScript and TypeScript
Each clone now carries `kind: exact | renamed`. DetectionToken gains a
raw_hash (equal to hash unless normalization rewrote the token); the
PreparedSource materializes raw hashes only when a token was normalized,
and the detector compares raw hashes over the matched run to classify a
pair. A dropped annotation salts the raw hash of the token before it, so a
pair differing only in annotations inside the matched run is `renamed`.
Reporters: console prints `Clone found (fmt, renamed)`, ai appends
`(renamed)`, json adds "kind", sarif uses rule jscpd/similar-code (declared
only when referenced), codeclimate the same check_name. Default runs are
unchanged apart from the additive "kind": "exact" JSON field.
Also: GitHub Action inputs, docs section, fixtures/type2-demo with a README
of runnable commands, unit tests in tokenizer/core/reporter, CLI flag
tests, and a binary-level integration test.
First release from the v5-only master. Bumps jscpd to 5.1.2 and every
sub-crate (cpd-core 0.1.11, cpd-tokenizer 0.1.13, cpd-finder 0.1.14,
cpd-reporter 0.1.12) so the new crates.io metadata from #988 and the
Windows drive-path fix in cpd-finder are republished. Adds the 5.1.2
changelog section (linux-arm64-musl package, cargo-binstall, Docker
image, jscpd --version naming, pre-commit hook args, repository split)
and refreshes the version examples in docs/.
Pre-release checks: cargo fmt, clippy -D warnings, cargo test --locked
(22 suites), cargo check --all-targets --target x86_64-pc-windows-msvc,
release-notes extraction dry-run against rust/CHANGELOG.md.
Everything belonging to the v4 TypeScript engine moves to the master-v4
branch: apps/, packages/, build-utils/, changesets, pnpm workspace, turbo,
the Node.js CI workflow and the ts-* release jobs. What remains is the
Rust workspace under rust/, the npm wrapper and platform packages, the
GitHub Action, Docker image, flake, skills, benchmark and fixtures/.
- rust.yml gains a smoke job that runs the release binary against
fixtures/ (json, sarif, html) and asserts clones and reporters
- release.yml keeps only the Rust jobs; pnpm removed from every workflow;
audit.yml keeps cargo audit + cargo deny; dependabot drops the root npm entry
- root package.json becomes a thin private package depending on the exact
jscpd@5 version so the pre-commit hook keeps installing the binary;
sync-version.mjs keeps it in step
- .pre-commit-hooks.yaml used v4-only flags that the v5 CLI rejects; fixed
- FORMATS.md is now generated from the Rust tokenizer (224 formats)
- README, docs, CONTRIBUTING, SECURITY, package and crate descriptions,
benchmark and skills rewritten for v5 on its own terms: no v5-vs-v4 speed
multipliers anywhere, a single "Looking for v4?" pointer to master-v4
Rust / crates.io
- report the invoked binary name in --version/--help (jscpd vs cpd)
- add repository, documentation, keywords, categories to all crates
- exclude tests/ from the jscpd crate package; add cargo-binstall metadata
- expand the crates.io README and render it on docs.rs
npm
- add funding field to jscpd, cpd and jscpd@4 packages
- add jscpd-linux-arm64-musl platform package and clearer unsupported-platform error
- enable provenance for the v4 changeset publish
Pipelines
- rust.yml: windows-latest in the matrix, drop dead schedule guard
- release.yml: cosign-sign release assets, floating v5 tag, LICENSE in tarballs,
invoke the new docker.yml (GHCR multi-arch image) after each release
- audit.yml: nightly cargo audit, cargo deny and pnpm audit
- Dockerfile: multi-stage image from release binaries (distroless), mcp target for Glama
Docs / community
- README: shields badges, single primary install per OS, GitHub Action snippet,
link to jscpd.dev, v5.1 features, 15 reporters, 224 formats, 8 platforms
- docs: @v5 action tag, Docker usage, baseline section, current crate versions
- YAML issue forms with engine selector, config.yml, CODEOWNERS
- remove stale .github/CONTRIBUTING.md stub, custom.md, TODO.md; untrack rust/jscpd-report.json
- examples: mark as v4 API, add a Rust cpd-finder example
Auto-discovery now also checks .config/jscpd.json (and
.config/.jscpd.json) between the root .jscpd.json and the package.json
jscpd key, per https://dot-config.github.io/. Root .jscpd.json keeps
precedence, so the new location is optional and never changes behavior
for existing setups. Paths inside the config still resolve relative to
the working directory, as with other auto-discovered sources.
Closes#979
Claude-Session: https://claude.ai/code/session_01VvWzWfEK6oV4HTe6EoGkff
Port the skipIsolated option from jscpd PR #628 to the Rust engine:
clones spanning two different folders of the same isolation group are
dropped, so team-owned monorepo packages are not compared with each
other while everything still matches against shared code.
- CLI: --skip-isolated / --skipIsolated "packages/a|packages/b,libs/a|libs/b"
- Config file: skipIsolated as nested arrays (jscpd shape)
- Detection: PathFilters bundles skip_local + skip_isolated path checks,
applied in both the primary and secondary clone passes
- MCP server: isolation groups apply to project scans; snippet checks
remain unfiltered
Improvements from reviewing the MCP tool surface against real agent
usage:
- Clone/match lists are sorted biggest-first (token_count desc) before
the 'limit' cap. Detection emits path order, so truncation previously
kept the alphabetically-first clones instead of the most important
ones. Applies to check_current_directory, check_duplication, and the
new get_file_clones (project clones sorted once per scan).
- check_duplication accepts file extensions as format aliases: "js",
"py", "tsx" resolve via get_format_by_extension, removing the most
likely first-use failure for agents that pass the extension. The
resolved format is echoed in the payload.
- check_duplication gains the same 'limit' argument (default 100) and
truncation contract as check_current_directory: 'count' is the
untruncated total, 'returned' the list length, 'note' on truncation.
- New tool get_file_clones(path, limit?) — clones involving one file,
for the dominant file-scoped agent workflow ("what in this file is
duplicated, where?"). Accepts scan-root-relative or absolute paths,
folds embedded-code-block fragment ids into their parent file, and
distinguishes "no clones" from "file was not part of the scan".
- display_path normalizes separators to '/' so tool output is uniform
across platforms.
7 new unit tests (alias resolution, limit contract, file filter,
unscanned-file note, missing path, biggest-first ordering across
differently-sized clone pairs); tools/list coverage updated to the
four-tool surface. Full workspace suite green.
Refs #891
check_current_directory previously answered with counts only, which
left an MCP client unable to see where the duplication actually is.
The server now keeps the clones from the last scan and returns them:
- duplications: [{format, fileA, startA, endA, fileB, startB, endB,
lines, tokens}] with paths relativized to the scan roots and
sub-format fragment ids (embedded code blocks) folded into their
parent file
- optional 'limit' argument (default 100) caps the list so heavily
duplicated projects cannot flood an LLM context; 'clones' always
carries the untruncated total, 'returned' the list length, and a
'note' marks truncation — no silent capping
- non-integer 'limit' is rejected with -32602
Unit test asserts list shape, cross-file attribution, and the
truncation contract; the end-to-end stdio session now exercises the
tool as well.
Refs #891
Closes the gap reported in #891: the HTTP-only jscpd-server cannot be
used by MCP clients that spawn and manage a stdio process themselves
(Claude Desktop/Code, Cursor, APM, ...). The Rust binary now serves the
Model Context Protocol directly:
cpd --mcp /path/to/project
Transport & protocol:
- newline-delimited JSON-RPC 2.0 on stdin/stdout; stdout carries
protocol messages only, logging goes to stderr; exits 0 on stdin EOF
- protocol revision 2025-06-18, accepting 2025-03-26 and 2024-11-05
clients via version negotiation; ping, tools/list, tools/call;
notifications ignored; parse errors reported (-32700) without
killing the session
Tools (mirroring the HTTP jscpd-server):
- check_duplication(code, format) — checks a snippet against the
scanned project; results are compact JSON in a text content block
with file/snippet line ranges; unknown formats and sub-threshold
snippets produce informative results
- get_statistics() — files, clones, full Statistics of the last scan
- check_current_directory() — re-scan the configured paths
Performance: the project is scanned once at startup and cached as
detection-ready token hashes (PreparedSource pools per format /
cross-format group), so snippet checks only tokenize the snippet and
run one pool detection — milliseconds even on large codebases. All
standard detection options (--min-tokens, --format, --cross-formats,
--ignore, ...) apply to the scan and to snippet checks.
Supporting refactor in cpd-finder: orchestrate::run() split into
build_thread_pool() + prepare_scan_in() (walk + tokenize, returning
PreparedScan) so the MCP server can keep prepared sources around;
run() behavior is unchanged and detect_prepared is untouched.
strip_types_formats made pub for snippet tokenization parity;
PreparedSource now derives Clone. hyperfine A/B vs a master-built
baseline shows no regression on the normal CLI path (39.0ms vs
41.2ms, within noise).
Tests: 12 unit tests for the protocol handler and tools, plus 2
end-to-end tests that spawn the real binary and drive a full stdio
session (initialize → tools/list → tools/call → EOF), asserting
stdout is protocol-only. Full workspace suite green.
Docs: docs/ai-ready.md MCP section restructured into stdio (Rust v5)
and Streamable HTTP (Node v4) transports with client config examples;
docs/rust.md options table; root README; rust/CHANGELOG.md 5.0.16.
The jscpd-server (Node) stdio subcommand asked for literally in #891
can follow separately; this provides the stdio transport via the v5
binary that ships with jscpd@5.
Refs #891
Adds a refactoring-hotspot summary computed from data the detector
already holds in memory, appended to the run output when requested:
- --summary enable the summary block (also `summary` in .jscpd.json)
- --summary-top <n> entries per list (default 10, `summaryTop`)
- --summary-by <m> ranking metric: tokens|lines|size|complexity (`summaryBy`)
Output: top-N files (tokens, lines, size, complexity estimate, dup%)
strictly ranked by the chosen metric — every row carries all metrics —
plus a top-N folder rollup (direct-parent aggregation, each file counted
once). totalFiles/totalFolders always report pre-truncation counts.
Complexity is a language-agnostic cyclomatic estimate: 1 + decision-point
tokens (if/elif/elsif/elseif/unless/for/foreach/while/until/case/cond/
when/catch/rescue/except/andalso/orelse/&&/||/and/or/?/??), matched
ASCII-case-insensitively so uppercase-keyword languages (SQL, PL/SQL,
Fortran, COBOL, BASIC) count too. Verified across all 157 fixture
languages.
Reporter integration:
- console / console-full: aligned tables after the normal output
- ai: compact one-line-per-entry block (LLM-token-efficient)
- json: additive `summary` key, absent when the flag is off — schema
unchanged for existing consumers
Performance: the only unconditional change is capturing the file's byte
size from the existing mmap; all summary work runs post-detection and
only when --summary is set. hyperfine A/B vs master baseline shows
identical times without the flag (39.8ms vs 40.2ms) and ~2ms summary
cost when enabled.
New: cpd-core/src/summary.rs (model + computation, 14 unit tests),
cpd-reporter/src/summary_render.rs. ReportContext gains an optional
summary reference (ReportContext::new is unchanged; with_summary added).
SourceFile gains a serde-defaulted `bytes` field. 6 new CLI integration
tests; full workspace suite green.
Follow-ups tracked in #934: HTML report section, TypeScript parity port.
Refs #934
Take master's lockfile (carries the security overrides from 1b03f0e)
and re-resolve this branch's @modelcontextprotocol/server + node v2
dependencies on top, so the overrides apply to the new subtree too.
Verified: --frozen-lockfile passes, jscpd-server build/typecheck and
all 94 tests pass, full workspace build and tests green.
Refs #908
Extends the size-based severity from 294de6b: when the existing
--threshold option is set and the run's total duplication percentage
exceeds it, every SARIF result is emitted at level "error" instead of
"warning" — the report now reflects the same failure the
ThresholdReporter exits non-zero for.
Uses the identical strictly-greater comparison as the ThresholdReporter
(equal to the threshold does not fail the build, so it does not
escalate severity either). Composes with --sarif-error-tokens: either
condition escalates a result. Default output is unchanged when
--threshold is unset.
Fixes#908
All SARIF results were emitted at level "warning" regardless of clone
size, so large copy-pastes and trivial duplicates looked identical to
SARIF consumers like GitHub code scanning.
Add an opt-in cutoff: clones with token_count >= --sarif-error-tokens
are reported at level "error"; smaller clones (and all clones when the
flag is unset) stay "warning", so default output is unchanged. The
option is also available as sarifErrorTokens / sarif-error-tokens in
.jscpd.json, and the rule's defaultConfiguration remains "warning"
(per-result level overrides it).
No default cutoff is imposed — the issue explicitly asked not to invent
an arbitrary one, so severity escalation is purely opt-in.
The Streamable HTTP transport requires servers to validate the Origin
header on every connection, and /mcp had no such check while listening on
the default 0.0.0.0 binding, so a browser could reach the tools through
DNS rebinding.
Guard the endpoint with the SDK's own request guards. Origin validation
is always on: a request with no Origin header is served, since non-browser
MCP clients send none, while a present, unlisted origin is answered with
403 and a JSON-RPC error carrying no id. Host validation stays a policy
decision — a loopback bind gets the localhost allowlist automatically, a
deliberate external bind is only restricted once the operator names the
hostnames, so remote deployments keep working. Both lists are extendable
with the new --allowed-origin and --allowed-host options.
Also make the test SSE reader follow the event framing rules: it now
splits events on blank lines, joins each event's data fields with a
newline and skips comment keep-alives, so a multi-line payload no longer
fails to parse and a notification emitted before the result is no longer
mistaken for the response.
Assert the cache fields on server/discover as well: the 2026-07-28
caching rules list it as a cacheable result alongside the list
operations.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Migrate the jscpd-server MCP endpoint from the monolithic
@modelcontextprotocol/sdk v1 to the split v2 packages
(@modelcontextprotocol/server and @modelcontextprotocol/node) and serve
protocol revision 2026-07-28 through the SDK's own primitives instead of
hand-rolled transport wiring.
- replace the session map and the initialize-gated POST handler with
createMcpHandler + toNodeHandler mounted on every method of /mcp, so
modern requests are direct and stateless: no initialize, no
Mcp-Session-Id, protocol version and client capabilities travel in the
per-request _meta envelope, and server/discover answers discovery
- let the SDK own the Streamable HTTP behaviour: MCP-Protocol-Version,
Mcp-Method and Mcp-Name header validation, the resultType
discriminator on every result, and the 405 answers for the 2025-era
session operations
- configure per-operation cache hints so cacheable results carry the
required ttlMs/cacheScope, with a shorter, private hint for the
statistics resource that changes on every rescan
- keep dual-era compatibility: 2025-era clients are served by the
stateless legacy fallback backed by the very same server factory
- hold the service and the MCP endpoint in explicit application state
instead of handler closures and an implicit per-session transport map
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The console reporters render their tables with cli-table3, which styles
the head and borders through its own @colors/colors dependency - a
separate singleton that configureColors() never touched. Piped output
therefore still carried escape sequences for the table, which is the
`jscpd . > out.txt` case from #893.
Toggling that second singleton is not reliable under pnpm (it is a
transitive dependency of cli-table3 and does not resolve from our own
packages), so the reporters now pass an explicit cli-table3 `style`
instead: the library defaults while colors are on, no styling at all
once they are off.
The helper reads `colors/safe` through `require` rather than a namespace
import on purpose. The published bundle compiles `import * as colors`
into a plain-object copy of the module exports, which snapshots
`enabled` at load time - the fix worked under vitest but not in the
built CLI until this was changed.
Also:
- FORCE_COLOR now takes precedence over NO_COLOR, and an empty
FORCE_COLOR counts as force-on, matching Node core, chalk and
@colors/colors. Docs updated.
- detectClones() checks the merged options object in the guard as well
as in the call, so the two cannot diverge.
- Add reporter-level regression tests asserting the rendered console and
consoleFull output carries no escape sequences once colors are off,
and still does while they are on.
apk