A third installable agent skill, broader than dry-refactoring's
duplication-only scope: find and fix duplicated code (delegates to
dry-refactoring), then find and remove or refactor dead code
(--dead-code), then find and simplify the largest/most complex files
(--complexity) — prioritized from --health/--dashboard's three
sub-scores and re-measured at the end to confirm the pass helped.
Also fills a gap the new skill's own cross-reference exposed: the
jscpd skill's option table never mentioned --dead-code at all despite
listing --complexity and --dashboard/--health. Added it and its
related flags, and cross-linked both new-workflow directions from the
jscpd skill's closing section. README.md and docs/ai-ready.md's skill
registries updated to list all three.
- Console health badge: reverse-video grade chip, eighth-block gauges per
dimension, one row per sub-score with what it was measured from.
- markdown/html reporters for --dashboard and --health, alongside the
existing console/json/badge; the health SVG badge moves into badge.rs.
- --dashboard's Duplication section lists a breakdown by format instead of
by file, sourced from the same jscpd statistics as everywhere else.
- Duplication no longer counts markup, stylesheet and template clones
(HTML, CSS, Handlebars, ...): a repeating style rule is not the
maintenance problem repeating programming logic is. The exclusion
follows the clone's own format, not the file it lives in, so a .svelte
or .vue file's style/template block is left out even though the
component itself is full weight.
- The duplication line and format-breakdown table now name what they
actually measured and, only for categories the project has files in,
what was left out (e.g. "5.4% in typescript (no text)") instead of a
fixed disclaimer.
One 0-100 score with an A-E grade, shown on top of --dashboard and alone
with --health.
- Dimensions are shares of the code lines, so size cancels out:
duplication (jscpd's percentage over code files), dead code, and the
share of code in complex files (complexity 50+). Prose and data files
are not counted.
- Each share is scored on a half-life curve (100 at zero, 50 at one
half-life), calibrated on 42 open-source projects so the median project
scores 75 per dimension; shares are mixed with a 2000-line prior so one
finding does not sink a small project.
- Sub-scores combine as a weighted geometric mean. A dimension that
cannot be measured is left out and named, and dead code weighs as much
as the share of the code it could read.
- --health-input FILE and the health config object add metrics from
other tools (a ready score, or a value with a half-life) and tune the
built-in half-lives and weights.
- Reporters: console badge, ai one-liner, json (jscpd-health.json,
jscpd-dashboard.json with every dashboard section) and badge
(jscpd-health-badge.svg). Exit gates apply as in the dashboard.
CpdClone::matched_lines moves to cpd-core so statistics and health count
duplicated lines the same way.
- --dashboard applies the exit gates of a clone run (--threshold,
--exit-code, --fail-on-empty) instead of always returning 0; the
baseline family warns and --fail-on-new-clones is refused, since no
clone report is written.
- A refused dead-code configuration (bad --dead-code-categories, a
--format no analyzer supports) now exits 1 as in --dead-code, rather
than printing a dashboard with an empty dead-code section.
- --workers is a budget for the whole dashboard run: the two concurrent
scans get half each, so neither the thread count nor the 64 MiB
tokenizer stacks double. --workers 1 runs them sequentially.
- --complexity scans with similarity disabled, so --similarity no longer
pays for function extraction it cannot use.
- --complexity honours a summaryBy from the config file, not only the
--summary-by flag (Options::summary_by_set).
- reStructuredText joins the prose formats at complexity 0.
--dashboard prints one screen: project size and largest formats,
duplication with clones per kind and the most duplicated files, total and
mean complexity with the most complex files, and dead code by category
with the largest findings (JavaScript, TypeScript, Python). The dead-code
scan runs beside clone detection; --summary-top sets the rows per list.
Data formats (JSON, YAML, TOML, CSV, ...) join markdown at complexity 0,
so a lock file's version ranges no longer top the complexity ranking.
Docs for --kind, --complexity and --dashboard; fixtures/dashboard-demo,
plus --kind and --complexity sections in the type3 and summary demos.
Copilot's review of #1062 raised seven points; each checked out and each is
fixed with a test that fails without it.
- A glob keeps its pattern. `import.meta.glob('./locales/*.js')` and
import(`./pages/${name}.vue`) used to keep every file under the directory
alive; they now reach what the pattern matches, `*` staying within one
segment as it does for the bundler. On Firefly an image glob,
'../../**/*.{png,jpg}', had been keeping all of src/ alive.
- Every pattern of an `import.meta.glob([...])` array is kept, not the first.
- A declared alias is not second-guessed: when `@` points at src/ and the file
is missing there, the `@/` project-root fallback no longer finds a
different file. A catch-all `*` path claims no prefix and does not count.
- Astro client <script> blocks are read as the modules Astro bundles them
into, so what they import is used. A script with any attribute but `src` is
left to the browser, as Astro leaves it.
- import('./x') in markup is found only inside expressions, not in comments
or plain text.
- A quoted attribute value is an expression only on a directive (`:prop`,
`@click`, `#slot`, `v-*`); elsewhere only its `{...}` interpolations are,
so `class="card"` no longer credits a declaration named `card`.
- An object key in markup, `{ label: value }`, is not a read.
Re-measuring on the thirty trending projects also exposed `?raw` imports, as
in `import script from './panzoom.js?raw'`: a bundler query now is stripped
before resolving. On one snapshot of those projects the previous release
reports 1093 unused files and this branch 481, with the share some import
names falling from 83% to 62%; the CHANGELOG now quotes those numbers
instead of ones measured against a development baseline. Components used
only from .mdx content are still reported, and the docs say so.
The CX column was one path per file plus every branch keyword the token
stream showed, and on the generic tokenizer that list was partly
unreachable: it splits punctuation into single characters, so the && and ||
entries never matched outside JavaScript and ?? counted twice. Measured
against lizard on ten GitHub-trending projects in nine languages, CX ranked
files with a median Spearman correlation of 0.83 and put their size at 0.75
of lizard's.
- Adjacent punctuation is joined before matching, so short-circuit
operators count in every language.
- Per-language rules: Rust match arms count by their =>, less one per match
since N arms are N paths; Swift adds guard and Go select. Where ? also
marks an optional (TypeScript, JavaScript, Swift, Kotlin, Groovy, C#) it
counts only as a ternary, told apart from String? and x?.y by being
written in the open; ?: is Elvis only in Kotlin and Groovy.
- One path per function: by declaring keyword (def, fn, func, function, fun,
and => for arrow functions), and in C, C++, Java, Objective-C and C# by the
head(args) { shape told apart from if (...) {. Languages with neither keep
the per-file baseline.
- The body of a """ or ''' string no longer counts. The line-based tokenizer
leaves its quotes in adjacent literals, and an odd number of triple quotes
in such a run opens or closes the string, which handles the PEP 257
opener """Summary. that a pair-of-tokens check read backwards. This is
limited to languages where """ is a delimiter: where a doubled quote is an
escape (C#, VB.NET, SQL, Pascal) it swallowed a 413-branch C# file.
- A keyword-shaped word the file binds as a name - case = 3, x.case,
f(case, when) - is not a branch in that file.
On the same ten projects the median correlation rose to 0.92 and the ratio
to 1.00, and every project improved. Where CX and lizard still differ they
differ on purpose: lizard counts a match once whatever its arms, and does
not count ??.
Detection, tokenization and clone output are untouched: the fixtures corpus
reports the same 248 clones. fixtures/summary-demo has one file per language
whose complexity can be counted by hand, and every CX value matches it.
basta reported most of a real Vue, Svelte or Astro project as dead: on
thirty GitHub-trending projects in those three ecosystems it flagged 1572
unused files, and some import in the tree contradicted 78% of them. The
misses were almost never in the analysis itself; they were imports only the
project's own build knows how to resolve. This reads them the way the build
does.
- Single-file components are read whole. The script of a .vue, .svelte or
.astro file is masked in place so positions stay real, and the markup is
scanned for the names it reads: <Foo />, kebab-case spellings, {{ }}
interpolations, Svelte actions, attribute expressions, and a literal
import() written in markup.
- Aliases come from vite.config.* resolve.alias and svelte.config.* kit.alias
as well as tsconfig paths, and SvelteKit's $lib is known without the
generated .svelte-kit/tsconfig.json no repository commits. The project's
own declaration now wins over the ~/ @/ project-root fallback.
- A computed import whose static head names a directory,
import(`./pages/${name}.vue`), and import.meta.glob('./locales/*.js') are
expanded into one namespace edge per file there, the way a bundler hands
back whole module objects. Such an import no longer marks its module as
dynamic access, so the module's own findings keep their confidence.
- In a monorepo a package name resolves through that package's package.json,
exports subpath by subpath with source conditions preferred over ./dist,
anywhere in the workspace pnpm-workspace.yaml, lerna.json, turbo.json or a
workspaces field marks out.
- A path literal in a build config resolves against the project root, and
Nitro's api/, routes/ and middleware/ are file-system routes.
On the same thirty projects reported unused files fell from 1572 to 468.
fixtures/dead-code-demo gains components/, bundler/ and monorepo/. On master
the bundler demo reports five files as certainly dead, four of them wrongly;
the monorepo demo reports badge.ts, which the app imports by package name.
Both now report only what is dead. The quote-aware byte scanning the markup
and config readers share lives in one place, lang::Quotes and
lang::outside_strings, so rust/crates keeps master's 25 clones.
Two defects found by checking every finding on the ten trending Python
repositories against an import graph rebuilt independently of the tool.
They accounted for 280 wrong findings out of 1,915; 4 remain, both in
cases where the checker is the generous one.
A quoted type annotation is a forward reference, not data. `def f(m:
"torch.nn.Conv1d")` reads `torch` exactly as surely as the unquoted form
does, and the `if TYPE_CHECKING:` idiom exists so that it *is* quoted —
the name may not be importable at run time. Without parsing them the
import that supplies the name looks unused, which is what made 275 of
834 unused-import findings wrong: 36% of the category, every one of them
reported at full confidence. The walk now tracks whether it is inside an
annotation and parses string literals found there with ruff's own
`parse_string_annotation`, which resolves in the source file's
coordinates so the references keep real offsets. Outside an annotation a
string stays what it was: weak evidence, never a read.
Absolute imports now also resolve against the roots the tree implies,
through a new optional `Analyzer::import_roots`. Python derives the
parent of every top-level package, which is what makes a `src/` layout
work from a file that belongs to no package: `utils/check_repo.py`
importing `transformers.testing_utils` matched no scan root and no
package of its own. Derived roots are tried last — a scan root is what
the user asked for, this is an inference.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Adds a second question about the same tree: not "what is duplicated" but
"what does nothing reach". A new `basta` crate builds the import graph from
the project's entry points and walks it, reporting unused files, unused
exports, unused module-private declarations, unused imports and — opt-in —
unused class members, across JavaScript, TypeScript, JSX, TSX and Python.
It ships two ways: `jscpd --dead-code` inside the existing binary, and a
standalone `basta` command. Both reuse the walker, the filters, all fifteen
reporters and the `--threshold` / `--exit-code` gates, so nothing new has to
be learned to run it.
Because it is a graph traversal rather than a reference count, dead code
cascades: a helper whose only caller is itself dead is reported too. Every
finding carries a confidence score and, below 100, the reasons it might be
wrong — `eval`, an unrecognised decorator, a wildcard re-export, a name seen
only in a string, a file in the scan that did not parse. `--min-confidence`
sets the floor at 60.
Entry points come from `package.json`, `pyproject.toml`, framework and
file-name conventions, and anything a shell script, CI workflow, Makefile or
Dockerfile names by path. CommonJS is read alongside ESM. A project that
renames its own import paths is read on its own terms: `paths` and `baseUrl`
from `tsconfig.json` and `jsconfig.json` are resolved, through a relative
`extends` chain and per package in a monorepo. A file named only by a path
string — a worker spawned through `new URL`, a build entry, a vitest setup
file — counts as used.
Languages plug in through one trait, `basta::lang::Analyzer`, which owns
parsing, specifier resolution, entry-point conventions, manifests, alias
configs and path traits. Nothing outside `lang/` is language-specific;
`docs/basta-extending.md` walks through adding one.
Validated against knip, fallow, vulture, skylos, fossil-mcp and repowise on
trending repositories. On a corpus of ten of them — 11k files, 3.3M lines —
every one of basta's 1,806 findings was checked against an import graph
rebuilt independently of it: no unused-file, unused-import or unused-symbol
claim survived as wrong.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
With `--follow-symlinks` the source id was the canonical path, so a file
reached through a link showed up as an absolute path outside the scan
root, `--ignore` (matched on the walked path) and the report disagreed
about its name, and a file reachable through two paths was two sources
(a file symlink next to its target even became a clone of itself).
The walked path, anchored at the canonical scan root, is now the id:
what reports show, what `--ignore` matches, what `--absolute` prints.
The canonical path travels alongside (`DiscoveredFile::real_path`,
`PreparedSource::real_path`) for reading the file and for the
`--skip-local` / `--skip-isolated` prefix checks. When links are
followed, files whose canonical path was already seen are dropped; the
entry that is the real file wins, else the first walked path.
Docs: `--follow-symlinks` row in the option table, the v4 default in the
migration table and README. Fixture `fixtures/follow-symlinks-demo`.
Closes#1059
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#375.
A snippet containing a NUL, an ANSI escape, a form feed or any other
control byte made the xml report unparseable: XML 1.0 has no way to
represent those characters, not even inside CDATA, so xmllint stopped at
"PCDATA invalid Char value 27" and every parser refused the file. The
reporter now replaces every character outside XML's Char production
(tab, LF, CR and U+0020 and up, minus U+FFFE/U+FFFF) with U+FFFD in
snippets and in path attributes. The existing "]]>" split is kept.
Path attributes were also escaped twice: quick-xml escapes attribute
values on its own, so the reporter's escape_xml() turned "r&d" into
"r&amp;d". The manual escape is gone and the sanitized path goes in
raw.
Tests parse the generated report to the end with quick-xml and check:
control characters become U+FFFD and none survive, a "]]>" round-trips
through the split sections, a path with & < " > is escaped exactly once
and reads back unchanged, and legal text is left borrowed.
Fixtures: fixtures/xml-report-demo with a clone whose body holds real
escape bytes, a form feed and a "]]>" literal; the README shows the
xmllint and Python checks. Verified the published 5.2.0 fails xmllint
on it and the fixed binary passes.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
print_appendices gave console-full the history block it lacked; the
History section now names both console reporters and an integration
test checks console-full renders it.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
The one-line sparkline was too small to read a trend from. The console
block now draws an eight-row bar chart: a percentage axis on the left
spanning the series' own min and max (a series moving between 2.0% and
2.4% would be a flat line on a 0-based axis), one column per point built
from eighth-block characters so every point keeps at least a sliver, an
x-axis, and point numbers that match a new `#` column in the table.
Past 20 points the columns narrow to one character so 30 points still
fit an 80-column terminal. The ai reporter keeps the compact sparkline.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#1002 (the --history half; --ratchet is deliberately not
implemented, the report shows the headroom instead).
`jscpd src --history v5.0.0..HEAD` (or `--history-since 2026-01-01`)
lists the commits with `git log`, checks each one out into a temporary
detached worktree (the --baseline-from-ref machinery, now shared through
baseline_ref::map_scan_paths), scans it with the run's own configuration
and keeps the totals; the current run is the series' last point.
`--history-every N` keeps every Nth commit counted from the newest,
`--history-limit N` (default 30) samples a long series evenly keeping
both ends. Nothing runs unless one of the flags is given.
Output: the console reporter appends a sparkline of the duplication
percentage, a table (commit, date, files, lines, clones, dup lines,
dup%, change, subject) with the change column red when duplication rose
and green when it fell, the overall trend line, and, when --threshold is
set and the latest value is under it, the headroom with the value the
threshold could be tightened to. The json reporter adds a `history`
object; the ai reporter prints one compact line per point. Other
reporters are unchanged.
New cpd_core::history (model, sparkline, change/headroom helpers with
unit tests), cpd::history (git log parsing, sampling, worktree scans),
cpd_reporter::history_render. Config keys history / historySince /
historyEvery / historyLimit; action input `history`. Integration tests
build a four-commit repo with fixed dates and check the JSON series,
the console block, every/limit thinning and the error paths. 969
workspace tests pass, clippy clean.
Docs: History section and option rows in docs/rust.md, README bullet,
action input row, fixtures/history-demo/README.md with a script that
builds the demo repository and the expected output.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#1047.
Four situations used to write an empty report and exit 0, so a CI job
that checks only the exit code saw a green run:
- `--format` naming a format that is not supported (a warning since #964)
is now `Error: --format: 'x' is not a supported format`, exit 1. Names
introduced through --formats-exts or --formats-names stay accepted.
- A scan path that does not exist is `Error: path does not exist: <p>`,
exit 1, before anything runs.
- A reporter that fails to write its output still prints the existing
`Reporter '<name>' error: ...` line; the run now ends with
`ERROR: a reporter failed to write its output` and exit 1, after the
other reporters have run.
- A scan that analyzes no files (paths exist, but nothing matched the
--format/--ignore/--pattern filters or every file was below
--min-tokens) prints `Warning: jscpd analyzed no files` and still
exits 0, so an intentionally empty tree passes. The new
`--fail-on-empty` flag (config key `failOnEmpty`, action input
`fail-on-empty`) makes it exit 1. Reports are written before the check.
Docs: an "Exit codes" section in docs/rust.md, option rows in
docs/rust.md and docs/ci-and-hooks.md, a README feature bullet.
Fixtures: fixtures/fail-on-empty-demo with a `code/` clone pair, a
`nothing/` comment-only file, and a README quoting every command's
output and exit code. Tests: the #964 warning test becomes an error
test, plus integration tests for each new exit path and for the config
key; 531 tests pass, clippy clean.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Reviews compare jscpd with detectors that "use language-specific
normalization" as if jscpd matched raw text. It does not: every format
has its own comment and string rules, JS/TS/JSX/TSX go through the oxc
parser (with TypeScript erasure for cross-format pools), Vue/Svelte/
Astro/Markdown/Razor are split into embedded languages, and tokens are
classified so the Type-2 pass keeps keywords while renaming identifiers.
The README intro and a new first Features bullet say so, and docs/rust.md
gains a "How detection works" section walking through format detection,
tokenization, modes and ignores, normalization, Rabin-Karp matching and
the near-miss passes, plus what stays out of scope (Type-4). Every claim
is taken from cpd-tokenizer (generic.rs comment styles, javascript.rs,
ts_strip.rs, sfc.rs, markdown.rs, razor.rs, tokenizer.rs mode filter,
strips_annotations, is_common_keyword).
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#1037.
Adds rust/scripts/build-pypi-wheels.py, which repacks the eight
jscpd-<platform>.tar.gz release tarballs into platform wheels: the same
binary under <pkg>.data/scripts/ as both `jscpd` and `cpd`, one wheel
per target, stdlib only, no maturin and no recompilation. glibc wheels
take their manylinux floor from the GLIBC_x.y symbols in the binary
(manylinux_2_34 today), musl builds are static so they tag as
musllinux_1_2, macOS and Windows use the Rust minimums. Semver
pre-releases map to PEP 440 (5.3.0-beta.1 -> 5.3.0b1); other labels
fail rather than guess.
release.yml gains a rust-publish-pypi job after the npm publish: it
takes the binary artifacts from this run, refills any missing target
from the published npm platform package (resumed releases, or a version
already on npm), builds the wheels, pip-installs the matching one and
checks `jscpd --version` and `cpd --version`, then uploads with PyPI
trusted publishing (environment `pypi`, no token, skip-existing). The
detect job treats a version missing from PyPI as pending, so the wheels
for the current release publish on the next push to master without a
new release commit. The GitHub release lists the PyPI package and waits
on the upload the same way it waits on crates.io and npm.
Docs: PyPI in the README install table and packages table, rust/README
and docs/rust.md install sections, and a `language: python` pre-commit
option in docs/ci-and-hooks.md so the hook no longer needs Node.js.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
NO_COLOR asks for plain output and nothing else, and many people export
it globally while still working in an interactive terminal. Treating it
as a tips switch would hide the tips from them for good. The non-TTY
check at the print site already covers pipes, files and CI logs, so the
environment switches stay CI and JSCPD_NO_TIPS.
Help text, docs and the skill table updated; the options test asserts
that NO_COLOR does not enable no_tips.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Sorting the html tokens (previous commit) restores source order, but the
wrapper tags of a Vue file — <template>, </template>, <script ...>,
</script>, <style ...>, </style> — are identical in every Vue file and
sit at both ends of it. In source order they stretch every template clone
to the last </style>: on fixtures/vue the html clone went from
[1:1 - 10:9] to [1:1 - 262:10] and duplicated lines from 289 to 541,
counting the script and style bodies as duplicated html.
A file with a <template> block keeps all of its markup inside that block,
so the skeleton outside the blocks is only wrapper tags and is now left
out of the html stream. The clone becomes the template body itself:
file1.vue:html [2:3 - 10:9], 75 tokens. Svelte and Astro have no template
wrapper; their top-level markup is the skeleton and is kept, so their
output is unchanged. The sort stays, for custom blocks that map to html.
Fixture: fixtures/sfc-demo/template-range with a README showing the
template-only range; docs/rust.md links it.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Public-output items from the #1020 review, done while nothing is released:
- SARIF and Code Climate rule names now match the clone vocabulary:
renamed -> jscpd/renamed-code, similar -> jscpd/similar-code
(was jscpd/similar-code for renamed and jscpd/near-miss-code for similar).
The similar-code rule text covers both the gap and the ast method.
- Duplicated-line statistics and the per-file summary count only the
matched lines of a merged clone. CpdClone gains a non-serialized
unmatched_lines field holding each fragment's gap lines, so enabling
the merge no longer moves --threshold.
- A merge whose similarity would fall below 0.5 (more unmatched than
matched tokens, e.g. one very long inserted line) is refused and the
halves stay separate; MIN_GAP_SIMILARITY documents the floor.
- Precedence is decided and documented: merged renamed halves are
reported as similar.
merge_gapped_clones is restructured around merge_into(); the running
matched-token counter was redundant with last.token_count and is gone.
Fixtures: type3-demo/long-line (refused merge) and type3-demo/renamed-halves
(precedence); README numbers re-measured for the whole directory.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Print the tips/sponsor lines only when stdout is a terminal, so CI logs,
pipes and agent hooks no longer get them without --no-tips. Also honor
NO_COLOR and JSCPD_NO_TIPS next to the existing CI env check.
Fixes#1008
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A JavaScript/TypeScript file with any parse diagnostic (a redeclaration,
a recoverable syntax error) was handed to the word-split fallback
tokenizer. Its tokens then never matched an oxc-tokenized file, so every
cross-file clone involving it vanished while its self-clones survived —
the symptom reported in #1023, where a second copy of a function in one
file made all a↔b clones disappear.
Tokens come from the lexer, not the parser: only `panicked` (or an empty
token stream) now triggers the fallback. Function extraction for
--similarity uses the partial AST on recoverable errors the same way.
Closes#1023.
`CpdClone.similarity_method` (JSON "method": "gap" | "ast") records which
near-miss mechanism produced a `similar` clone. The two scores are not on
one scale (gap = matched tokens over the merged span, ast = structural
Jaccard of whole functions), so the console prints
`Clone found (javascript, similar (gap) ~0.91)` /
`similar (ast) ~0.75`, the ai reporter `[~0.91 gap]` / `[~0.75 ast]`,
JSON carries "method" and SARIF a `similarity_method` property. Exact and
renamed clones are unchanged.
Function extraction is now a language plug-in: `FunctionExtractor`
(grammar id, served formats, extract) with a static `EXTRACTORS` registry
consulted by `extract_functions` / `supports_functions` /
`supported_function_formats`. `OxcExtractor` is the first entry (JS/TS/
JSX/TSX). Every RawFunction/FunctionSig records its grammar and the scorer
refuses pairs across grammars, so a tree-sitter-backed extractor for
another language is a self-contained addition with no change to scoring,
CLI, MCP or reporters. Module docs describe the two steps; the MCP note
lists supported formats from the registry.
Off by default. `--similarity RATIO` (config `similarity`, Action input
`similarity`, MCP `check_duplication` argument) summarizes every function,
method and arrow function of a JavaScript/TypeScript/JSX/TSX file by the
bag of 4-grams over the pre-order sequence of its AST node types (oxc),
indexes the bags with MinHash + LSH banding, and reports pairs whose
weighted Jaccard index reaches RATIO as `similar` clones with a
`similarity` value. Names and literal values are not part of the summary:
a renamed copy scores 1.0, one inserted line ~0.9, two inserted statements
plus renames ~0.75.
Functions must clear --min-tokens/--min-lines, nested functions are never
paired with their parent, and pairs already covered by an exact, renamed
or merged clone are skipped. Values outside (0, 1] warn and disable the
option. Reporting reuses the Type-3 vocabulary (console `similar ~0.75`,
JSON kind/similarity, SARIF jscpd/near-miss-code).
New: cpd-core::similarity (FunctionSig, SimilarityIndex, bag_jaccard,
minhash), cpd-tokenizer::functions (oxc visitor extractor), MCP function
index built lazily per scan. fixtures/type3-demo/similar-functions with
README numbers, docs, unit + integration + MCP stdio tests. Also adds the
missing max_gap_lines/similarity fields to the --debug config output.
Off by default (0). When set to N, clones of the same file pair whose
fragments follow each other in both files with at most N unmatched lines
between them are merged into one clone of kind `similar`, with `tokens` =
matched tokens and `similarity` = matched tokens over the tokens of the
longer merged span. Adjacent exact windows that share a boundary token are
counted once; chains merge transitively; overlapping or reordered fragments
never merge. The pass runs after every other filter and returns its input
untouched at 0, so default output is unchanged.
Reporting: console `Clone found (fmt, similar ~0.91)`, ai `[~0.91]`, JSON
"kind"/"similarity" (rounded to three decimals), SARIF rule
jscpd/near-miss-code with a similarity property (declared only when
referenced), Code Climate check_name.
Config key maxGapLines (kebab alias), GitHub Action input max-gap-lines,
docs section, fixtures/type3-demo with a README of runnable commands, unit
tests for the merge pass and reporters, and a binary-level integration test.
Closes#998.
Three opt-in flags (config keys ignoreIdentifiers, ignoreLiterals,
ignoreAnnotations) normalize token classes before hashing so that blocks
which differ only in names, values or annotations are detected:
- identifiers hash as `$id`; keywords are kept (the JS/TS tokenizer
classifies them, other languages use a built-in common-keyword list)
- string literals hash as `$str`, numeric literals as `$num`
- `@Name`, `@a.b.Name` and `@Name(...)` runs are dropped in Java, Kotlin,
Scala, Groovy, Python, Dart, Swift, JavaScript and TypeScript
Each clone now carries `kind: exact | renamed`. DetectionToken gains a
raw_hash (equal to hash unless normalization rewrote the token); the
PreparedSource materializes raw hashes only when a token was normalized,
and the detector compares raw hashes over the matched run to classify a
pair. A dropped annotation salts the raw hash of the token before it, so a
pair differing only in annotations inside the matched run is `renamed`.
Reporters: console prints `Clone found (fmt, renamed)`, ai appends
`(renamed)`, json adds "kind", sarif uses rule jscpd/similar-code (declared
only when referenced), codeclimate the same check_name. Default runs are
unchanged apart from the additive "kind": "exact" JSON field.
Also: GitHub Action inputs, docs section, fixtures/type2-demo with a README
of runnable commands, unit tests in tokenizer/core/reporter, CLI flag
tests, and a binary-level integration test.
First release from the v5-only master. Bumps jscpd to 5.1.2 and every
sub-crate (cpd-core 0.1.11, cpd-tokenizer 0.1.13, cpd-finder 0.1.14,
cpd-reporter 0.1.12) so the new crates.io metadata from #988 and the
Windows drive-path fix in cpd-finder are republished. Adds the 5.1.2
changelog section (linux-arm64-musl package, cargo-binstall, Docker
image, jscpd --version naming, pre-commit hook args, repository split)
and refreshes the version examples in docs/.
Pre-release checks: cargo fmt, clippy -D warnings, cargo test --locked
(22 suites), cargo check --all-targets --target x86_64-pc-windows-msvc,
release-notes extraction dry-run against rust/CHANGELOG.md.
Everything belonging to the v4 TypeScript engine moves to the master-v4
branch: apps/, packages/, build-utils/, changesets, pnpm workspace, turbo,
the Node.js CI workflow and the ts-* release jobs. What remains is the
Rust workspace under rust/, the npm wrapper and platform packages, the
GitHub Action, Docker image, flake, skills, benchmark and fixtures/.
- rust.yml gains a smoke job that runs the release binary against
fixtures/ (json, sarif, html) and asserts clones and reporters
- release.yml keeps only the Rust jobs; pnpm removed from every workflow;
audit.yml keeps cargo audit + cargo deny; dependabot drops the root npm entry
- root package.json becomes a thin private package depending on the exact
jscpd@5 version so the pre-commit hook keeps installing the binary;
sync-version.mjs keeps it in step
- .pre-commit-hooks.yaml used v4-only flags that the v5 CLI rejects; fixed
- FORMATS.md is now generated from the Rust tokenizer (224 formats)
- README, docs, CONTRIBUTING, SECURITY, package and crate descriptions,
benchmark and skills rewritten for v5 on its own terms: no v5-vs-v4 speed
multipliers anywhere, a single "Looking for v4?" pointer to master-v4
Rust / crates.io
- report the invoked binary name in --version/--help (jscpd vs cpd)
- add repository, documentation, keywords, categories to all crates
- exclude tests/ from the jscpd crate package; add cargo-binstall metadata
- expand the crates.io README and render it on docs.rs
npm
- add funding field to jscpd, cpd and jscpd@4 packages
- add jscpd-linux-arm64-musl platform package and clearer unsupported-platform error
- enable provenance for the v4 changeset publish
Pipelines
- rust.yml: windows-latest in the matrix, drop dead schedule guard
- release.yml: cosign-sign release assets, floating v5 tag, LICENSE in tarballs,
invoke the new docker.yml (GHCR multi-arch image) after each release
- audit.yml: nightly cargo audit, cargo deny and pnpm audit
- Dockerfile: multi-stage image from release binaries (distroless), mcp target for Glama
Docs / community
- README: shields badges, single primary install per OS, GitHub Action snippet,
link to jscpd.dev, v5.1 features, 15 reporters, 224 formats, 8 platforms
- docs: @v5 action tag, Docker usage, baseline section, current crate versions
- YAML issue forms with engine selector, config.yml, CODEOWNERS
- remove stale .github/CONTRIBUTING.md stub, custom.md, TODO.md; untrack rust/jscpd-report.json
- examples: mark as v4 API, add a Rust cpd-finder example
Auto-discovery now also checks .config/jscpd.json (and
.config/.jscpd.json) between the root .jscpd.json and the package.json
jscpd key, per https://dot-config.github.io/. Root .jscpd.json keeps
precedence, so the new location is optional and never changes behavior
for existing setups. Paths inside the config still resolve relative to
the working directory, as with other auto-discovered sources.
Closes#979
Claude-Session: https://claude.ai/code/session_01VvWzWfEK6oV4HTe6EoGkff
Port the skipIsolated option from jscpd PR #628 to the Rust engine:
clones spanning two different folders of the same isolation group are
dropped, so team-owned monorepo packages are not compared with each
other while everything still matches against shared code.
- CLI: --skip-isolated / --skipIsolated "packages/a|packages/b,libs/a|libs/b"
- Config file: skipIsolated as nested arrays (jscpd shape)
- Detection: PathFilters bundles skip_local + skip_isolated path checks,
applied in both the primary and secondary clone passes
- MCP server: isolation groups apply to project scans; snippet checks
remain unfiltered
Improvements from reviewing the MCP tool surface against real agent
usage:
- Clone/match lists are sorted biggest-first (token_count desc) before
the 'limit' cap. Detection emits path order, so truncation previously
kept the alphabetically-first clones instead of the most important
ones. Applies to check_current_directory, check_duplication, and the
new get_file_clones (project clones sorted once per scan).
- check_duplication accepts file extensions as format aliases: "js",
"py", "tsx" resolve via get_format_by_extension, removing the most
likely first-use failure for agents that pass the extension. The
resolved format is echoed in the payload.
- check_duplication gains the same 'limit' argument (default 100) and
truncation contract as check_current_directory: 'count' is the
untruncated total, 'returned' the list length, 'note' on truncation.
- New tool get_file_clones(path, limit?) — clones involving one file,
for the dominant file-scoped agent workflow ("what in this file is
duplicated, where?"). Accepts scan-root-relative or absolute paths,
folds embedded-code-block fragment ids into their parent file, and
distinguishes "no clones" from "file was not part of the scan".
- display_path normalizes separators to '/' so tool output is uniform
across platforms.
7 new unit tests (alias resolution, limit contract, file filter,
unscanned-file note, missing path, biggest-first ordering across
differently-sized clone pairs); tools/list coverage updated to the
four-tool surface. Full workspace suite green.
Refs #891
check_current_directory previously answered with counts only, which
left an MCP client unable to see where the duplication actually is.
The server now keeps the clones from the last scan and returns them:
- duplications: [{format, fileA, startA, endA, fileB, startB, endB,
lines, tokens}] with paths relativized to the scan roots and
sub-format fragment ids (embedded code blocks) folded into their
parent file
- optional 'limit' argument (default 100) caps the list so heavily
duplicated projects cannot flood an LLM context; 'clones' always
carries the untruncated total, 'returned' the list length, and a
'note' marks truncation — no silent capping
- non-integer 'limit' is rejected with -32602
Unit test asserts list shape, cross-file attribution, and the
truncation contract; the end-to-end stdio session now exercises the
tool as well.
Refs #891
Closes the gap reported in #891: the HTTP-only jscpd-server cannot be
used by MCP clients that spawn and manage a stdio process themselves
(Claude Desktop/Code, Cursor, APM, ...). The Rust binary now serves the
Model Context Protocol directly:
cpd --mcp /path/to/project
Transport & protocol:
- newline-delimited JSON-RPC 2.0 on stdin/stdout; stdout carries
protocol messages only, logging goes to stderr; exits 0 on stdin EOF
- protocol revision 2025-06-18, accepting 2025-03-26 and 2024-11-05
clients via version negotiation; ping, tools/list, tools/call;
notifications ignored; parse errors reported (-32700) without
killing the session
Tools (mirroring the HTTP jscpd-server):
- check_duplication(code, format) — checks a snippet against the
scanned project; results are compact JSON in a text content block
with file/snippet line ranges; unknown formats and sub-threshold
snippets produce informative results
- get_statistics() — files, clones, full Statistics of the last scan
- check_current_directory() — re-scan the configured paths
Performance: the project is scanned once at startup and cached as
detection-ready token hashes (PreparedSource pools per format /
cross-format group), so snippet checks only tokenize the snippet and
run one pool detection — milliseconds even on large codebases. All
standard detection options (--min-tokens, --format, --cross-formats,
--ignore, ...) apply to the scan and to snippet checks.
Supporting refactor in cpd-finder: orchestrate::run() split into
build_thread_pool() + prepare_scan_in() (walk + tokenize, returning
PreparedScan) so the MCP server can keep prepared sources around;
run() behavior is unchanged and detect_prepared is untouched.
strip_types_formats made pub for snippet tokenization parity;
PreparedSource now derives Clone. hyperfine A/B vs a master-built
baseline shows no regression on the normal CLI path (39.0ms vs
41.2ms, within noise).
Tests: 12 unit tests for the protocol handler and tools, plus 2
end-to-end tests that spawn the real binary and drive a full stdio
session (initialize → tools/list → tools/call → EOF), asserting
stdout is protocol-only. Full workspace suite green.
Docs: docs/ai-ready.md MCP section restructured into stdio (Rust v5)
and Streamable HTTP (Node v4) transports with client config examples;
docs/rust.md options table; root README; rust/CHANGELOG.md 5.0.16.
The jscpd-server (Node) stdio subcommand asked for literally in #891
can follow separately; this provides the stdio transport via the v5
binary that ships with jscpd@5.
Refs #891
Adds a refactoring-hotspot summary computed from data the detector
already holds in memory, appended to the run output when requested:
- --summary enable the summary block (also `summary` in .jscpd.json)
- --summary-top <n> entries per list (default 10, `summaryTop`)
- --summary-by <m> ranking metric: tokens|lines|size|complexity (`summaryBy`)
Output: top-N files (tokens, lines, size, complexity estimate, dup%)
strictly ranked by the chosen metric — every row carries all metrics —
plus a top-N folder rollup (direct-parent aggregation, each file counted
once). totalFiles/totalFolders always report pre-truncation counts.
Complexity is a language-agnostic cyclomatic estimate: 1 + decision-point
tokens (if/elif/elsif/elseif/unless/for/foreach/while/until/case/cond/
when/catch/rescue/except/andalso/orelse/&&/||/and/or/?/??), matched
ASCII-case-insensitively so uppercase-keyword languages (SQL, PL/SQL,
Fortran, COBOL, BASIC) count too. Verified across all 157 fixture
languages.
Reporter integration:
- console / console-full: aligned tables after the normal output
- ai: compact one-line-per-entry block (LLM-token-efficient)
- json: additive `summary` key, absent when the flag is off — schema
unchanged for existing consumers
Performance: the only unconditional change is capturing the file's byte
size from the existing mmap; all summary work runs post-detection and
only when --summary is set. hyperfine A/B vs master baseline shows
identical times without the flag (39.8ms vs 40.2ms) and ~2ms summary
cost when enabled.
New: cpd-core/src/summary.rs (model + computation, 14 unit tests),
cpd-reporter/src/summary_render.rs. ReportContext gains an optional
summary reference (ReportContext::new is unchanged; with_summary added).
SourceFile gains a serde-defaulted `bytes` field. 6 new CLI integration
tests; full workspace suite green.
Follow-ups tracked in #934: HTML report section, TypeScript parity port.
Refs #934
apk