- Console tables (project/duplication-by-format/complexity/dead-code) and
the "largest:" summary line interpolated file paths and format names
raw: the Markdown escaping fix in the previous commit only covered
Markdown, not console. A newline in a path or a custom --formats-exts
name forged an extra table row with none of the other columns' data.
Sanitized at table()'s one choke point and at the "largest:" line.
- format_list() (the --health console/ai duplication line) had the same
gap for the formats it lists, independent of the console table fix.
- --complexity --fail-on-empty returned before the reporter loop ran, so
-r json never wrote a report; the clone-detection and dead-code paths
write reports first and apply the gate to the exit code afterward.
--complexity now does the same.
- dead_code::config()'s early Ok(None) for an unsupported --format
returned before parsing --dead-code-categories or clamping
--min-confidence, so --dashboard --format css --dead-code-categories
bogus silently succeeded despite the comment above it saying that must
stay a refusal. Moved both checks before the format check.
8 new tests.
Two independent reviews of the health/dashboard work landing since v5.2.1,
before cutting 5.3.0.
Scoring:
- Markup lines stayed in the duplication denominator while excluded from
the numerator, so adding one unique, un-duplicated HTML file could swing
a project from grade D to grade B. The denominator now excludes markup
too, and an all-markup project skips the dimension entirely instead of
scoring ~75 from the small-project prior alone.
- is_markup() matched file extensions (plantuml, puml, tpl) instead of the
tokenizer's actual format names (plant-uml, smarty), so those formats
were never really excluded.
- A health-input weight of 1e308 overflowed the weighted geometric mean to
NaN/100 with exit 0; validate() now rejects non-finite or unreasonably
large weights.
- External metric ids could duplicate each other or collide with a
built-in dimension id.
Output safety:
- Markdown output (dimension ids, format names, file paths, dead-code
finding names) was interpolated unescaped: a hostile path or
--health-input id could break a table or inject raw HTML/a heading
(GitHub renders HTML embedded in Markdown), or split console/ai output
via an embedded newline.
CLI:
- A config file with a known field of the wrong JSON type (e.g. "entry":
"src/index.js" where an array is expected) discarded the entire config,
--threshold included, instead of just that field — on any
auto-discovered .jscpd.json, no new flags required.
- --dashboard --format css hard-errored instead of dropping the dead-code
section.
- --dead-code/--complexity didn't conflict with --mcp.
- --complexity silently ignored --threshold, --exit-code,
--fail-on-empty, the baseline family, --history and --kind.
- --dead-code --min-confidence 200 silently reported nothing instead of
clamping to 100, unlike the standalone basta binary.
- --dashboard --max-lines let the dead-code section silently cover files
the other sections skipped (basta needs whole files for its import
graph).
- duplication.percentage/deadCode.percentage in dashboard JSON carried raw
float division noise instead of the two decimals the console already
shows.
26 new tests. Full default-path parity re-verified against the released
5.2.1 binary (13 reporters, option matrix) — byte-identical except timing.
One 0-100 score with an A-E grade, shown on top of --dashboard and alone
with --health.
- Dimensions are shares of the code lines, so size cancels out:
duplication (jscpd's percentage over code files), dead code, and the
share of code in complex files (complexity 50+). Prose and data files
are not counted.
- Each share is scored on a half-life curve (100 at zero, 50 at one
half-life), calibrated on 42 open-source projects so the median project
scores 75 per dimension; shares are mixed with a 2000-line prior so one
finding does not sink a small project.
- Sub-scores combine as a weighted geometric mean. A dimension that
cannot be measured is left out and named, and dead code weighs as much
as the share of the code it could read.
- --health-input FILE and the health config object add metrics from
other tools (a ready score, or a value with a half-life) and tune the
built-in half-lives and weights.
- Reporters: console badge, ai one-liner, json (jscpd-health.json,
jscpd-dashboard.json with every dashboard section) and badge
(jscpd-health-badge.svg). Exit gates apply as in the dashboard.
CpdClone::matched_lines moves to cpd-core so statistics and health count
duplicated lines the same way.
- --dashboard applies the exit gates of a clone run (--threshold,
--exit-code, --fail-on-empty) instead of always returning 0; the
baseline family warns and --fail-on-new-clones is refused, since no
clone report is written.
- A refused dead-code configuration (bad --dead-code-categories, a
--format no analyzer supports) now exits 1 as in --dead-code, rather
than printing a dashboard with an empty dead-code section.
- --workers is a budget for the whole dashboard run: the two concurrent
scans get half each, so neither the thread count nor the 64 MiB
tokenizer stacks double. --workers 1 runs them sequentially.
- --complexity scans with similarity disabled, so --similarity no longer
pays for function extraction it cannot use.
- --complexity honours a summaryBy from the config file, not only the
--summary-by flag (Options::summary_by_set).
- reStructuredText joins the prose formats at complexity 0.
--dashboard prints one screen: project size and largest formats,
duplication with clones per kind and the most duplicated files, total and
mean complexity with the most complex files, and dead code by category
with the largest findings (JavaScript, TypeScript, Python). The dead-code
scan runs beside clone detection; --summary-top sets the rows per list.
Data formats (JSON, YAML, TOML, CSV, ...) join markdown at complexity 0,
so a lock file's version ranges no longer top the complexity ranking.
Docs for --kind, --complexity and --dashboard; fixtures/dashboard-demo,
plus --kind and --complexity sections in the type3 and summary demos.
--kind exact,renamed,similar,gap,ast keeps only the clones of those kinds
(config key "kind"). The filter runs before statistics, so percentages
describe what is reported. An unknown kind is an error; a kind whose
detector is off (ast without --similarity, ...) is a warning, and
detection is never switched on implicitly.
--complexity walks and tokenizes like a normal run, skips clone
detection, and prints the --summary tables ranked by complexity without
the DUP% column (console, ai; json writes jscpd-complexity.json).
Markdown prose now has complexity 0: "if" in a README is a word.
Folding the symlink test assertions into a helper put it between
#[cfg(unix)] and symlink_layout, so the attribute moved to the helper and
std::os::unix::fs::symlink no longer compiled on Windows.
b72abf09 also searched the directory of each scanned path. fixtures/ has
a .jscpd.json with "silent": true, so `cd rust && jscpd ../fixtures -r ai`
started printing only the summary line instead of the clones that
`npx jscpd` lists. Config discovery is back to the working directory;
output now matches jscpd 5.2.1 across all reporters.
`jscpd rust/` reported 48 clones. This folds every one that was repetition
rather than a deliberate copy, leaving 34 across `rust/` and 14 in
`rust/crates`, all of the kinds the #1053 pass kept on purpose: fixture
corpora, `use` blocks, the TypeScript file and its JavaScript twin,
`impl Reporter` headers and the `ts_strip` visitor pair.
- basta's JSONC reader copied a string-skipping loop into both `strip_jsonc`
and `drop_trailing_commas`. It is now `strip_comments` followed by a comma
pass over `lang::outside_strings`, the quote-aware scanning the bundler
config readers already share.
- The JavaScript and Python analyzer tests each built their own module
index, import lookup, binding-reference lookup and member-read list. They
live once in `test_scan` as `index_of`, `import`, `binding_reference` and
`member_reads`.
- `test_scan::TempTree` writes a test's files and removes them when dropped,
so a failing assertion no longer leaves its directory behind; the script
and `extends` tests that repeated the setup use it.
- Four tsconfig `paths` tests with one shape are one table-driven test.
- The CLI integration tests share `duplicate_shapes`, `history_stdout` and
`assert_only_the_real_file_was_scanned`.
- `scripts/publish-npm.sh` repeated a 43-line "publish the platform package,
then the main packages under --force-main" tail in two branches, and the
"publish unless already live, or show the dry run" block five times. They
are `publish_platform_target` and `publish_package`. Run against the old
script with the build, the registry check and `npm` stubbed, all eleven
combinations of --target, --all, --force-main, --dry-run, --provenance, a
failed build and missing platforms print the same lines and exit the same.
Behaviour is unchanged: the fixtures corpus reports in every mode, the
summary, the console output and the dead-code demo are byte-identical.
Two failures on the branch, neither on macOS, which is why they surfaced
only in CI.
`cargo deny` rejected `unicode_names2`, which `ruff_python_parser` pulls in
and which carries `Unicode-DFS-2016` alongside its MIT/Apache dual licence.
That licence covers the character tables and is the older wording of
`Unicode-3.0`, already on the allow list; added beside it.
The two new config-discovery tests asserted on `../project/.jscpd.json`,
and Windows prints the separator it joined with, so both failed there.
They now normalise backslashes before comparing — the assertion is about
which file was read, not about how the path is spelled.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
`cd rust && jscpd --basta ..` silently ignored the repository's own
`.jscpd.json`: config discovery only ever tried `.jscpd.json`,
`.config/jscpd.json` and `package.json` relative to the working
directory. The configuration is a property of the code being scanned,
not of where the shell happens to be, so a scan aimed at a project read
none of its ignores, formats or thresholds.
The cwd is still searched first and still wins, so nothing about an
existing invocation changes; each scanned path's directory is searched
after it, and a path that is also the cwd is not searched twice. A file
argument contributes the directory holding it. `ConfigSource` now
carries the path it was read from, so the "Using config from" line names
the file that was actually used rather than assuming it was the local one.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Adds a second question about the same tree: not "what is duplicated" but
"what does nothing reach". A new `basta` crate builds the import graph from
the project's entry points and walks it, reporting unused files, unused
exports, unused module-private declarations, unused imports and — opt-in —
unused class members, across JavaScript, TypeScript, JSX, TSX and Python.
It ships two ways: `jscpd --dead-code` inside the existing binary, and a
standalone `basta` command. Both reuse the walker, the filters, all fifteen
reporters and the `--threshold` / `--exit-code` gates, so nothing new has to
be learned to run it.
Because it is a graph traversal rather than a reference count, dead code
cascades: a helper whose only caller is itself dead is reported too. Every
finding carries a confidence score and, below 100, the reasons it might be
wrong — `eval`, an unrecognised decorator, a wildcard re-export, a name seen
only in a string, a file in the scan that did not parse. `--min-confidence`
sets the floor at 60.
Entry points come from `package.json`, `pyproject.toml`, framework and
file-name conventions, and anything a shell script, CI workflow, Makefile or
Dockerfile names by path. CommonJS is read alongside ESM. A project that
renames its own import paths is read on its own terms: `paths` and `baseUrl`
from `tsconfig.json` and `jsconfig.json` are resolved, through a relative
`extends` chain and per package in a monorepo. A file named only by a path
string — a worker spawned through `new URL`, a build entry, a vitest setup
file — counts as used.
Languages plug in through one trait, `basta::lang::Analyzer`, which owns
parsing, specifier resolution, entry-point conventions, manifests, alias
configs and path traits. Nothing outside `lang/` is language-specific;
`docs/basta-extending.md` walks through adding one.
Validated against knip, fallow, vulture, skylos, fossil-mcp and repowise on
trending repositories. On a corpus of ten of them — 11k files, 3.3M lines —
every one of basta's 1,806 findings was checked against an import graph
rebuilt independently of it: no unused-file, unused-import or unused-symbol
claim survived as wrong.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
With `--follow-symlinks` the source id was the canonical path, so a file
reached through a link showed up as an absolute path outside the scan
root, `--ignore` (matched on the walked path) and the report disagreed
about its name, and a file reachable through two paths was two sources
(a file symlink next to its target even became a clone of itself).
The walked path, anchored at the canonical scan root, is now the id:
what reports show, what `--ignore` matches, what `--absolute` prints.
The canonical path travels alongside (`DiscoveredFile::real_path`,
`PreparedSource::real_path`) for reading the file and for the
`--skip-local` / `--skip-isolated` prefix checks. When links are
followed, files whose canonical path was already seen are dropped; the
entry that is the real file wins, else the first walked path.
Docs: `--follow-symlinks` row in the option table, the v4 default in the
migration table and README. Fixture `fixtures/follow-symlinks-demo`.
Closes#1059
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
print_appendices gave console-full the history block it lacked; the
History section now names both console reporters and an integration
test checks console-full renders it.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Phase 3 of the DRY pass: report_result / report_to_file_timed fixtures
for the stdout and time-printing reporters, rule_ids and
first_result_properties in the sarif tests, cross_formats_with in the
cli tests, run_isolating in the skip-isolated finder tests, and a
tests/common module for the cpd crate holding the invoice/credit-note
snippets the CLI and MCP tests both used.
rust/crates: 12 clones / 0.4% left, all deliberate (fixture corpora,
use blocks, a TS file and its JS twin, impl headers, AST visitors).
971 workspace tests pass, clippy clean.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Phase 2 of the DRY pass. cpd/tests/integration.rs was 23% duplicated:
every reporter test rebuilt the same temp root, wrote the same greet
pair, ran cpd with the same assertion block and parsed the report by
hand. Now: GREET_DUP / greet_root / write_greet_pair, run_ok_in,
read_json, scan_json (json+silent scans returning the parsed report and
stderr), run_scratch (exit code + stderr for a scratch dir), config_dir
/ stderr_in_dir for config discovery, commit_dates for --history and
baseline_root_with_pair for the baseline suites. mcp_stdio.rs gains
spawn_mcp / drive and INITIALIZE / INITIALIZED constants; cpd-finder's
five integration tests share tests/common (temp_dir, duplicate_js,
fixtures); mcp.rs unit tests share scratch_dir.
rust/crates: 69 clones / 2.4% before the pass, 21 / 0.7% after.
971 workspace tests pass, clippy clean.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#1002 (the --history half; --ratchet is deliberately not
implemented, the report shows the headroom instead).
`jscpd src --history v5.0.0..HEAD` (or `--history-since 2026-01-01`)
lists the commits with `git log`, checks each one out into a temporary
detached worktree (the --baseline-from-ref machinery, now shared through
baseline_ref::map_scan_paths), scans it with the run's own configuration
and keeps the totals; the current run is the series' last point.
`--history-every N` keeps every Nth commit counted from the newest,
`--history-limit N` (default 30) samples a long series evenly keeping
both ends. Nothing runs unless one of the flags is given.
Output: the console reporter appends a sparkline of the duplication
percentage, a table (commit, date, files, lines, clones, dup lines,
dup%, change, subject) with the change column red when duplication rose
and green when it fell, the overall trend line, and, when --threshold is
set and the latest value is under it, the headroom with the value the
threshold could be tightened to. The json reporter adds a `history`
object; the ai reporter prints one compact line per point. Other
reporters are unchanged.
New cpd_core::history (model, sparkline, change/headroom helpers with
unit tests), cpd::history (git log parsing, sampling, worktree scans),
cpd_reporter::history_render. Config keys history / historySince /
historyEvery / historyLimit; action input `history`. Integration tests
build a four-commit repo with fixed dates and check the JSON series,
the console block, every/limit thinning and the error paths. 969
workspace tests pass, clippy clean.
Docs: History section and option rows in docs/rust.md, README bullet,
action input row, fixtures/history-demo/README.md with a script that
builds the demo repository and the expected output.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#1047.
Four situations used to write an empty report and exit 0, so a CI job
that checks only the exit code saw a green run:
- `--format` naming a format that is not supported (a warning since #964)
is now `Error: --format: 'x' is not a supported format`, exit 1. Names
introduced through --formats-exts or --formats-names stay accepted.
- A scan path that does not exist is `Error: path does not exist: <p>`,
exit 1, before anything runs.
- A reporter that fails to write its output still prints the existing
`Reporter '<name>' error: ...` line; the run now ends with
`ERROR: a reporter failed to write its output` and exit 1, after the
other reporters have run.
- A scan that analyzes no files (paths exist, but nothing matched the
--format/--ignore/--pattern filters or every file was below
--min-tokens) prints `Warning: jscpd analyzed no files` and still
exits 0, so an intentionally empty tree passes. The new
`--fail-on-empty` flag (config key `failOnEmpty`, action input
`fail-on-empty`) makes it exit 1. Reports are written before the check.
Docs: an "Exit codes" section in docs/rust.md, option rows in
docs/rust.md and docs/ci-and-hooks.md, a README feature bullet.
Fixtures: fixtures/fail-on-empty-demo with a `code/` clone pair, a
`nothing/` comment-only file, and a README quoting every command's
output and exit code. Tests: the #964 warning test becomes an error
test, plus integration tests for each new exit path and for the config
key; 531 tests pass, clippy clean.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
NO_COLOR asks for plain output and nothing else, and many people export
it globally while still working in an interactive terminal. Treating it
as a tips switch would hide the tips from them for good. The non-TTY
check at the print site already covers pipes, files and CI logs, so the
environment switches stay CI and JSCPD_NO_TIPS.
Help text, docs and the skill table updated; the options test asserts
that NO_COLOR does not enable no_tips.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Print the tips/sponsor lines only when stdout is a terminal, so CI logs,
pipes and agent hooks no longer get them without --no-tips. Also honor
NO_COLOR and JSCPD_NO_TIPS next to the existing CI env check.
Fixes#1008
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A JavaScript/TypeScript file with any parse diagnostic (a redeclaration,
a recoverable syntax error) was handed to the word-split fallback
tokenizer. Its tokens then never matched an oxc-tokenized file, so every
cross-file clone involving it vanished while its self-clones survived —
the symptom reported in #1023, where a second copy of a function in one
file made all a↔b clones disappear.
Tokens come from the lexer, not the parser: only `panicked` (or an empty
token stream) now triggers the fallback. Function extraction for
--similarity uses the partial AST on recoverable errors the same way.
Closes#1023.
Off by default. `--similarity RATIO` (config `similarity`, Action input
`similarity`, MCP `check_duplication` argument) summarizes every function,
method and arrow function of a JavaScript/TypeScript/JSX/TSX file by the
bag of 4-grams over the pre-order sequence of its AST node types (oxc),
indexes the bags with MinHash + LSH banding, and reports pairs whose
weighted Jaccard index reaches RATIO as `similar` clones with a
`similarity` value. Names and literal values are not part of the summary:
a renamed copy scores 1.0, one inserted line ~0.9, two inserted statements
plus renames ~0.75.
Functions must clear --min-tokens/--min-lines, nested functions are never
paired with their parent, and pairs already covered by an exact, renamed
or merged clone are skipped. Values outside (0, 1] warn and disable the
option. Reporting reuses the Type-3 vocabulary (console `similar ~0.75`,
JSON kind/similarity, SARIF jscpd/near-miss-code).
New: cpd-core::similarity (FunctionSig, SimilarityIndex, bag_jaccard,
minhash), cpd-tokenizer::functions (oxc visitor extractor), MCP function
index built lazily per scan. fixtures/type3-demo/similar-functions with
README numbers, docs, unit + integration + MCP stdio tests. Also adds the
missing max_gap_lines/similarity fields to the --debug config output.
Off by default (0). When set to N, clones of the same file pair whose
fragments follow each other in both files with at most N unmatched lines
between them are merged into one clone of kind `similar`, with `tokens` =
matched tokens and `similarity` = matched tokens over the tokens of the
longer merged span. Adjacent exact windows that share a boundary token are
counted once; chains merge transitively; overlapping or reordered fragments
never merge. The pass runs after every other filter and returns its input
untouched at 0, so default output is unchanged.
Reporting: console `Clone found (fmt, similar ~0.91)`, ai `[~0.91]`, JSON
"kind"/"similarity" (rounded to three decimals), SARIF rule
jscpd/near-miss-code with a similarity property (declared only when
referenced), Code Climate check_name.
Config key maxGapLines (kebab alias), GitHub Action input max-gap-lines,
docs section, fixtures/type3-demo with a README of runnable commands, unit
tests for the merge pass and reporters, and a binary-level integration test.
Closes#998.
Three opt-in flags (config keys ignoreIdentifiers, ignoreLiterals,
ignoreAnnotations) normalize token classes before hashing so that blocks
which differ only in names, values or annotations are detected:
- identifiers hash as `$id`; keywords are kept (the JS/TS tokenizer
classifies them, other languages use a built-in common-keyword list)
- string literals hash as `$str`, numeric literals as `$num`
- `@Name`, `@a.b.Name` and `@Name(...)` runs are dropped in Java, Kotlin,
Scala, Groovy, Python, Dart, Swift, JavaScript and TypeScript
Each clone now carries `kind: exact | renamed`. DetectionToken gains a
raw_hash (equal to hash unless normalization rewrote the token); the
PreparedSource materializes raw hashes only when a token was normalized,
and the detector compares raw hashes over the matched run to classify a
pair. A dropped annotation salts the raw hash of the token before it, so a
pair differing only in annotations inside the matched run is `renamed`.
Reporters: console prints `Clone found (fmt, renamed)`, ai appends
`(renamed)`, json adds "kind", sarif uses rule jscpd/similar-code (declared
only when referenced), codeclimate the same check_name. Default runs are
unchanged apart from the additive "kind": "exact" JSON field.
Also: GitHub Action inputs, docs section, fixtures/type2-demo with a README
of runnable commands, unit tests in tokenizer/core/reporter, CLI flag
tests, and a binary-level integration test.
`resolve_config_paths` treated `ignorePattern` entries without `*` or `?`
as relative paths and joined them onto the config directory, a leftover
from when `ignore` and `ignorePattern` were conflated. A plain regex such
as `"Copyright 2026 Example Authors"` in `.jscpd.json` therefore matched
nothing, while the same string via `--ignore-pattern` worked. Config
entries are now passed through untouched.
Regexes that fail to compile were also dropped silently by the tokenizer;
the CLI now prints a `Warning: --ignore-pattern: invalid regex ...` line
for each one so a typo no longer disables a pattern without feedback.
Adds integration tests for both behaviors.
Rust / crates.io
- report the invoked binary name in --version/--help (jscpd vs cpd)
- add repository, documentation, keywords, categories to all crates
- exclude tests/ from the jscpd crate package; add cargo-binstall metadata
- expand the crates.io README and render it on docs.rs
npm
- add funding field to jscpd, cpd and jscpd@4 packages
- add jscpd-linux-arm64-musl platform package and clearer unsupported-platform error
- enable provenance for the v4 changeset publish
Pipelines
- rust.yml: windows-latest in the matrix, drop dead schedule guard
- release.yml: cosign-sign release assets, floating v5 tag, LICENSE in tarballs,
invoke the new docker.yml (GHCR multi-arch image) after each release
- audit.yml: nightly cargo audit, cargo deny and pnpm audit
- Dockerfile: multi-stage image from release binaries (distroless), mcp target for Glama
Docs / community
- README: shields badges, single primary install per OS, GitHub Action snippet,
link to jscpd.dev, v5.1 features, 15 reporters, 224 formats, 8 platforms
- docs: @v5 action tag, Docker usage, baseline section, current crate versions
- YAML issue forms with engine selector, config.yml, CODEOWNERS
- remove stale .github/CONTRIBUTING.md stub, custom.md, TODO.md; untrack rust/jscpd-report.json
- examples: mark as v4 API, add a Rust cpd-finder example
Auto-discovery now also checks .config/jscpd.json (and
.config/.jscpd.json) between the root .jscpd.json and the package.json
jscpd key, per https://dot-config.github.io/. Root .jscpd.json keeps
precedence, so the new location is optional and never changes behavior
for existing setups. Paths inside the config still resolve relative to
the working directory, as with other auto-discovered sources.
Closes#979
Claude-Session: https://claude.ai/code/session_01VvWzWfEK6oV4HTe6EoGkff
An unrecognized --format token (e.g. 'cs' instead of 'csharp', or a
typo) silently matched 0 files and reported a clean scan with exit 0,
making a misconfigured CI gate indistinguishable from a clean codebase.
Print a stderr warning naming the unsupported value, mirroring the
existing --cross-formats validation; custom formats declared via
--formats-exts remain legal.
Fixes#964
Claude-Session: https://claude.ai/code/session_01VvWzWfEK6oV4HTe6EoGkff
New `codeclimate` reporter (alias: `gitlab`) writes
gl-code-quality-report.json in the CodeClimate issue format, restricted
to the subset GitLab defines as its Code Quality report format, ready to
be declared as an `artifacts:reports:codequality` artifact. Unlike the
SARIF reporter, which GitLab ingests as security vulnerability findings,
this surfaces duplicates as code quality issues in merge requests.
Each clone yields one issue per fragment (each describing the other
location, plus the CodeClimate `other_locations` field), with a
deterministic fingerprint derived from the clone's content hash — the
same snippet_pair_hash identity used by SARIF partialFingerprints and
the baseline file, now shared via `shared::clone_pair_hash`. Severity is
"minor", escalating to "major" for clones absent from a configured
baseline or when the run exceeds --threshold.
Closes#958
Claude-Session: https://claude.ai/code/session_01UkyKpExH2hcHawXqGp2geA
Implements phase 2 of #944: an ephemeral baseline built from a git
ref's tree, for PR gates without a committed baseline file.
cpd --baseline-from-ref origin/main --fail-on-new-clones .
- checks the base ref out into a temporary detached git worktree
(removed afterwards; shells out to git like blame does), scans it
with the same detection configuration, and compares the current run
against that in-memory fingerprint set via the phase 1 machinery
(new baseline::apply_in_memory)
- scan paths are remapped into the base tree; paths absent there are
simply new code
- a missing ref fails with a clear shallow-checkout hint (git fetch /
fetch-depth: 0); conflicts with --baseline and --update-baseline,
and --fail-on-new-clones now accepts either baseline source
- config key baselineFromRef; GitHub Action input baseline-from-ref
Refs #944
Implements phase 1 of #944: a committed baseline of clone fingerprints
so CI tolerates recorded duplication and fails only on regressions.
- --baseline <file>: content-hash fingerprints (same value as the SARIF
partialFingerprints jscpdCloneHash/v1) with per-fingerprint
multiplicity counts; clones absent from the file are marked new
- --update-baseline: rewrite the file from the current run (creating it
if missing) and print added/removed fingerprint counts
- --fail-on-new-clones[=N]: exit 1 when more than N (default 0) new
clones are found, independent of --threshold
- config keys baseline / failOnNewClones, with the baseline path
resolved relative to the config file
- reporters: [NEW] markers and "(N new)" found count in console and
console-full, per-clone isNew plus newClones/newDuplicatedLines
statistics in json, level "error" for new clones in sarif, and
jscpd_new_clones / jscpd_new_duplicated_lines gauges in openmetrics
- baseline file is versioned and sorted one fingerprint per line for
reviewable diffs and trivial merges
- GitHub Action: baseline, update-baseline, fail-on-new-clones inputs
Refs #944
The 1.97 clippy cleanup (35d287e) dropped 'mut' from the path binding
because set_extension() is cfg-gated to Windows and clippy on
Linux/macOS flagged unused_mut — breaking the Windows-only compile
path that only the release workflow builds. Pick the .exe suffix via
cfg! inside join() instead, so both branches type-check on every
platform and no conditional mutability remains.
Improvements from reviewing the MCP tool surface against real agent
usage:
- Clone/match lists are sorted biggest-first (token_count desc) before
the 'limit' cap. Detection emits path order, so truncation previously
kept the alphabetically-first clones instead of the most important
ones. Applies to check_current_directory, check_duplication, and the
new get_file_clones (project clones sorted once per scan).
- check_duplication accepts file extensions as format aliases: "js",
"py", "tsx" resolve via get_format_by_extension, removing the most
likely first-use failure for agents that pass the extension. The
resolved format is echoed in the payload.
- check_duplication gains the same 'limit' argument (default 100) and
truncation contract as check_current_directory: 'count' is the
untruncated total, 'returned' the list length, 'note' on truncation.
- New tool get_file_clones(path, limit?) — clones involving one file,
for the dominant file-scoped agent workflow ("what in this file is
duplicated, where?"). Accepts scan-root-relative or absolute paths,
folds embedded-code-block fragment ids into their parent file, and
distinguishes "no clones" from "file was not part of the scan".
- display_path normalizes separators to '/' so tool output is uniform
across platforms.
7 new unit tests (alias resolution, limit contract, file filter,
unscanned-file note, missing path, biggest-first ordering across
differently-sized clone pairs); tools/list coverage updated to the
four-tool surface. Full workspace suite green.
Refs #891
check_current_directory previously answered with counts only, which
left an MCP client unable to see where the duplication actually is.
The server now keeps the clones from the last scan and returns them:
- duplications: [{format, fileA, startA, endA, fileB, startB, endB,
lines, tokens}] with paths relativized to the scan roots and
sub-format fragment ids (embedded code blocks) folded into their
parent file
- optional 'limit' argument (default 100) caps the list so heavily
duplicated projects cannot flood an LLM context; 'clones' always
carries the untruncated total, 'returned' the list length, and a
'note' marks truncation — no silent capping
- non-integer 'limit' is rejected with -32602
Unit test asserts list shape, cross-file attribution, and the
truncation contract; the end-to-end stdio session now exercises the
tool as well.
Refs #891
Closes the gap reported in #891: the HTTP-only jscpd-server cannot be
used by MCP clients that spawn and manage a stdio process themselves
(Claude Desktop/Code, Cursor, APM, ...). The Rust binary now serves the
Model Context Protocol directly:
cpd --mcp /path/to/project
Transport & protocol:
- newline-delimited JSON-RPC 2.0 on stdin/stdout; stdout carries
protocol messages only, logging goes to stderr; exits 0 on stdin EOF
- protocol revision 2025-06-18, accepting 2025-03-26 and 2024-11-05
clients via version negotiation; ping, tools/list, tools/call;
notifications ignored; parse errors reported (-32700) without
killing the session
Tools (mirroring the HTTP jscpd-server):
- check_duplication(code, format) — checks a snippet against the
scanned project; results are compact JSON in a text content block
with file/snippet line ranges; unknown formats and sub-threshold
snippets produce informative results
- get_statistics() — files, clones, full Statistics of the last scan
- check_current_directory() — re-scan the configured paths
Performance: the project is scanned once at startup and cached as
detection-ready token hashes (PreparedSource pools per format /
cross-format group), so snippet checks only tokenize the snippet and
run one pool detection — milliseconds even on large codebases. All
standard detection options (--min-tokens, --format, --cross-formats,
--ignore, ...) apply to the scan and to snippet checks.
Supporting refactor in cpd-finder: orchestrate::run() split into
build_thread_pool() + prepare_scan_in() (walk + tokenize, returning
PreparedScan) so the MCP server can keep prepared sources around;
run() behavior is unchanged and detect_prepared is untouched.
strip_types_formats made pub for snippet tokenization parity;
PreparedSource now derives Clone. hyperfine A/B vs a master-built
baseline shows no regression on the normal CLI path (39.0ms vs
41.2ms, within noise).
Tests: 12 unit tests for the protocol handler and tools, plus 2
end-to-end tests that spawn the real binary and drive a full stdio
session (initialize → tools/list → tools/call → EOF), asserting
stdout is protocol-only. Full workspace suite green.
Docs: docs/ai-ready.md MCP section restructured into stdio (Rust v5)
and Streamable HTTP (Node v4) transports with client config examples;
docs/rust.md options table; root README; rust/CHANGELOG.md 5.0.16.
The jscpd-server (Node) stdio subcommand asked for literally in #891
can follow separately; this provides the stdio transport via the v5
binary that ships with jscpd@5.
Refs #891
Adds a refactoring-hotspot summary computed from data the detector
already holds in memory, appended to the run output when requested:
- --summary enable the summary block (also `summary` in .jscpd.json)
- --summary-top <n> entries per list (default 10, `summaryTop`)
- --summary-by <m> ranking metric: tokens|lines|size|complexity (`summaryBy`)
Output: top-N files (tokens, lines, size, complexity estimate, dup%)
strictly ranked by the chosen metric — every row carries all metrics —
plus a top-N folder rollup (direct-parent aggregation, each file counted
once). totalFiles/totalFolders always report pre-truncation counts.
Complexity is a language-agnostic cyclomatic estimate: 1 + decision-point
tokens (if/elif/elsif/elseif/unless/for/foreach/while/until/case/cond/
when/catch/rescue/except/andalso/orelse/&&/||/and/or/?/??), matched
ASCII-case-insensitively so uppercase-keyword languages (SQL, PL/SQL,
Fortran, COBOL, BASIC) count too. Verified across all 157 fixture
languages.
Reporter integration:
- console / console-full: aligned tables after the normal output
- ai: compact one-line-per-entry block (LLM-token-efficient)
- json: additive `summary` key, absent when the flag is off — schema
unchanged for existing consumers
Performance: the only unconditional change is capturing the file's byte
size from the existing mmap; all summary work runs post-detection and
only when --summary is set. hyperfine A/B vs master baseline shows
identical times without the flag (39.8ms vs 40.2ms) and ~2ms summary
cost when enabled.
New: cpd-core/src/summary.rs (model + computation, 14 unit tests),
cpd-reporter/src/summary_render.rs. ReportContext gains an optional
summary reference (ReportContext::new is unchanged; with_summary added).
SourceFile gains a serde-defaulted `bytes` field. 6 new CLI integration
tests; full workspace suite green.
Follow-ups tracked in #934: HTML report section, TypeScript parity port.
Refs #934
Each e2e test process ran 'cargo build --bin cpd' via maybe_bin().
Under nextest's parallel per-process execution, cargo refreshes
target/debug/cpd by removing and re-creating a hardlink, so a test
spawning the binary during another test's rebuild window failed with
NotFound (flaked on macOS CI in the Rust workflow).
Use the compile-time CARGO_BIN_EXE_cpd instead — cargo guarantees the
bin target is built before integration tests run, with the correct
target dir and suffix. Also cuts the suite from ~6s to ~0.8s.
Refs #908
Extends the size-based severity from 294de6b: when the existing
--threshold option is set and the run's total duplication percentage
exceeds it, every SARIF result is emitted at level "error" instead of
"warning" — the report now reflects the same failure the
ThresholdReporter exits non-zero for.
Uses the identical strictly-greater comparison as the ThresholdReporter
(equal to the threshold does not fail the build, so it does not
escalate severity either). Composes with --sarif-error-tokens: either
condition escalates a result. Default output is unchanged when
--threshold is unset.
Fixes#908
All SARIF results were emitted at level "warning" regardless of clone
size, so large copy-pastes and trivial duplicates looked identical to
SARIF consumers like GitHub code scanning.
Add an opt-in cutoff: clones with token_count >= --sarif-error-tokens
are reported at level "error"; smaller clones (and all clones when the
flag is unset) stay "warning", so default output is unchanged. The
option is also available as sarifErrorTokens / sarif-error-tokens in
.jscpd.json, and the rule's defaultConfiguration remains "warning"
(per-result level overrides it).
No default cutoff is imposed — the issue explicitly asked not to invent
an arbitrary one, so severity escalation is purely opt-in.
Fixes#915
The SARIF reporter hardcoded tool.driver.version as "5.0.3", and the
HTML reporter stamped env!("CARGO_PKG_VERSION") of cpd-reporter
(0.1.7) — neither matched what 'cpd --version' prints.
Add ReporterOptions.tool_version, set by the cpd binary from its own
CARGO_PKG_VERSION at build time, and use it in the SARIF and HTML
reporters. The field defaults to cpd-reporter's crate version for
library consumers that don't set it.
Covered by a unit test (custom version flows through to driver.version)
and an e2e assertion that the generated SARIF matches the cpd crate
version.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Blame enrichment runs after path normalization, so it received
scan-root-relative source_ids and canonicalized them against the CWD —
failing silently when scan root != CWD (same failure shape as #872, for
blame instead of snippets).
- Move clean_source_id and resolve_fragment_path to cpd-core/paths so
cpd-finder and cpd-reporter share one implementation; this also removes
blame.rs's stale private clean_source_id copy that still had the
Windows drive-letter bug
- blame enrichment now reads files via source_root + source_id, keyed by
cleaned source_id for reporter lookups
- find_git_root now canonicalizes its start path: walking up a relative
path terminated at the empty path, mis-reporting a repo rooted at the
CWD as "" and breaking repo_root canonicalization in blame_file (this
affected relative-path invocations even before scan-root relativization)
- Add integration test covering --blame with a subdirectory scan root
- Use parent directory as scan root when a scan path is a file, so
source_id remains the filename (not empty)
- Fix clean_source_id to preserve Windows drive-letter colons (C:\)
while still stripping format suffixes (doc.md:javascript)
- Clean format suffixes from SARIF artifact URIs
- Deduplicate SARIF artifacts by (source_root, clean_source_id) so
identical relative paths under different scan roots get distinct
artifact entries with matching uriBaseId
- Add integration tests for single-file scan and multi-root SARIF dedup
- Add unit tests for clean_source_id and resolve_fragment_path
Store the scan root on Fragment.source_root so source_id can be
scan-root-relative (for display and SARIF output) while reporters
reconstruct the absolute path via root + source_id for file I/O.
This resolves the tension between #828 (scan-root-relative paths for
SARIF) and #892 (CWD-relative paths for snippet reads). Both needs are
now satisfied: SARIF gets clean relative URIs with originalUriBaseIds
referencing each scan root, and reporters join source_root + source_id
to read files regardless of CWD.
File reporters (html, json, xml, console-full) re-read source files from
disk to render code snippets. Previously, source IDs were relativized
against the scan root; when the scan root differed from the CWD (e.g.
`cpd frontend/`), the resulting paths couldn't be resolved and snippets
were silently empty.
Replace `relativize_to_scan_root` with `relativize_to_cwd`: paths under
the CWD become CWD-relative (resolvable), paths outside stay absolute
(still readable). This fixes#872.
Also add reporter tests that use real files on disk and assert snippet
content appears in the output, closing the gap that let this bug ship
undetected.
Signed-off-by: darronz <136805+darronz@users.noreply.github.com>
Detect duplicates across related formats via format equivalence groups
sharing one comparison pool, e.g. --cross-formats "javascript,typescript"
or the js-ts preset (javascript,jsx,typescript,tsx).
When a group mixes TypeScript with JavaScript, TS files are compared with
erasable type-only syntax stripped from the detection token stream (type
annotations, generics, interface/type declarations, as/satisfies, ?/!
markers, access modifiers, implements clauses, type-only imports/exports,
overload signatures, declare statements) so that
`function f(a: number): void` matches `function f(a)`. Stripping is
AST-driven (oxc_ast_visit over the already-parsed program) and only drops
tokens, so reported clone positions always reference the original source.
- cpd-tokenizer: new ts_strip module, tokenize_js_stripped,
TokenizeOptions.strip_types_formats (empty by default)
- cpd-finder: RunConfig.cross_formats, build_pools() pool grouping,
strip-set computation for TS-family formats grouped with JS
- cpd: --cross-formats flag, crossFormats config field (string,
array-of-strings, or array-of-arrays), js-ts preset, overlap merging,
unknown-format warnings, --debug output
- docs: rust.md option + Cross-Format Groups section, CHANGELOG entry
- tests: tokenizer strip tests, pool unit tests, cross-pool core test,
cpd-finder integration tests, CLI parse/config tests, e2e binary tests
with fixtures; default behavior unchanged (golden parity intact)
Non-erasable TS syntax with runtime semantics (enum, non-declare
namespace, parameter properties, import =, export =) is left intact and
will not cross-match. Cross-format clones are attributed to one member
format in per-format statistics.
Closes#810
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes the Windows release CI failure where integration tests could not find the cpd binary because they looked for target/debug/cpd without the .exe extension.
Closes#827 follow-up
apk