- FinderError was never constructed; run() cannot fail, so its error type
is Infallible and the documented run(&config).unwrap() keeps compiling.
- tokenize_sfc_block_inner and tokenize_to_detection_inner forwarded to
tokenize_format_to_detection; call it directly.
- extract_line_number was Some(error.line()).
- Timer wrapped Instant; use Instant.
serde, anyhow, thiserror and log from cpd-finder; thiserror from
cpd-reporter; xxhash-rust and dev serde_json from cpd-tokenizer; anyhow,
log and env_logger from cpd. None of them is referenced in code.
`tokenize_line_content`, which cuts every line of every non-JavaScript file
into tokens, was a 218-line loop with a cyclomatic complexity of 49. Each of
its nine token classes repeated the same index, column and byte-offset
bookkeeping and its own ignore-region check.
A `LineCursor` now owns the position, the column and the ignore-region kind,
and each class is one call on it: `take`, `take_while`, `take_string`,
`take_word` or `take_rest`. The loop that dispatches between them is 37 lines
with a complexity of 15. The separate Lua `--[[` branch goes: the line comment
check right after it already matched Lua's `--` and produced the same token.
`comment_kind` is folded into `LineCursor::kind`.
Checked against the previous implementation, copied beside the new one for a
one-off differential test: every text file in `fixtures/` and `rust/` — 839
of them — under all seven comment styles, plus edge cases for unterminated
and escaped strings, block comments across lines, ignore regions, and
multi-byte and non-breaking-space input, produced 5,039,506 tokens identical
in kind, text, line, column and offset. On ten trending projects (7,181 files,
11.2 million tokens) the JSON report is byte-identical, and fifteen
alternating single-worker runs put old and new within noise of each other
(median 4.04 s old, 4.01 s new).
basta reads JavaScript, TypeScript and Python, so `jscpd --dead-code` has no
view of the Rust crates, and the compiler does not warn about an unused `pub`
item. Every crate in the workspace is `publish = false`, though, so a `pub`
item nothing in the workspace uses is dead as surely as a private one. A scan
of every `pub` item's uses across the crates, their tests and the fuzz
targets, checked by hand, found:
- `CloneKind::is_exact`: used nowhere.
- `ConfigResult::has_diagnostics`, `basta::lang::language_of`,
`TokenizeOptions::normalizes` and `TokenizeOptions::with_code_ignore_patterns`:
called only by tests written for them. The language test now asks
`analyzer_for` directly.
- `stats_with_pct` is a reporter test fixture that was compiled into the
library; it moves inside the `#[cfg(test)]` fixtures module that re-exported it.
- Fields kept alive by `#[allow(dead_code)]`: `MarkdownFence::front_matter`
and `SfcBlock::tag` are written and never read, and `Options::list` is never
read because `main` reads `cli.list`.
- `scripts/regression_check.sh` and its 1528-line `tests/golden.json`: a gate
for the Wave 2/3 migration that nothing runs, against a report schema that
has since changed.
Behaviour is unchanged: the JSON reports for the fixtures corpus in mild, weak
and strict modes, with Type-2 normalization, and with the complexity summary,
the console output and the dead-code demo output are byte-identical before and
after.
`cargo publish` verifies by building the packaged crate against the
*published* versions of its dependencies, not the local paths. basta uses
`cpd_core::deadcode`, `cpd_reporter::deadcode` and a public `LineIndex` —
none of which exist in the versions on crates.io — so every attempt failed
with "failed to verify package tarball", and basta could not reach crates.io
until a jscpd release happened to carry those three crates along.
That is the coupling this workflow exists to avoid. It now walks basta's own
dependency graph, publishes in topological order whatever is missing, and
waits for the index between crates. jscpd is excluded by construction: it is
a sibling, not a dependency, and releasing it is not this workflow's job.
Bumps the three crates that actually changed: cpd-core 0.1.13 -> 0.1.14
(the `deadcode` module), cpd-tokenizer 0.1.15 -> 0.1.16 (`LineIndex` made
public, `.pyi`), cpd-reporter 0.1.14 -> 0.1.15 (the dead-code reporters).
cpd-finder is untouched and stays where it is.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Adds a second question about the same tree: not "what is duplicated" but
"what does nothing reach". A new `basta` crate builds the import graph from
the project's entry points and walks it, reporting unused files, unused
exports, unused module-private declarations, unused imports and — opt-in —
unused class members, across JavaScript, TypeScript, JSX, TSX and Python.
It ships two ways: `jscpd --dead-code` inside the existing binary, and a
standalone `basta` command. Both reuse the walker, the filters, all fifteen
reporters and the `--threshold` / `--exit-code` gates, so nothing new has to
be learned to run it.
Because it is a graph traversal rather than a reference count, dead code
cascades: a helper whose only caller is itself dead is reported too. Every
finding carries a confidence score and, below 100, the reasons it might be
wrong — `eval`, an unrecognised decorator, a wildcard re-export, a name seen
only in a string, a file in the scan that did not parse. `--min-confidence`
sets the floor at 60.
Entry points come from `package.json`, `pyproject.toml`, framework and
file-name conventions, and anything a shell script, CI workflow, Makefile or
Dockerfile names by path. CommonJS is read alongside ESM. A project that
renames its own import paths is read on its own terms: `paths` and `baseUrl`
from `tsconfig.json` and `jsconfig.json` are resolved, through a relative
`extends` chain and per package in a monorepo. A file named only by a path
string — a worker spawned through `new URL`, a build entry, a vitest setup
file — counts as used.
Languages plug in through one trait, `basta::lang::Analyzer`, which owns
parsing, specifier resolution, entry-point conventions, manifests, alias
configs and path traits. Nothing outside `lang/` is language-specific;
`docs/basta-extending.md` walks through adding one.
Validated against knip, fallow, vulture, skylos, fossil-mcp and repowise on
trending repositories. On a corpus of ten of them — 11k files, 3.3M lines —
every one of basta's 1,806 findings was checked against an import graph
rebuilt independently of it: no unused-file, unused-import or unused-symbol
claim survived as wrong.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Phase 1 of the jscpd-guided DRY pass (69 clones, 2.4% in rust/crates):
- cpd-core: Location::new, Fragment::new/with_blame and CpdClone::exact
replace the 20-line struct literals repeated across the reporter tests
and fixtures.
- cpd-reporter: ReportContext::new instead of the copied struct literal
in every test; shared::fixtures::report_to_file replaces seven
near-identical run_<reporter>_report helpers; shared::print_appendices
prints the --summary and --history blocks for both console reporters.
- cpd-tokenizer: embedded::html_only_maps and tokenize_blocks_shifted
hold the code razor.rs and sfc.rs had copied from each other.
971 workspace tests pass, clippy clean.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Copilot review: gating on a <template> block let a script-only Vue
component keep <script>/<style> wrapper tags in its html stream. A Vue
file is blocks-only whatever blocks it has, so gate on the format instead.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Sorting the html tokens (previous commit) restores source order, but the
wrapper tags of a Vue file — <template>, </template>, <script ...>,
</script>, <style ...>, </style> — are identical in every Vue file and
sit at both ends of it. In source order they stretch every template clone
to the last </style>: on fixtures/vue the html clone went from
[1:1 - 10:9] to [1:1 - 262:10] and duplicated lines from 289 to 541,
counting the script and style bodies as duplicated html.
A file with a <template> block keeps all of its markup inside that block,
so the skeleton outside the blocks is only wrapper tags and is now left
out of the html stream. The clone becomes the template body itself:
file1.vue:html [2:3 - 10:9], 75 tokens. Svelte and Astro have no template
wrapper; their top-level markup is the skeleton and is kept, so their
output is unchanged. The sort stays, for custom blocks that map to html.
Fixture: fixtures/sfc-demo/template-range with a README showing the
template-only range; docs/rust.md links it.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Restore source-offset order after combining Vue SFC wrapper markup with template content so clone spans cannot jump across unrelated lines.
Add a regression test for the token-order invariant. Verified with the full Cargo workspace test suite, Clippy, formatting, and the malformed PR 3257 reproduction.
`comment_style` names no arm for markdown, so markdown prose reaches the
`_ => CommentStyle::CStyle` fallback and takes the comments of C.
Markdown prose can hold `/*`: inside a glob such as `docs/**`, in a code span,
or in plain text. It opens a block comment that never closes, and every later
clone in that file goes unreported. The file still counts as analyzed and its
token total still looks healthy, so nothing signals the loss.
CommonMark defines no comment of its own. Its only comment is the HTML comment,
as HTML block type 2. https://spec.commonmark.org/0.31.2/
This adds the arm, and drops the `#[allow(dead_code)]` on `CommentStyle::None`,
which until now was constructed nowhere.
Measured with jscpd 5.1.2 on darwin-arm64. Two files that share a paragraph are
reported as a clone. Add one line holding `docs/**` above the paragraph in one
of them, and the clone is gone; a closing `*/` later in the file brings it
back. A `/*` inside a fenced block is harmless, because markdown.rs sends a
fence to the grammar of its own language.
`markup` (html, xml, svg) is absent from every arm too, so it takes the same
fallback. HTML does have `/* */` inside a `<style>` block, so that case needs a
different answer and is left alone here.
Verified on linux-x64 with rustc 1.94.1. The workspace pins 1.97.1 and its
crates could not be fetched there, so `cargo build`, `cargo nextest run
--workspace` and the new cpd-finder integration test were not run. What was
run:
- `cargo fmt --all --check` — clean.
- generic.rs compiled on its own against the three model types it uses: 12 of
12 unit tests pass. With the markdown arm deleted, the two new tests fail and
the other ten still pass.
- clippy over generic.rs with `-D warnings` — clean.
- On the fixture pair, `tokenize_generic(.., "markdown")` gives second.md 345
comment tokens and no words below the `docs/**` line without the arm, and 50
words with it — the same 50 that first.md carries.
- jscpd 5.1.2, linux-x64, on the same pair: no clone. Drop the line holding
`docs/**` and it reports one markdown clone, 4 lines and 54 tokens.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XacBaczr3GsFfEWWqGdz99
A JavaScript/TypeScript file with any parse diagnostic (a redeclaration,
a recoverable syntax error) was handed to the word-split fallback
tokenizer. Its tokens then never matched an oxc-tokenized file, so every
cross-file clone involving it vanished while its self-clones survived —
the symptom reported in #1023, where a second copy of a function in one
file made all a↔b clones disappear.
Tokens come from the lexer, not the parser: only `panicked` (or an empty
token stream) now triggers the fallback. Function extraction for
--similarity uses the partial AST on recoverable errors the same way.
Closes#1023.
Function extraction is now a language plug-in: `FunctionExtractor`
(grammar id, served formats, extract) with a static `EXTRACTORS` registry
consulted by `extract_functions` / `supports_functions` /
`supported_function_formats`. `OxcExtractor` is the first entry (JS/TS/
JSX/TSX). Every RawFunction/FunctionSig records its grammar and the scorer
refuses pairs across grammars, so a tree-sitter-backed extractor for
another language is a self-contained addition with no change to scoring,
CLI, MCP or reporters. Module docs describe the two steps; the MCP note
lists supported formats from the registry.
Off by default. `--similarity RATIO` (config `similarity`, Action input
`similarity`, MCP `check_duplication` argument) summarizes every function,
method and arrow function of a JavaScript/TypeScript/JSX/TSX file by the
bag of 4-grams over the pre-order sequence of its AST node types (oxc),
indexes the bags with MinHash + LSH banding, and reports pairs whose
weighted Jaccard index reaches RATIO as `similar` clones with a
`similarity` value. Names and literal values are not part of the summary:
a renamed copy scores 1.0, one inserted line ~0.9, two inserted statements
plus renames ~0.75.
Functions must clear --min-tokens/--min-lines, nested functions are never
paired with their parent, and pairs already covered by an exact, renamed
or merged clone are skipped. Values outside (0, 1] warn and disable the
option. Reporting reuses the Type-3 vocabulary (console `similar ~0.75`,
JSON kind/similarity, SARIF jscpd/near-miss-code).
New: cpd-core::similarity (FunctionSig, SimilarityIndex, bag_jaccard,
minhash), cpd-tokenizer::functions (oxc visitor extractor), MCP function
index built lazily per scan. fixtures/type3-demo/similar-functions with
README numbers, docs, unit + integration + MCP stdio tests. Also adds the
missing max_gap_lines/similarity fields to the --debug config output.
Closes#998.
Three opt-in flags (config keys ignoreIdentifiers, ignoreLiterals,
ignoreAnnotations) normalize token classes before hashing so that blocks
which differ only in names, values or annotations are detected:
- identifiers hash as `$id`; keywords are kept (the JS/TS tokenizer
classifies them, other languages use a built-in common-keyword list)
- string literals hash as `$str`, numeric literals as `$num`
- `@Name`, `@a.b.Name` and `@Name(...)` runs are dropped in Java, Kotlin,
Scala, Groovy, Python, Dart, Swift, JavaScript and TypeScript
Each clone now carries `kind: exact | renamed`. DetectionToken gains a
raw_hash (equal to hash unless normalization rewrote the token); the
PreparedSource materializes raw hashes only when a token was normalized,
and the detector compares raw hashes over the matched run to classify a
pair. A dropped annotation salts the raw hash of the token before it, so a
pair differing only in annotations inside the matched run is `renamed`.
Reporters: console prints `Clone found (fmt, renamed)`, ai appends
`(renamed)`, json adds "kind", sarif uses rule jscpd/similar-code (declared
only when referenced), codeclimate the same check_name. Default runs are
unchanged apart from the additive "kind": "exact" JSON field.
Also: GitHub Action inputs, docs section, fixtures/type2-demo with a README
of runnable commands, unit tests in tokenizer/core/reporter, CLI flag
tests, and a binary-level integration test.
First release from the v5-only master. Bumps jscpd to 5.1.2 and every
sub-crate (cpd-core 0.1.11, cpd-tokenizer 0.1.13, cpd-finder 0.1.14,
cpd-reporter 0.1.12) so the new crates.io metadata from #988 and the
Windows drive-path fix in cpd-finder are republished. Adds the 5.1.2
changelog section (linux-arm64-musl package, cargo-binstall, Docker
image, jscpd --version naming, pre-commit hook args, repository split)
and refreshes the version examples in docs/.
Pre-release checks: cargo fmt, clippy -D warnings, cargo test --locked
(22 suites), cargo check --all-targets --target x86_64-pc-windows-msvc,
release-notes extraction dry-run against rust/CHANGELOG.md.
Rust / crates.io
- report the invoked binary name in --version/--help (jscpd vs cpd)
- add repository, documentation, keywords, categories to all crates
- exclude tests/ from the jscpd crate package; add cargo-binstall metadata
- expand the crates.io README and render it on docs.rs
npm
- add funding field to jscpd, cpd and jscpd@4 packages
- add jscpd-linux-arm64-musl platform package and clearer unsupported-platform error
- enable provenance for the v4 changeset publish
Pipelines
- rust.yml: windows-latest in the matrix, drop dead schedule guard
- release.yml: cosign-sign release assets, floating v5 tag, LICENSE in tarballs,
invoke the new docker.yml (GHCR multi-arch image) after each release
- audit.yml: nightly cargo audit, cargo deny and pnpm audit
- Dockerfile: multi-stage image from release binaries (distroless), mcp target for Glama
Docs / community
- README: shields badges, single primary install per OS, GitHub Action snippet,
link to jscpd.dev, v5.1 features, 15 reporters, 224 formats, 8 platforms
- docs: @v5 action tag, Docker usage, baseline section, current crate versions
- YAML issue forms with engine selector, config.yml, CODEOWNERS
- remove stale .github/CONTRIBUTING.md stub, custom.md, TODO.md; untrack rust/jscpd-report.json
- examples: mark as v4 API, add a Rust cpd-finder example
Republishes the jscpd npm wrapper with platform pins that match the
release. jscpd@5.1.0 shipped optionalDependencies pinned to the 5.0.16
binaries, so `npm i jscpd@5.1.0` resolved an engine one release behind
and `jscpd --version` printed `cpd 5.0.16` — every 5.1.0 fix, including
the Windows --baseline-from-ref fix, was absent for jscpd users. The cpd
package, the platform packages and the crates.io release were all correct;
only the wrapper's pins were stale.
Root cause in scripts/sync-version.mjs: the wrapper's version and its
platform pins were written behind one `version !== npmVersion` guard, so
once anything set version before the script ran, the guard read "already
up to date" and skipped the pins. Version and pins are now updated
independently, and the script ends by verifying that every npm version and
platform pin equals the release version, exiting non-zero otherwise.
release.yml runs this script, so the same mismatch now fails the release
rather than publishing. This is the 5.0.13 defect again — that fix covered
the cpd package but not the jscpd wrapper.
Sub-crates bumped for the code that changed since the published 5.1.0
(cpd-tokenizer 0.1.12, cpd-finder 0.1.13, cpd-reporter 0.1.11); cpd-core
is unchanged at 0.1.10. The only source delta is the clippy collapsible_if
rewrites unmasked by the MSRV correction, which are behaviour-preserving.
Pre-release verification: cargo fmt --check, clippy --workspace
--all-targets -D warnings, 847 tests, cargo +1.96 check at the declared
MSRV, and — per the Windows-only blind spot that blocked 5.0.15, 5.0.16
and 5.1.0 — cargo check --workspace --all-targets --target
x86_64-pc-windows-msvc, all clean. Release-notes extraction for 5.1.1
dry-run against rust/CHANGELOG.md returns the right section.
Raising rust-version to 1.96 turned on 11 clippy::collapsible_if errors
across four crates, failing the Clippy step with -D warnings.
The lint is MSRV-gated: its suggested fix is a let-chain, stabilised in
Rust 1.88, so clippy stayed silent while the workspace claimed 1.87. The
violations were always there — the wrong MSRV was suppressing them.
Applied with `cargo clippy --fix` (machine-applicable suggestions only),
then `cargo fmt` to repair the indentation the fixer leaves behind. Every
site is a plain nested if with no else branch, so each collapse is a pure
rewrite with no change in control flow.
cpd-tokenizer javascript.rs, razor.rs, sfc.rs, ts_strip.rs (1 each)
cpd-finder walker.rs (5)
cpd-reporter sarif.rs (2)
jscpd main.rs (3)
Verified: clippy --workspace -- -D warnings clean, cargo fmt --check
clean, 847 tests pass, and cargo +1.96 check --workspace --all-targets
--locked still passes (let-chains predate the MSRV by eight releases).
Revert the 5.1.1 bump (its release run was cancelled before publishing
anything; crates.io already has 5.1.0 and npm has nothing newer than
5.0.16, so the registry-based detect resumes the 5.1.0 release on this
push) and fold the fingerprint fix into the 5.1.0 changelog section.
Claude-Session: https://claude.ai/code/session_01VvWzWfEK6oV4HTe6EoGkff
oxc_span alone at 0.147 left the other oxc crates at 0.144, so the
GetSpan trait no longer matched the AST types and cpd-tokenizer failed
to compile. The oxc crates only work when versioned together.
Claude-Session: https://claude.ai/code/session_01VvWzWfEK6oV4HTe6EoGkff
Coordinated bump of all six oxc crates (parser, allocator, span, ast,
ast_visit, syntax) — supersedes #920, which bumped only oxc_parser and
failed CI because oxc 0.144 requires rustc >= 1.95.
Adaptations for oxc 0.144 breaking changes:
- ParserReturn.errors renamed to .diagnostics
- TSModuleDeclaration split into TSExternalModuleDeclaration and
TSNamespaceDeclaration (oxc-project/oxc#25284); ts_strip visitor
updated to handle both with the same erasure semantics
Bump cpd/jscpd to 5.0.14 and changed sub-crates (cpd-core 0.1.7,
cpd-tokenizer 0.1.8, cpd-finder 0.1.9). Includes --cross-formats (#810)
and the prose-only markdown fix (#883).
sync-version.mjs now also syncs the cpd package optionalDependencies
(cpd@5.0.12 shipped pointing at the 5.0.11 binaries, which forced the
manual npm-only cpd@5.0.13) and its inter-crate dependency regex now
matches the actual Cargo.toml formatting (space before closing brace),
so dependency refs are bumped along with crate versions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The markdown display-path tokenizer only extracted fenced code blocks,
so .md files without fences produced 0 display tokens and were dropped
by the orchestrator's min-tokens gate before detection ever ran: they
never counted as "Files analyzed" during autodetect, and -f markdown
matched 0 files — a silent no-op for Markdown-only projects.
Tokenize the prose body on the display path too (blanking fenced code
and front matter, mirroring tokenize_markdown_maps), so prose-only
markdown is analyzed and identical prose pairs are detected as clones.
Embedded code fences keep being detected under their own sub-format
pools (javascript, python, ...), verified by the new integration test.
Fixes#883
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Detect duplicates across related formats via format equivalence groups
sharing one comparison pool, e.g. --cross-formats "javascript,typescript"
or the js-ts preset (javascript,jsx,typescript,tsx).
When a group mixes TypeScript with JavaScript, TS files are compared with
erasable type-only syntax stripped from the detection token stream (type
annotations, generics, interface/type declarations, as/satisfies, ?/!
markers, access modifiers, implements clauses, type-only imports/exports,
overload signatures, declare statements) so that
`function f(a: number): void` matches `function f(a)`. Stripping is
AST-driven (oxc_ast_visit over the already-parsed program) and only drops
tokens, so reported clone positions always reference the original source.
- cpd-tokenizer: new ts_strip module, tokenize_js_stripped,
TokenizeOptions.strip_types_formats (empty by default)
- cpd-finder: RunConfig.cross_formats, build_pools() pool grouping,
strip-set computation for TS-family formats grouped with JS
- cpd: --cross-formats flag, crossFormats config field (string,
array-of-strings, or array-of-arrays), js-ts preset, overlap merging,
unknown-format warnings, --debug output
- docs: rust.md option + Cross-Format Groups section, CHANGELOG entry
- tests: tokenizer strip tests, pool unit tests, cross-pool core test,
cpd-finder integration tests, CLI parse/config tests, e2e binary tests
with fixtures; default behavior unchanged (golden parity intact)
Non-erasable TS syntax with runtime semantics (enum, non-declare
namespace, parameter properties, import =, export =) is left intact and
will not cross-match. Cross-format clones are attributed to one member
format in per-format statistics.
Closes#810
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
These formats were absent from comment_style()'s match arms and fell
through to the C-style default, so ';' line comments were tokenized as
code. As a result --mode weak / --skip-comments could not drop them, and
the identical comment header every Emacs Lisp file carries (the GNU GPL
notice) was reported as duplicate code.
Route lisp/clojure/scheme/racket to the existing Semicolon comment style
(';' is their line comment) and add a tokenizer regression test.
Closes#849
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The boundary check at the start of the block for `if in_code` merans
that the match of `ch` against the similar boundary characters further
down would never be reached. We can remove this block in the `match`
entirely.
Extract Candidate struct and continuation/grow logic from
add_secondary_clones into SecondaryOpen methods. Optimize
LineCoverage interval merge and extend checks. Refactor
markdown front matter and fence parsing for clarity.
- Add curl install method (jscpd.dev/install.sh) to README Quick Start
- Remove unused badges (Ukraine, jscpd SVG, npm downloads, codecov,
FOSSA, Open Collective, nodei.co) from README; keep npm version,
license, CI, and NPM badge
- Remove Star History and GA tracker from README; restore NPM badge
- Add GA tracker to all sub-READMEs
- Set homepage to https://jscpd.dev in all package.json files
- Set homepage to https://jscpd.dev in all Cargo.toml files
apk