Off by default. `--similarity RATIO` (config `similarity`, Action input
`similarity`, MCP `check_duplication` argument) summarizes every function,
method and arrow function of a JavaScript/TypeScript/JSX/TSX file by the
bag of 4-grams over the pre-order sequence of its AST node types (oxc),
indexes the bags with MinHash + LSH banding, and reports pairs whose
weighted Jaccard index reaches RATIO as `similar` clones with a
`similarity` value. Names and literal values are not part of the summary:
a renamed copy scores 1.0, one inserted line ~0.9, two inserted statements
plus renames ~0.75.
Functions must clear --min-tokens/--min-lines, nested functions are never
paired with their parent, and pairs already covered by an exact, renamed
or merged clone are skipped. Values outside (0, 1] warn and disable the
option. Reporting reuses the Type-3 vocabulary (console `similar ~0.75`,
JSON kind/similarity, SARIF jscpd/near-miss-code).
New: cpd-core::similarity (FunctionSig, SimilarityIndex, bag_jaccard,
minhash), cpd-tokenizer::functions (oxc visitor extractor), MCP function
index built lazily per scan. fixtures/type3-demo/similar-functions with
README numbers, docs, unit + integration + MCP stdio tests. Also adds the
missing max_gap_lines/similarity fields to the --debug config output.
Everything belonging to the v4 TypeScript engine moves to the master-v4
branch: apps/, packages/, build-utils/, changesets, pnpm workspace, turbo,
the Node.js CI workflow and the ts-* release jobs. What remains is the
Rust workspace under rust/, the npm wrapper and platform packages, the
GitHub Action, Docker image, flake, skills, benchmark and fixtures/.
- rust.yml gains a smoke job that runs the release binary against
fixtures/ (json, sarif, html) and asserts clones and reporters
- release.yml keeps only the Rust jobs; pnpm removed from every workflow;
audit.yml keeps cargo audit + cargo deny; dependabot drops the root npm entry
- root package.json becomes a thin private package depending on the exact
jscpd@5 version so the pre-commit hook keeps installing the binary;
sync-version.mjs keeps it in step
- .pre-commit-hooks.yaml used v4-only flags that the v5 CLI rejects; fixed
- FORMATS.md is now generated from the Rust tokenizer (224 formats)
- README, docs, CONTRIBUTING, SECURITY, package and crate descriptions,
benchmark and skills rewritten for v5 on its own terms: no v5-vs-v4 speed
multipliers anywhere, a single "Looking for v4?" pointer to master-v4
Improvements from reviewing the MCP tool surface against real agent
usage:
- Clone/match lists are sorted biggest-first (token_count desc) before
the 'limit' cap. Detection emits path order, so truncation previously
kept the alphabetically-first clones instead of the most important
ones. Applies to check_current_directory, check_duplication, and the
new get_file_clones (project clones sorted once per scan).
- check_duplication accepts file extensions as format aliases: "js",
"py", "tsx" resolve via get_format_by_extension, removing the most
likely first-use failure for agents that pass the extension. The
resolved format is echoed in the payload.
- check_duplication gains the same 'limit' argument (default 100) and
truncation contract as check_current_directory: 'count' is the
untruncated total, 'returned' the list length, 'note' on truncation.
- New tool get_file_clones(path, limit?) — clones involving one file,
for the dominant file-scoped agent workflow ("what in this file is
duplicated, where?"). Accepts scan-root-relative or absolute paths,
folds embedded-code-block fragment ids into their parent file, and
distinguishes "no clones" from "file was not part of the scan".
- display_path normalizes separators to '/' so tool output is uniform
across platforms.
7 new unit tests (alias resolution, limit contract, file filter,
unscanned-file note, missing path, biggest-first ordering across
differently-sized clone pairs); tools/list coverage updated to the
four-tool surface. Full workspace suite green.
Refs #891
check_current_directory previously answered with counts only, which
left an MCP client unable to see where the duplication actually is.
The server now keeps the clones from the last scan and returns them:
- duplications: [{format, fileA, startA, endA, fileB, startB, endB,
lines, tokens}] with paths relativized to the scan roots and
sub-format fragment ids (embedded code blocks) folded into their
parent file
- optional 'limit' argument (default 100) caps the list so heavily
duplicated projects cannot flood an LLM context; 'clones' always
carries the untruncated total, 'returned' the list length, and a
'note' marks truncation — no silent capping
- non-integer 'limit' is rejected with -32602
Unit test asserts list shape, cross-file attribution, and the
truncation contract; the end-to-end stdio session now exercises the
tool as well.
Refs #891
Closes the gap reported in #891: the HTTP-only jscpd-server cannot be
used by MCP clients that spawn and manage a stdio process themselves
(Claude Desktop/Code, Cursor, APM, ...). The Rust binary now serves the
Model Context Protocol directly:
cpd --mcp /path/to/project
Transport & protocol:
- newline-delimited JSON-RPC 2.0 on stdin/stdout; stdout carries
protocol messages only, logging goes to stderr; exits 0 on stdin EOF
- protocol revision 2025-06-18, accepting 2025-03-26 and 2024-11-05
clients via version negotiation; ping, tools/list, tools/call;
notifications ignored; parse errors reported (-32700) without
killing the session
Tools (mirroring the HTTP jscpd-server):
- check_duplication(code, format) — checks a snippet against the
scanned project; results are compact JSON in a text content block
with file/snippet line ranges; unknown formats and sub-threshold
snippets produce informative results
- get_statistics() — files, clones, full Statistics of the last scan
- check_current_directory() — re-scan the configured paths
Performance: the project is scanned once at startup and cached as
detection-ready token hashes (PreparedSource pools per format /
cross-format group), so snippet checks only tokenize the snippet and
run one pool detection — milliseconds even on large codebases. All
standard detection options (--min-tokens, --format, --cross-formats,
--ignore, ...) apply to the scan and to snippet checks.
Supporting refactor in cpd-finder: orchestrate::run() split into
build_thread_pool() + prepare_scan_in() (walk + tokenize, returning
PreparedScan) so the MCP server can keep prepared sources around;
run() behavior is unchanged and detect_prepared is untouched.
strip_types_formats made pub for snippet tokenization parity;
PreparedSource now derives Clone. hyperfine A/B vs a master-built
baseline shows no regression on the normal CLI path (39.0ms vs
41.2ms, within noise).
Tests: 12 unit tests for the protocol handler and tools, plus 2
end-to-end tests that spawn the real binary and drive a full stdio
session (initialize → tools/list → tools/call → EOF), asserting
stdout is protocol-only. Full workspace suite green.
Docs: docs/ai-ready.md MCP section restructured into stdio (Rust v5)
and Streamable HTTP (Node v4) transports with client config examples;
docs/rust.md options table; root README; rust/CHANGELOG.md 5.0.16.
The jscpd-server (Node) stdio subcommand asked for literally in #891
can follow separately; this provides the stdio transport via the v5
binary that ships with jscpd@5.
Refs #891
Adds a refactoring-hotspot summary computed from data the detector
already holds in memory, appended to the run output when requested:
- --summary enable the summary block (also `summary` in .jscpd.json)
- --summary-top <n> entries per list (default 10, `summaryTop`)
- --summary-by <m> ranking metric: tokens|lines|size|complexity (`summaryBy`)
Output: top-N files (tokens, lines, size, complexity estimate, dup%)
strictly ranked by the chosen metric — every row carries all metrics —
plus a top-N folder rollup (direct-parent aggregation, each file counted
once). totalFiles/totalFolders always report pre-truncation counts.
Complexity is a language-agnostic cyclomatic estimate: 1 + decision-point
tokens (if/elif/elsif/elseif/unless/for/foreach/while/until/case/cond/
when/catch/rescue/except/andalso/orelse/&&/||/and/or/?/??), matched
ASCII-case-insensitively so uppercase-keyword languages (SQL, PL/SQL,
Fortran, COBOL, BASIC) count too. Verified across all 157 fixture
languages.
Reporter integration:
- console / console-full: aligned tables after the normal output
- ai: compact one-line-per-entry block (LLM-token-efficient)
- json: additive `summary` key, absent when the flag is off — schema
unchanged for existing consumers
Performance: the only unconditional change is capturing the file's byte
size from the existing mmap; all summary work runs post-detection and
only when --summary is set. hyperfine A/B vs master baseline shows
identical times without the flag (39.8ms vs 40.2ms) and ~2ms summary
cost when enabled.
New: cpd-core/src/summary.rs (model + computation, 14 unit tests),
cpd-reporter/src/summary_render.rs. ReportContext gains an optional
summary reference (ReportContext::new is unchanged; with_summary added).
SourceFile gains a serde-defaulted `bytes` field. 6 new CLI integration
tests; full workspace suite green.
Follow-ups tracked in #934: HTML report section, TypeScript parity port.
Refs #934
The Streamable HTTP transport requires servers to validate the Origin
header on every connection, and /mcp had no such check while listening on
the default 0.0.0.0 binding, so a browser could reach the tools through
DNS rebinding.
Guard the endpoint with the SDK's own request guards. Origin validation
is always on: a request with no Origin header is served, since non-browser
MCP clients send none, while a present, unlisted origin is answered with
403 and a JSON-RPC error carrying no id. Host validation stays a policy
decision — a loopback bind gets the localhost allowlist automatically, a
deliberate external bind is only restricted once the operator names the
hostnames, so remote deployments keep working. Both lists are extendable
with the new --allowed-origin and --allowed-host options.
Also make the test SSE reader follow the event framing rules: it now
splits events on blank lines, joins each event's data fields with a
newline and skips comment keep-alives, so a multi-line payload no longer
fails to parse and a notification emitted before the result is no longer
mistaken for the response.
Assert the cache fields on server/discover as well: the 2026-07-28
caching rules list it as a cacheable result alongside the list
operations.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Migrate the jscpd-server MCP endpoint from the monolithic
@modelcontextprotocol/sdk v1 to the split v2 packages
(@modelcontextprotocol/server and @modelcontextprotocol/node) and serve
protocol revision 2026-07-28 through the SDK's own primitives instead of
hand-rolled transport wiring.
- replace the session map and the initialize-gated POST handler with
createMcpHandler + toNodeHandler mounted on every method of /mcp, so
modern requests are direct and stateless: no initialize, no
Mcp-Session-Id, protocol version and client capabilities travel in the
per-request _meta envelope, and server/discover answers discovery
- let the SDK own the Streamable HTTP behaviour: MCP-Protocol-Version,
Mcp-Method and Mcp-Name header validation, the resultType
discriminator on every result, and the 405 answers for the 2025-era
session operations
- configure per-operation cache hints so cacheable results carry the
required ttlMs/cacheScope, with a shorter, private hint for the
statistics resource that changes on every rescan
- keep dual-era compatibility: 2025-era clients are served by the
stateless legacy fallback backed by the very same server factory
- hold the service and the MCP endpoint in explicit application state
instead of handler closures and an implicit per-session transport map
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Split the overloaded README.md into focused docs:
- docs/typescript.md: v4 (Node.js) CLI, reporters, API, config
- docs/rust.md: v5 (Rust) CLI, reporters, blame, differences from v4
- docs/ai-ready.md: AI reporter, agent skills, MCP server
- docs/api.md: Programming APIs (TypeScript and Rust)
- docs/packages.md: Monorepo package and crate overview
- Trim README.md to a concise entry point with links to docs/
- Slim down rust/README.md, rust/jscpd/README.md, rust/npm/README.md:
remove duplicated content, link to docs/rust.md for details
- Fix factual errors verified against actual CLI output:
- Format count: 223 (v5) and 224 (v4), not '225+'
- Reporter output filenames: jscpd-report.* not cpd.*
- --max-size: no default in v5 (not 512KB)
- --formats-exts: same hyphenated form in both v4 and v5
- --no-gitignore: same flag name in both v4 and v5
- Update rust/jscpd/package.json: add 'cpd' bin so jscpd@5
installs both jscpd and cpd commands
- Update crate versions in rust/README.md to match source (0.1.3/0.1.4)
apk