Closes#1002 (the --history half; --ratchet is deliberately not
implemented, the report shows the headroom instead).
`jscpd src --history v5.0.0..HEAD` (or `--history-since 2026-01-01`)
lists the commits with `git log`, checks each one out into a temporary
detached worktree (the --baseline-from-ref machinery, now shared through
baseline_ref::map_scan_paths), scans it with the run's own configuration
and keeps the totals; the current run is the series' last point.
`--history-every N` keeps every Nth commit counted from the newest,
`--history-limit N` (default 30) samples a long series evenly keeping
both ends. Nothing runs unless one of the flags is given.
Output: the console reporter appends a sparkline of the duplication
percentage, a table (commit, date, files, lines, clones, dup lines,
dup%, change, subject) with the change column red when duplication rose
and green when it fell, the overall trend line, and, when --threshold is
set and the latest value is under it, the headroom with the value the
threshold could be tightened to. The json reporter adds a `history`
object; the ai reporter prints one compact line per point. Other
reporters are unchanged.
New cpd_core::history (model, sparkline, change/headroom helpers with
unit tests), cpd::history (git log parsing, sampling, worktree scans),
cpd_reporter::history_render. Config keys history / historySince /
historyEvery / historyLimit; action input `history`. Integration tests
build a four-commit repo with fixed dates and check the JSON series,
the console block, every/limit thinning and the error paths. 969
workspace tests pass, clippy clean.
Docs: History section and option rows in docs/rust.md, README bullet,
action input row, fixtures/history-demo/README.md with a script that
builds the demo repository and the expected output.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#1047.
Four situations used to write an empty report and exit 0, so a CI job
that checks only the exit code saw a green run:
- `--format` naming a format that is not supported (a warning since #964)
is now `Error: --format: 'x' is not a supported format`, exit 1. Names
introduced through --formats-exts or --formats-names stay accepted.
- A scan path that does not exist is `Error: path does not exist: <p>`,
exit 1, before anything runs.
- A reporter that fails to write its output still prints the existing
`Reporter '<name>' error: ...` line; the run now ends with
`ERROR: a reporter failed to write its output` and exit 1, after the
other reporters have run.
- A scan that analyzes no files (paths exist, but nothing matched the
--format/--ignore/--pattern filters or every file was below
--min-tokens) prints `Warning: jscpd analyzed no files` and still
exits 0, so an intentionally empty tree passes. The new
`--fail-on-empty` flag (config key `failOnEmpty`, action input
`fail-on-empty`) makes it exit 1. Reports are written before the check.
Docs: an "Exit codes" section in docs/rust.md, option rows in
docs/rust.md and docs/ci-and-hooks.md, a README feature bullet.
Fixtures: fixtures/fail-on-empty-demo with a `code/` clone pair, a
`nothing/` comment-only file, and a README quoting every command's
output and exit code. Tests: the #964 warning test becomes an error
test, plus integration tests for each new exit path and for the config
key; 531 tests pass, clippy clean.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Off by default. `--similarity RATIO` (config `similarity`, Action input
`similarity`, MCP `check_duplication` argument) summarizes every function,
method and arrow function of a JavaScript/TypeScript/JSX/TSX file by the
bag of 4-grams over the pre-order sequence of its AST node types (oxc),
indexes the bags with MinHash + LSH banding, and reports pairs whose
weighted Jaccard index reaches RATIO as `similar` clones with a
`similarity` value. Names and literal values are not part of the summary:
a renamed copy scores 1.0, one inserted line ~0.9, two inserted statements
plus renames ~0.75.
Functions must clear --min-tokens/--min-lines, nested functions are never
paired with their parent, and pairs already covered by an exact, renamed
or merged clone are skipped. Values outside (0, 1] warn and disable the
option. Reporting reuses the Type-3 vocabulary (console `similar ~0.75`,
JSON kind/similarity, SARIF jscpd/near-miss-code).
New: cpd-core::similarity (FunctionSig, SimilarityIndex, bag_jaccard,
minhash), cpd-tokenizer::functions (oxc visitor extractor), MCP function
index built lazily per scan. fixtures/type3-demo/similar-functions with
README numbers, docs, unit + integration + MCP stdio tests. Also adds the
missing max_gap_lines/similarity fields to the --debug config output.
Off by default (0). When set to N, clones of the same file pair whose
fragments follow each other in both files with at most N unmatched lines
between them are merged into one clone of kind `similar`, with `tokens` =
matched tokens and `similarity` = matched tokens over the tokens of the
longer merged span. Adjacent exact windows that share a boundary token are
counted once; chains merge transitively; overlapping or reordered fragments
never merge. The pass runs after every other filter and returns its input
untouched at 0, so default output is unchanged.
Reporting: console `Clone found (fmt, similar ~0.91)`, ai `[~0.91]`, JSON
"kind"/"similarity" (rounded to three decimals), SARIF rule
jscpd/near-miss-code with a similarity property (declared only when
referenced), Code Climate check_name.
Config key maxGapLines (kebab alias), GitHub Action input max-gap-lines,
docs section, fixtures/type3-demo with a README of runnable commands, unit
tests for the merge pass and reporters, and a binary-level integration test.
Closes#998.
Three opt-in flags (config keys ignoreIdentifiers, ignoreLiterals,
ignoreAnnotations) normalize token classes before hashing so that blocks
which differ only in names, values or annotations are detected:
- identifiers hash as `$id`; keywords are kept (the JS/TS tokenizer
classifies them, other languages use a built-in common-keyword list)
- string literals hash as `$str`, numeric literals as `$num`
- `@Name`, `@a.b.Name` and `@Name(...)` runs are dropped in Java, Kotlin,
Scala, Groovy, Python, Dart, Swift, JavaScript and TypeScript
Each clone now carries `kind: exact | renamed`. DetectionToken gains a
raw_hash (equal to hash unless normalization rewrote the token); the
PreparedSource materializes raw hashes only when a token was normalized,
and the detector compares raw hashes over the matched run to classify a
pair. A dropped annotation salts the raw hash of the token before it, so a
pair differing only in annotations inside the matched run is `renamed`.
Reporters: console prints `Clone found (fmt, renamed)`, ai appends
`(renamed)`, json adds "kind", sarif uses rule jscpd/similar-code (declared
only when referenced), codeclimate the same check_name. Default runs are
unchanged apart from the additive "kind": "exact" JSON field.
Also: GitHub Action inputs, docs section, fixtures/type2-demo with a README
of runnable commands, unit tests in tokenizer/core/reporter, CLI flag
tests, and a binary-level integration test.
Implements phase 2 of #944: an ephemeral baseline built from a git
ref's tree, for PR gates without a committed baseline file.
cpd --baseline-from-ref origin/main --fail-on-new-clones .
- checks the base ref out into a temporary detached git worktree
(removed afterwards; shells out to git like blame does), scans it
with the same detection configuration, and compares the current run
against that in-memory fingerprint set via the phase 1 machinery
(new baseline::apply_in_memory)
- scan paths are remapped into the base tree; paths absent there are
simply new code
- a missing ref fails with a clear shallow-checkout hint (git fetch /
fetch-depth: 0); conflicts with --baseline and --update-baseline,
and --fail-on-new-clones now accepts either baseline source
- config key baselineFromRef; GitHub Action input baseline-from-ref
Refs #944
Implements phase 1 of #944: a committed baseline of clone fingerprints
so CI tolerates recorded duplication and fails only on regressions.
- --baseline <file>: content-hash fingerprints (same value as the SARIF
partialFingerprints jscpdCloneHash/v1) with per-fingerprint
multiplicity counts; clones absent from the file are marked new
- --update-baseline: rewrite the file from the current run (creating it
if missing) and print added/removed fingerprint counts
- --fail-on-new-clones[=N]: exit 1 when more than N (default 0) new
clones are found, independent of --threshold
- config keys baseline / failOnNewClones, with the baseline path
resolved relative to the config file
- reporters: [NEW] markers and "(N new)" found count in console and
console-full, per-clone isNew plus newClones/newDuplicatedLines
statistics in json, level "error" for new clones in sarif, and
jscpd_new_clones / jscpd_new_duplicated_lines gauges in openmetrics
- baseline file is versioned and sorted one fingerprint per line for
reviewable diffs and trivial merges
- GitHub Action: baseline, update-baseline, fail-on-new-clones inputs
Refs #944
apk