Two failures on the branch, neither on macOS, which is why they surfaced
only in CI.
`cargo deny` rejected `unicode_names2`, which `ruff_python_parser` pulls in
and which carries `Unicode-DFS-2016` alongside its MIT/Apache dual licence.
That licence covers the character tables and is the older wording of
`Unicode-3.0`, already on the allow list; added beside it.
The two new config-discovery tests asserted on `../project/.jscpd.json`,
and Windows prints the separator it joined with, so both failed there.
They now normalise backslashes before comparing — the assertion is about
which file was read, not about how the path is spelled.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
`cargo publish` verifies by building the packaged crate against the
*published* versions of its dependencies, not the local paths. basta uses
`cpd_core::deadcode`, `cpd_reporter::deadcode` and a public `LineIndex` —
none of which exist in the versions on crates.io — so every attempt failed
with "failed to verify package tarball", and basta could not reach crates.io
until a jscpd release happened to carry those three crates along.
That is the coupling this workflow exists to avoid. It now walks basta's own
dependency graph, publishes in topological order whatever is missing, and
waits for the index between crates. jscpd is excluded by construction: it is
a sibling, not a dependency, and releasing it is not this workflow's job.
Bumps the three crates that actually changed: cpd-core 0.1.13 -> 0.1.14
(the `deadcode` module), cpd-tokenizer 0.1.15 -> 0.1.16 (`LineIndex` made
public, `.pyi`), cpd-reporter 0.1.14 -> 0.1.15 (the dead-code reporters).
cpd-finder is untouched and stays where it is.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
`basta@0.1.0` published with `files: ["run-basta.js", "platform-map.js"]`,
so npmjs.com/package/basta shows an empty page — the one place most people
will look before installing anything. npm takes the README from the
published tarball, so a version bump is the only way to put one there.
The README is npm-facing rather than a copy of the crate's: it leads with
`npm install -g basta` and `npx basta src`, and it names the known gaps —
no Vue/Svelte/Astro, class members matched by name, PEP 420 namespace
packages — so a reader can judge whether the tool fits before running it.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
`cd rust && jscpd --basta ..` silently ignored the repository's own
`.jscpd.json`: config discovery only ever tried `.jscpd.json`,
`.config/jscpd.json` and `package.json` relative to the working
directory. The configuration is a property of the code being scanned,
not of where the shell happens to be, so a scan aimed at a project read
none of its ignores, formats or thresholds.
The cwd is still searched first and still wins, so nothing about an
existing invocation changes; each scanned path's directory is searched
after it, and a path that is also the cwd is not searched twice. A file
argument contributes the directory holding it. `ConfigSource` now
carries the path it was read from, so the "Using config from" line names
the file that was actually used rather than assuming it was the local one.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
basta is a separate product from jscpd — a different question about the
same tree — so it gets its own version, its own prebuilt packages and its
own trigger. jscpd goes out on every push to master that leaves a version
unpublished; basta goes out only when a `basta-v*` tag is pushed, or by
hand through workflow_dispatch. Neither release can drag the other along
or hold it up.
`basta-release.yml` owns the whole thing: it checks that the crate, the npm
manifest and the tag all name the same version and that the working tree is
already synced, publishes the crate, builds the eight targets and publishes
`basta-<platform>` plus the `basta` wrapper. It ends by asserting that npm's
`latest` really points at this release — the package carries 1.x versions
from an unrelated tool of the same name, and a dist-tag that quietly stayed
on those would hand every `npm i basta` the wrong program.
The one place the two cannot be separated is crates.io: `jscpd` depends on
`basta`, so cargo refuses to publish jscpd until a matching basta exists.
The jscpd crates job therefore still publishes basta when it is missing, and
that is now commented as the dependency requirement it is. Both paths skip a
version already on the registry, so whichever runs first wins.
npm needed the wrapper pattern `jscpd` and `cpd` already use: `rust/basta/`
holds the manifest, a platform map naming `basta-<platform>`, and a launcher
that falls back to a local `cargo build` when no platform package is
installed. `npm-prebuilt-package.mjs` grew a `--product` flag; its output for
jscpd is byte-identical to before. `prebuilt-targets.json` now carries a
platform label rather than a jscpd-specific blurb, so one table serves both
products. `sync-version.mjs` syncs and verifies the basta wrapper against the
basta crate version — the same independent version/pin update the jscpd block
does, for the same reason it does it.
Nothing is published by this commit.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Two defects found by checking every finding on the ten trending Python
repositories against an import graph rebuilt independently of the tool.
They accounted for 280 wrong findings out of 1,915; 4 remain, both in
cases where the checker is the generous one.
A quoted type annotation is a forward reference, not data. `def f(m:
"torch.nn.Conv1d")` reads `torch` exactly as surely as the unquoted form
does, and the `if TYPE_CHECKING:` idiom exists so that it *is* quoted —
the name may not be importable at run time. Without parsing them the
import that supplies the name looks unused, which is what made 275 of
834 unused-import findings wrong: 36% of the category, every one of them
reported at full confidence. The walk now tracks whether it is inside an
annotation and parses string literals found there with ruff's own
`parse_string_annotation`, which resolves in the source file's
coordinates so the references keep real offsets. Outside an annotation a
string stays what it was: weak evidence, never a read.
Absolute imports now also resolve against the roots the tree implies,
through a new optional `Analyzer::import_roots`. Python derives the
parent of every top-level package, which is what makes a `src/` layout
work from a file that belongs to no package: `utils/check_repo.py`
importing `transformers.testing_utils` matched no scan root and no
package of its own. Derived roots are tried last — a scan root is what
the user asked for, this is an inference.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Adds a second question about the same tree: not "what is duplicated" but
"what does nothing reach". A new `basta` crate builds the import graph from
the project's entry points and walks it, reporting unused files, unused
exports, unused module-private declarations, unused imports and — opt-in —
unused class members, across JavaScript, TypeScript, JSX, TSX and Python.
It ships two ways: `jscpd --dead-code` inside the existing binary, and a
standalone `basta` command. Both reuse the walker, the filters, all fifteen
reporters and the `--threshold` / `--exit-code` gates, so nothing new has to
be learned to run it.
Because it is a graph traversal rather than a reference count, dead code
cascades: a helper whose only caller is itself dead is reported too. Every
finding carries a confidence score and, below 100, the reasons it might be
wrong — `eval`, an unrecognised decorator, a wildcard re-export, a name seen
only in a string, a file in the scan that did not parse. `--min-confidence`
sets the floor at 60.
Entry points come from `package.json`, `pyproject.toml`, framework and
file-name conventions, and anything a shell script, CI workflow, Makefile or
Dockerfile names by path. CommonJS is read alongside ESM. A project that
renames its own import paths is read on its own terms: `paths` and `baseUrl`
from `tsconfig.json` and `jsconfig.json` are resolved, through a relative
`extends` chain and per package in a monorepo. A file named only by a path
string — a worker spawned through `new URL`, a build entry, a vitest setup
file — counts as used.
Languages plug in through one trait, `basta::lang::Analyzer`, which owns
parsing, specifier resolution, entry-point conventions, manifests, alias
configs and path traits. Nothing outside `lang/` is language-specific;
`docs/basta-extending.md` walks through adding one.
Validated against knip, fallow, vulture, skylos, fossil-mcp and repowise on
trending repositories. On a corpus of ten of them — 11k files, 3.3M lines —
every one of basta's 1,806 findings was checked against an import graph
rebuilt independently of it: no unused-file, unused-import or unused-symbol
claim survived as wrong.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
`--skip-local folder1 folder2` shows cross-folder clones only, so a
file found through a symlink inside folder1 belongs to folder1 and its
clones with other folder1 files are dropped. `--skip-isolated` also
compares walked ids; the canonical path stays as a fallback so a group
folder that is itself a symlink still matches the files found through
it.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
With `--follow-symlinks` the source id was the canonical path, so a file
reached through a link showed up as an absolute path outside the scan
root, `--ignore` (matched on the walked path) and the report disagreed
about its name, and a file reachable through two paths was two sources
(a file symlink next to its target even became a clone of itself).
The walked path, anchored at the canonical scan root, is now the id:
what reports show, what `--ignore` matches, what `--absolute` prints.
The canonical path travels alongside (`DiscoveredFile::real_path`,
`PreparedSource::real_path`) for reading the file and for the
`--skip-local` / `--skip-isolated` prefix checks. When links are
followed, files whose canonical path was already seen are dropped; the
entry that is the real file wins, else the first walked path.
Docs: `--follow-symlinks` row in the option table, the v4 default in the
migration table and README. Fixture `fixtures/follow-symlinks-demo`.
Closes#1059
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#375.
A snippet containing a NUL, an ANSI escape, a form feed or any other
control byte made the xml report unparseable: XML 1.0 has no way to
represent those characters, not even inside CDATA, so xmllint stopped at
"PCDATA invalid Char value 27" and every parser refused the file. The
reporter now replaces every character outside XML's Char production
(tab, LF, CR and U+0020 and up, minus U+FFFE/U+FFFF) with U+FFFD in
snippets and in path attributes. The existing "]]>" split is kept.
Path attributes were also escaped twice: quick-xml escapes attribute
values on its own, so the reporter's escape_xml() turned "r&d" into
"r&amp;d". The manual escape is gone and the sanitized path goes in
raw.
Tests parse the generated report to the end with quick-xml and check:
control characters become U+FFFD and none survive, a "]]>" round-trips
through the split sections, a path with & < " > is escaped exactly once
and reads back unchanged, and legal text is left borrowed.
Fixtures: fixtures/xml-report-demo with a clone whose body holds real
escape bytes, a form feed and a "]]>" literal; the README shows the
xmllint and Python checks. Verified the published 5.2.0 fails xmllint
on it and the fixed binary passes.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
The PyPI long description pointed the repo-hosted pre-commit hook at
"v5.3.0 or later"; the python hook ships with the next release, which
is 5.2.1. The PEP 440 examples use the same version.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
print_appendices gave console-full the history block it lacked; the
History section now names both console reporters and an integration
test checks console-full renders it.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Phase 3 of the DRY pass: report_result / report_to_file_timed fixtures
for the stdout and time-printing reporters, rule_ids and
first_result_properties in the sarif tests, cross_formats_with in the
cli tests, run_isolating in the skip-isolated finder tests, and a
tests/common module for the cpd crate holding the invoice/credit-note
snippets the CLI and MCP tests both used.
rust/crates: 12 clones / 0.4% left, all deliberate (fixture corpora,
use blocks, a TS file and its JS twin, impl headers, AST visitors).
971 workspace tests pass, clippy clean.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Phase 2 of the DRY pass. cpd/tests/integration.rs was 23% duplicated:
every reporter test rebuilt the same temp root, wrote the same greet
pair, ran cpd with the same assertion block and parsed the report by
hand. Now: GREET_DUP / greet_root / write_greet_pair, run_ok_in,
read_json, scan_json (json+silent scans returning the parsed report and
stderr), run_scratch (exit code + stderr for a scratch dir), config_dir
/ stderr_in_dir for config discovery, commit_dates for --history and
baseline_root_with_pair for the baseline suites. mcp_stdio.rs gains
spawn_mcp / drive and INITIALIZE / INITIALIZED constants; cpd-finder's
five integration tests share tests/common (temp_dir, duplicate_js,
fixtures); mcp.rs unit tests share scratch_dir.
rust/crates: 69 clones / 2.4% before the pass, 21 / 0.7% after.
971 workspace tests pass, clippy clean.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Phase 1 of the jscpd-guided DRY pass (69 clones, 2.4% in rust/crates):
- cpd-core: Location::new, Fragment::new/with_blame and CpdClone::exact
replace the 20-line struct literals repeated across the reporter tests
and fixtures.
- cpd-reporter: ReportContext::new instead of the copied struct literal
in every test; shared::fixtures::report_to_file replaces seven
near-identical run_<reporter>_report helpers; shared::print_appendices
prints the --summary and --history blocks for both console reporters.
- cpd-tokenizer: embedded::html_only_maps and tokenize_blocks_shifted
hold the code razor.rs and sfc.rs had copied from each other.
971 workspace tests pass, clippy clean.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
The one-line sparkline was too small to read a trend from. The console
block now draws an eight-row bar chart: a percentage axis on the left
spanning the series' own min and max (a series moving between 2.0% and
2.4% would be a flat line on a 0-based axis), one column per point built
from eighth-block characters so every point keeps at least a sliver, an
x-axis, and point numbers that match a new `#` column in the table.
Past 20 points the columns narrow to one character so 30 points still
fit an 80-column terminal. The ai reporter keeps the compact sparkline.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
CITATION.cff (CFF 1.2.0, validated with cffconvert) gives GitHub's
"Cite this repository" button BibTeX and APA output for the project.
The README gets a short Citation section with a ready-made @software
entry. sync-version.mjs keeps the file's version in step with
rust/package.json and stamps date-released whenever the version
changes, and verifies the version like the other manifests.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#1002 (the --history half; --ratchet is deliberately not
implemented, the report shows the headroom instead).
`jscpd src --history v5.0.0..HEAD` (or `--history-since 2026-01-01`)
lists the commits with `git log`, checks each one out into a temporary
detached worktree (the --baseline-from-ref machinery, now shared through
baseline_ref::map_scan_paths), scans it with the run's own configuration
and keeps the totals; the current run is the series' last point.
`--history-every N` keeps every Nth commit counted from the newest,
`--history-limit N` (default 30) samples a long series evenly keeping
both ends. Nothing runs unless one of the flags is given.
Output: the console reporter appends a sparkline of the duplication
percentage, a table (commit, date, files, lines, clones, dup lines,
dup%, change, subject) with the change column red when duplication rose
and green when it fell, the overall trend line, and, when --threshold is
set and the latest value is under it, the headroom with the value the
threshold could be tightened to. The json reporter adds a `history`
object; the ai reporter prints one compact line per point. Other
reporters are unchanged.
New cpd_core::history (model, sparkline, change/headroom helpers with
unit tests), cpd::history (git log parsing, sampling, worktree scans),
cpd_reporter::history_render. Config keys history / historySince /
historyEvery / historyLimit; action input `history`. Integration tests
build a four-commit repo with fixed dates and check the JSON series,
the console block, every/limit thinning and the error paths. 969
workspace tests pass, clippy clean.
Docs: History section and option rows in docs/rust.md, README bullet,
action input row, fixtures/history-demo/README.md with a script that
builds the demo repository and the expected output.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#1047.
Four situations used to write an empty report and exit 0, so a CI job
that checks only the exit code saw a green run:
- `--format` naming a format that is not supported (a warning since #964)
is now `Error: --format: 'x' is not a supported format`, exit 1. Names
introduced through --formats-exts or --formats-names stay accepted.
- A scan path that does not exist is `Error: path does not exist: <p>`,
exit 1, before anything runs.
- A reporter that fails to write its output still prints the existing
`Reporter '<name>' error: ...` line; the run now ends with
`ERROR: a reporter failed to write its output` and exit 1, after the
other reporters have run.
- A scan that analyzes no files (paths exist, but nothing matched the
--format/--ignore/--pattern filters or every file was below
--min-tokens) prints `Warning: jscpd analyzed no files` and still
exits 0, so an intentionally empty tree passes. The new
`--fail-on-empty` flag (config key `failOnEmpty`, action input
`fail-on-empty`) makes it exit 1. Reports are written before the check.
Docs: an "Exit codes" section in docs/rust.md, option rows in
docs/rust.md and docs/ci-and-hooks.md, a README feature bullet.
Fixtures: fixtures/fail-on-empty-demo with a `code/` clone pair, a
`nothing/` comment-only file, and a README quoting every command's
output and exit code. Tests: the #964 warning test becomes an error
test, plus integration tests for each new exit path and for the config
key; 531 tests pass, clippy clean.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
`--ignore-pattern` has no `-i` short flag (only `--ignore` does), and
ignore globs are matched against the whole file path, so bare directory
names such as `node_modules` in the config example did not exclude
anything (verified with 5.2.0: 3 of 4 files still scanned). Use the
`**/dir/**` form the other docs already use and say so in the table.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
The PyPI long description said 150+ formats and little else. It now
matches the README: 224 formats, 15 reporters, exact / Type-2 / Type-3
clones, baseline and threshold gating, the pip/pipx/uv install paths,
the Node-free pre-commit hook and the repo-hosted hook, plus links.
Metadata of a published release cannot change, so this lands with the
next upload.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
The `jscpd` hook in .pre-commit-hooks.yaml switches from `language: node`
to `language: python`: pre-commit now `pip install`s this repository,
whose new root pyproject.toml is an empty project that depends on the
`jscpd` wheel of the same version, so the hook needs no Node.js on the
machine. pre-commit is itself a Python tool, so Python is always there.
The root package.json existed only for the node hook and is removed;
sync-version.mjs now keeps pyproject.toml's version and `jscpd==` pin
in step with rust/package.json instead (mapped to PEP 440 the same way
build-pypi-wheels.py does) and verifies both before a release.
Verified with `pre-commit try-repo . jscpd` against locally built
5.2.0 wheels. Tags up to v5.2.0 keep the node hook, since a hook is
read from the pinned rev.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#1037.
Adds rust/scripts/build-pypi-wheels.py, which repacks the eight
jscpd-<platform>.tar.gz release tarballs into platform wheels: the same
binary under <pkg>.data/scripts/ as both `jscpd` and `cpd`, one wheel
per target, stdlib only, no maturin and no recompilation. glibc wheels
take their manylinux floor from the GLIBC_x.y symbols in the binary
(manylinux_2_34 today), musl builds are static so they tag as
musllinux_1_2, macOS and Windows use the Rust minimums. Semver
pre-releases map to PEP 440 (5.3.0-beta.1 -> 5.3.0b1); other labels
fail rather than guess.
release.yml gains a rust-publish-pypi job after the npm publish: it
takes the binary artifacts from this run, refills any missing target
from the published npm platform package (resumed releases, or a version
already on npm), builds the wheels, pip-installs the matching one and
checks `jscpd --version` and `cpd --version`, then uploads with PyPI
trusted publishing (environment `pypi`, no token, skip-existing). The
detect job treats a version missing from PyPI as pending, so the wheels
for the current release publish on the next push to master without a
new release commit. The GitHub release lists the PyPI package and waits
on the upload the same way it waits on crates.io and npm.
Docs: PyPI in the README install table and packages table, rust/README
and docs/rust.md install sections, and a `language: python` pre-commit
option in docs/ci-and-hooks.md so the hook no longer needs Node.js.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
An open clone was enlarged whenever the next window matched any stored
occurrence, without checking that the clone's anchor still matched. When
a third file had stored a window that continues the same text, the
anchored fragment was stretched past what the anchor file contains: the
clone was dropped by flush_clone when the anchor ran out of tokens, or
reported with mismatched ends when it did not (fixtures/haxe had such a
pair, [1:1 - 62:76] against [1:1 - 62:2]).
The match now asks first whether the anchor continues at the same offset;
if it does, the clone grows regardless of what the store holds for that
hash, and if it does not, the clone is flushed and a new one starts on the
occurrence the store matched. On the fixtures corpus this recovers 14
pairs among identical files and splits the misaligned haxe pair into two
token-identical ones; every *-demo directory's output is unchanged.
Regression test with the three-file shape from #1033 (a: X; b: X' T X'';
c: X T Z) asserting a↔c has exactly |X| tokens and every reported pair
holds identical token runs. Demo: fixtures/nway-demo/lost-pair.
Closes#1033.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
NO_COLOR asks for plain output and nothing else, and many people export
it globally while still working in an interactive terminal. Treating it
as a tips switch would hide the tips from them for good. The non-TTY
check at the print site already covers pipes, files and CI logs, so the
environment switches stay CI and JSCPD_NO_TIPS.
Help text, docs and the skill table updated; the options test asserts
that NO_COLOR does not enable no_tips.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Copilot review: gating on a <template> block let a script-only Vue
component keep <script>/<style> wrapper tags in its html stream. A Vue
file is blocks-only whatever blocks it has, so gate on the format instead.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Sorting the html tokens (previous commit) restores source order, but the
wrapper tags of a Vue file — <template>, </template>, <script ...>,
</script>, <style ...>, </style> — are identical in every Vue file and
sit at both ends of it. In source order they stretch every template clone
to the last </style>: on fixtures/vue the html clone went from
[1:1 - 10:9] to [1:1 - 262:10] and duplicated lines from 289 to 541,
counting the script and style bodies as duplicated html.
A file with a <template> block keeps all of its markup inside that block,
so the skeleton outside the blocks is only wrapper tags and is now left
out of the html stream. The clone becomes the template body itself:
file1.vue:html [2:3 - 10:9], 75 tokens. Svelte and Astro have no template
wrapper; their top-level markup is the skeleton and is kept, so their
output is unchanged. The sort stays, for custom blocks that map to html.
Fixture: fixtures/sfc-demo/template-range with a README showing the
template-only range; docs/rust.md links it.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Restore source-offset order after combining Vue SFC wrapper markup with template content so clone spans cannot jump across unrelated lines.
Add a regression test for the token-order invariant. Verified with the full Cargo workspace test suite, Clippy, formatting, and the malformed PR 3257 reproduction.
Public-output items from the #1020 review, done while nothing is released:
- SARIF and Code Climate rule names now match the clone vocabulary:
renamed -> jscpd/renamed-code, similar -> jscpd/similar-code
(was jscpd/similar-code for renamed and jscpd/near-miss-code for similar).
The similar-code rule text covers both the gap and the ast method.
- Duplicated-line statistics and the per-file summary count only the
matched lines of a merged clone. CpdClone gains a non-serialized
unmatched_lines field holding each fragment's gap lines, so enabling
the merge no longer moves --threshold.
- A merge whose similarity would fall below 0.5 (more unmatched than
matched tokens, e.g. one very long inserted line) is refused and the
halves stay separate; MIN_GAP_SIMILARITY documents the floor.
- Precedence is decided and documented: merged renamed halves are
reported as similar.
merge_gapped_clones is restructured around merge_into(); the running
matched-token counter was redundant with last.token_count and is gone.
Fixtures: type3-demo/long-line (refused merge) and type3-demo/renamed-halves
(precedence); README numbers re-measured for the whole directory.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Print the tips/sponsor lines only when stdout is a terminal, so CI logs,
pipes and agent hooks no longer get them without --no-tips. Also honor
NO_COLOR and JSCPD_NO_TIPS next to the existing CI env check.
Fixes#1008
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Each tool now states when to use it, what it returns (top-level keys and
per-item fields), its thresholds and limits, and that it is read-only,
with a title, MCP annotations (readOnlyHint, destructiveHint,
idempotentHint, openWorldHint), examples and defaults on parameters, and
additionalProperties: false where no input is taken. The server
instructions describe the intended workflow across the four tools. A unit
test guards title, annotations and parameter descriptions.
Motivation: Glama's quality score weighs purpose clarity, usage guidance,
behavioural transparency and parameter semantics per tool, plus
disambiguation across tools; agents read the same text.
`comment_style` names no arm for markdown, so markdown prose reaches the
`_ => CommentStyle::CStyle` fallback and takes the comments of C.
Markdown prose can hold `/*`: inside a glob such as `docs/**`, in a code span,
or in plain text. It opens a block comment that never closes, and every later
clone in that file goes unreported. The file still counts as analyzed and its
token total still looks healthy, so nothing signals the loss.
CommonMark defines no comment of its own. Its only comment is the HTML comment,
as HTML block type 2. https://spec.commonmark.org/0.31.2/
This adds the arm, and drops the `#[allow(dead_code)]` on `CommentStyle::None`,
which until now was constructed nowhere.
Measured with jscpd 5.1.2 on darwin-arm64. Two files that share a paragraph are
reported as a clone. Add one line holding `docs/**` above the paragraph in one
of them, and the clone is gone; a closing `*/` later in the file brings it
back. A `/*` inside a fenced block is harmless, because markdown.rs sends a
fence to the grammar of its own language.
`markup` (html, xml, svg) is absent from every arm too, so it takes the same
fallback. HTML does have `/* */` inside a `<style>` block, so that case needs a
different answer and is left alone here.
Verified on linux-x64 with rustc 1.94.1. The workspace pins 1.97.1 and its
crates could not be fetched there, so `cargo build`, `cargo nextest run
--workspace` and the new cpd-finder integration test were not run. What was
run:
- `cargo fmt --all --check` — clean.
- generic.rs compiled on its own against the three model types it uses: 12 of
12 unit tests pass. With the markdown arm deleted, the two new tests fail and
the other ten still pass.
- clippy over generic.rs with `-D warnings` — clean.
- On the fixture pair, `tokenize_generic(.., "markdown")` gives second.md 345
comment tokens and no words below the `docs/**` line without the arm, and 50
words with it — the same 50 that first.md carries.
- jscpd 5.1.2, linux-x64, on the same pair: no clone. Drop the line holding
`docs/**` and it reports one markdown clone, 4 lines and 54 tokens.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XacBaczr3GsFfEWWqGdz99
A JavaScript/TypeScript file with any parse diagnostic (a redeclaration,
a recoverable syntax error) was handed to the word-split fallback
tokenizer. Its tokens then never matched an oxc-tokenized file, so every
cross-file clone involving it vanished while its self-clones survived —
the symptom reported in #1023, where a second copy of a function in one
file made all a↔b clones disappear.
Tokens come from the lexer, not the parser: only `panicked` (or an empty
token stream) now triggers the fallback. Function extraction for
--similarity uses the partial AST on recoverable errors the same way.
Closes#1023.
`CpdClone.similarity_method` (JSON "method": "gap" | "ast") records which
near-miss mechanism produced a `similar` clone. The two scores are not on
one scale (gap = matched tokens over the merged span, ast = structural
Jaccard of whole functions), so the console prints
`Clone found (javascript, similar (gap) ~0.91)` /
`similar (ast) ~0.75`, the ai reporter `[~0.91 gap]` / `[~0.75 ast]`,
JSON carries "method" and SARIF a `similarity_method` property. Exact and
renamed clones are unchanged.
Function extraction is now a language plug-in: `FunctionExtractor`
(grammar id, served formats, extract) with a static `EXTRACTORS` registry
consulted by `extract_functions` / `supports_functions` /
`supported_function_formats`. `OxcExtractor` is the first entry (JS/TS/
JSX/TSX). Every RawFunction/FunctionSig records its grammar and the scorer
refuses pairs across grammars, so a tree-sitter-backed extractor for
another language is a self-contained addition with no change to scoring,
CLI, MCP or reporters. Module docs describe the two steps; the MCP note
lists supported formats from the registry.
Off by default. `--similarity RATIO` (config `similarity`, Action input
`similarity`, MCP `check_duplication` argument) summarizes every function,
method and arrow function of a JavaScript/TypeScript/JSX/TSX file by the
bag of 4-grams over the pre-order sequence of its AST node types (oxc),
indexes the bags with MinHash + LSH banding, and reports pairs whose
weighted Jaccard index reaches RATIO as `similar` clones with a
`similarity` value. Names and literal values are not part of the summary:
a renamed copy scores 1.0, one inserted line ~0.9, two inserted statements
plus renames ~0.75.
Functions must clear --min-tokens/--min-lines, nested functions are never
paired with their parent, and pairs already covered by an exact, renamed
or merged clone are skipped. Values outside (0, 1] warn and disable the
option. Reporting reuses the Type-3 vocabulary (console `similar ~0.75`,
JSON kind/similarity, SARIF jscpd/near-miss-code).
New: cpd-core::similarity (FunctionSig, SimilarityIndex, bag_jaccard,
minhash), cpd-tokenizer::functions (oxc visitor extractor), MCP function
index built lazily per scan. fixtures/type3-demo/similar-functions with
README numbers, docs, unit + integration + MCP stdio tests. Also adds the
missing max_gap_lines/similarity fields to the --debug config output.
Off by default (0). When set to N, clones of the same file pair whose
fragments follow each other in both files with at most N unmatched lines
between them are merged into one clone of kind `similar`, with `tokens` =
matched tokens and `similarity` = matched tokens over the tokens of the
longer merged span. Adjacent exact windows that share a boundary token are
counted once; chains merge transitively; overlapping or reordered fragments
never merge. The pass runs after every other filter and returns its input
untouched at 0, so default output is unchanged.
Reporting: console `Clone found (fmt, similar ~0.91)`, ai `[~0.91]`, JSON
"kind"/"similarity" (rounded to three decimals), SARIF rule
jscpd/near-miss-code with a similarity property (declared only when
referenced), Code Climate check_name.
Config key maxGapLines (kebab alias), GitHub Action input max-gap-lines,
docs section, fixtures/type3-demo with a README of runnable commands, unit
tests for the merge pass and reporters, and a binary-level integration test.
apk