Single-file component support (Vue/Svelte/Astro), bundler alias
resolution (vite.config, svelte.config, tsconfig paths) and monorepo
workspace package-name resolution landed on master after basta 0.1.1
was tagged (#1062); the published crate and npm packages don't have
them yet.
- Console tables (project/duplication-by-format/complexity/dead-code) and
the "largest:" summary line interpolated file paths and format names
raw: the Markdown escaping fix in the previous commit only covered
Markdown, not console. A newline in a path or a custom --formats-exts
name forged an extra table row with none of the other columns' data.
Sanitized at table()'s one choke point and at the "largest:" line.
- format_list() (the --health console/ai duplication line) had the same
gap for the formats it lists, independent of the console table fix.
- --complexity --fail-on-empty returned before the reporter loop ran, so
-r json never wrote a report; the clone-detection and dead-code paths
write reports first and apply the gate to the exit code afterward.
--complexity now does the same.
- dead_code::config()'s early Ok(None) for an unsupported --format
returned before parsing --dead-code-categories or clamping
--min-confidence, so --dashboard --format css --dead-code-categories
bogus silently succeeded despite the comment above it saying that must
stay a refusal. Moved both checks before the format check.
8 new tests.
Two independent reviews of the health/dashboard work landing since v5.2.1,
before cutting 5.3.0.
Scoring:
- Markup lines stayed in the duplication denominator while excluded from
the numerator, so adding one unique, un-duplicated HTML file could swing
a project from grade D to grade B. The denominator now excludes markup
too, and an all-markup project skips the dimension entirely instead of
scoring ~75 from the small-project prior alone.
- is_markup() matched file extensions (plantuml, puml, tpl) instead of the
tokenizer's actual format names (plant-uml, smarty), so those formats
were never really excluded.
- A health-input weight of 1e308 overflowed the weighted geometric mean to
NaN/100 with exit 0; validate() now rejects non-finite or unreasonably
large weights.
- External metric ids could duplicate each other or collide with a
built-in dimension id.
Output safety:
- Markdown output (dimension ids, format names, file paths, dead-code
finding names) was interpolated unescaped: a hostile path or
--health-input id could break a table or inject raw HTML/a heading
(GitHub renders HTML embedded in Markdown), or split console/ai output
via an embedded newline.
CLI:
- A config file with a known field of the wrong JSON type (e.g. "entry":
"src/index.js" where an array is expected) discarded the entire config,
--threshold included, instead of just that field — on any
auto-discovered .jscpd.json, no new flags required.
- --dashboard --format css hard-errored instead of dropping the dead-code
section.
- --dead-code/--complexity didn't conflict with --mcp.
- --complexity silently ignored --threshold, --exit-code,
--fail-on-empty, the baseline family, --history and --kind.
- --dead-code --min-confidence 200 silently reported nothing instead of
clamping to 100, unlike the standalone basta binary.
- --dashboard --max-lines let the dead-code section silently cover files
the other sections skipped (basta needs whole files for its import
graph).
- duplication.percentage/deadCode.percentage in dashboard JSON carried raw
float division noise instead of the two decimals the console already
shows.
26 new tests. Full default-path parity re-verified against the released
5.2.1 binary (13 reporters, option matrix) — byte-identical except timing.
Dependabot opened five separate PRs bumping oxc_syntax, oxc_parser,
oxc_allocator, oxc_semantic and oxc_ast individually (#1066-#1070), but the
oxc crates share AST/Visitor trait definitions across the family and only
compile against matching versions of each other — bumping one alone breaks
the build (Visit trait signature mismatches, missing .span() methods) on
every CI job. oxc_ast_visit and oxc_span also needed the same bump, but
dependabot had not opened PRs for those two.
Bumps all seven together instead, and fixes the one real API change
between 0.147 and 0.150: ParserReturn::panicked was renamed to
::fatal_error (same meaning, confirmed against oxc_parser's source).
cpd-tokenizer's own oxc_parser/oxc_ast pins stay at 0.147, coexisting
fine as a separate resolution in the same Cargo.lock (no basta/
cpd-tokenizer API overlap), so this stays scoped to basta.
display_source_path() returned OS-native separators via to_string_lossy(),
so a summary path (dashboard's complexity/duplication sections) read
"src\route.js" on Windows instead of "src/route.js" everywhere else — the
convention this codebase already follows in mcp.rs, ai.rs and
summary.rs's parent_dir(). Fixes the Windows CI failure in
dashboard_has_the_health_badge_and_a_json_report, pre-existing before
this branch's dashboard/health work.
- Console health badge: reverse-video grade chip, eighth-block gauges per
dimension, one row per sub-score with what it was measured from.
- markdown/html reporters for --dashboard and --health, alongside the
existing console/json/badge; the health SVG badge moves into badge.rs.
- --dashboard's Duplication section lists a breakdown by format instead of
by file, sourced from the same jscpd statistics as everywhere else.
- Duplication no longer counts markup, stylesheet and template clones
(HTML, CSS, Handlebars, ...): a repeating style rule is not the
maintenance problem repeating programming logic is. The exclusion
follows the clone's own format, not the file it lives in, so a .svelte
or .vue file's style/template block is left out even though the
component itself is full weight.
- The duplication line and format-breakdown table now name what they
actually measured and, only for categories the project has files in,
what was left out (e.g. "5.4% in typescript (no text)") instead of a
fixed disclaimer.
One 0-100 score with an A-E grade, shown on top of --dashboard and alone
with --health.
- Dimensions are shares of the code lines, so size cancels out:
duplication (jscpd's percentage over code files), dead code, and the
share of code in complex files (complexity 50+). Prose and data files
are not counted.
- Each share is scored on a half-life curve (100 at zero, 50 at one
half-life), calibrated on 42 open-source projects so the median project
scores 75 per dimension; shares are mixed with a 2000-line prior so one
finding does not sink a small project.
- Sub-scores combine as a weighted geometric mean. A dimension that
cannot be measured is left out and named, and dead code weighs as much
as the share of the code it could read.
- --health-input FILE and the health config object add metrics from
other tools (a ready score, or a value with a half-life) and tune the
built-in half-lives and weights.
- Reporters: console badge, ai one-liner, json (jscpd-health.json,
jscpd-dashboard.json with every dashboard section) and badge
(jscpd-health-badge.svg). Exit gates apply as in the dashboard.
CpdClone::matched_lines moves to cpd-core so statistics and health count
duplicated lines the same way.
- --dashboard applies the exit gates of a clone run (--threshold,
--exit-code, --fail-on-empty) instead of always returning 0; the
baseline family warns and --fail-on-new-clones is refused, since no
clone report is written.
- A refused dead-code configuration (bad --dead-code-categories, a
--format no analyzer supports) now exits 1 as in --dead-code, rather
than printing a dashboard with an empty dead-code section.
- --workers is a budget for the whole dashboard run: the two concurrent
scans get half each, so neither the thread count nor the 64 MiB
tokenizer stacks double. --workers 1 runs them sequentially.
- --complexity scans with similarity disabled, so --similarity no longer
pays for function extraction it cannot use.
- --complexity honours a summaryBy from the config file, not only the
--summary-by flag (Options::summary_by_set).
- reStructuredText joins the prose formats at complexity 0.
`comment_style` names no arm for txt, log or csv, so they fall to the
`_ => CommentStyle::CStyle` fallback, the same way markdown did before
11553327. A glob such as `docs/**` in a text file opens a block comment
that never closes, and every later clone in that file goes unreported;
`https://` turns the rest of its line into a comment.
With jscpd 5.2.1, two .txt files sharing a paragraph report no clone once
one of them holds `--ignore docs/**` above it (412 tokens, 0 clones);
with this change the clone is reported (130 tokens, 1 clone).
--dashboard prints one screen: project size and largest formats,
duplication with clones per kind and the most duplicated files, total and
mean complexity with the most complex files, and dead code by category
with the largest findings (JavaScript, TypeScript, Python). The dead-code
scan runs beside clone detection; --summary-top sets the rows per list.
Data formats (JSON, YAML, TOML, CSV, ...) join markdown at complexity 0,
so a lock file's version ranges no longer top the complexity ranking.
Docs for --kind, --complexity and --dashboard; fixtures/dashboard-demo,
plus --kind and --complexity sections in the type3 and summary demos.
--kind exact,renamed,similar,gap,ast keeps only the clones of those kinds
(config key "kind"). The filter runs before statistics, so percentages
describe what is reported. An unknown kind is an error; a kind whose
detector is off (ast without --similarity, ...) is a warning, and
detection is never switched on implicitly.
--complexity walks and tokenizes like a normal run, skips clone
detection, and prints the --summary tables ranked by complexity without
the DUP% column (console, ai; json writes jscpd-complexity.json).
Markdown prose now has complexity 0: "if" in a README is a word.
Folding the symlink test assertions into a helper put it between
#[cfg(unix)] and symlink_layout, so the attribute moved to the helper and
std::os::unix::fs::symlink no longer compiled on Windows.
b72abf09 also searched the directory of each scanned path. fixtures/ has
a .jscpd.json with "silent": true, so `cd rust && jscpd ../fixtures -r ai`
started printing only the summary line instead of the clones that
`npx jscpd` lists. Config discovery is back to the working directory;
output now matches jscpd 5.2.1 across all reporters.
- FinderError was never constructed; run() cannot fail, so its error type
is Infallible and the documented run(&config).unwrap() keeps compiling.
- tokenize_sfc_block_inner and tokenize_to_detection_inner forwarded to
tokenize_format_to_detection; call it directly.
- extract_line_number was Some(error.line()).
- Timer wrapped Instant; use Instant.
serde, anyhow, thiserror and log from cpd-finder; thiserror from
cpd-reporter; xxhash-rust and dev serde_json from cpd-tokenizer; anyhow,
log and env_logger from cpd. None of them is referenced in code.
`tokenize_line_content`, which cuts every line of every non-JavaScript file
into tokens, was a 218-line loop with a cyclomatic complexity of 49. Each of
its nine token classes repeated the same index, column and byte-offset
bookkeeping and its own ignore-region check.
A `LineCursor` now owns the position, the column and the ignore-region kind,
and each class is one call on it: `take`, `take_while`, `take_string`,
`take_word` or `take_rest`. The loop that dispatches between them is 37 lines
with a complexity of 15. The separate Lua `--[[` branch goes: the line comment
check right after it already matched Lua's `--` and produced the same token.
`comment_kind` is folded into `LineCursor::kind`.
Checked against the previous implementation, copied beside the new one for a
one-off differential test: every text file in `fixtures/` and `rust/` — 839
of them — under all seven comment styles, plus edge cases for unterminated
and escaped strings, block comments across lines, ignore regions, and
multi-byte and non-breaking-space input, produced 5,039,506 tokens identical
in kind, text, line, column and offset. On ten trending projects (7,181 files,
11.2 million tokens) the JSON report is byte-identical, and fifteen
alternating single-worker runs put old and new within noise of each other
(median 4.04 s old, 4.01 s new).
`main` was the most complex function in the workspace: one 437-line body
with a cyclomatic complexity of 84 and thirteen `process::exit` calls,
running config loading, flag validation, detection, baselines, blame,
history, reporter planning, output and exit codes in sequence.
It is now a short dispatch over functions that each do one of those
things: `load_options`, `check_format_names`, `validate_baseline_flags`,
`scan_paths`, `run_config`, `detect_and_report`, `display_paths`,
`apply_baseline`, `blame`, `history`, `plan_reporters`, `run_reporters`,
`print_time_and_tips` and `exit_status`. A run that ends early returns
`Err(Exit(code))`, and only `main` exits the process. The largest function
left has a complexity of 15.
Reporter planning is a pure function now, and has tests. One of them pins
the rule that took care to carry over: a `--reporters silent` run prints no
timing line, but adding `--threshold` adds a reporter that prints.
Behaviour is unchanged. The previous and the new binary agree on stdout,
stderr, exit code and written report across 28 invocations covering --list,
--debug, every warning and validation error, a missing path and config,
--silent and --threshold in combination, unknown reporters, --exit-code
with and without clones, empty scans with and without --fail-on-empty,
--absolute, --summary, --baseline with --update-baseline, and --blame; and
the fixtures corpus reports in every mode are byte-identical.
`jscpd rust/` reported 48 clones. This folds every one that was repetition
rather than a deliberate copy, leaving 34 across `rust/` and 14 in
`rust/crates`, all of the kinds the #1053 pass kept on purpose: fixture
corpora, `use` blocks, the TypeScript file and its JavaScript twin,
`impl Reporter` headers and the `ts_strip` visitor pair.
- basta's JSONC reader copied a string-skipping loop into both `strip_jsonc`
and `drop_trailing_commas`. It is now `strip_comments` followed by a comma
pass over `lang::outside_strings`, the quote-aware scanning the bundler
config readers already share.
- The JavaScript and Python analyzer tests each built their own module
index, import lookup, binding-reference lookup and member-read list. They
live once in `test_scan` as `index_of`, `import`, `binding_reference` and
`member_reads`.
- `test_scan::TempTree` writes a test's files and removes them when dropped,
so a failing assertion no longer leaves its directory behind; the script
and `extends` tests that repeated the setup use it.
- Four tsconfig `paths` tests with one shape are one table-driven test.
- The CLI integration tests share `duplicate_shapes`, `history_stdout` and
`assert_only_the_real_file_was_scanned`.
- `scripts/publish-npm.sh` repeated a 43-line "publish the platform package,
then the main packages under --force-main" tail in two branches, and the
"publish unless already live, or show the dry run" block five times. They
are `publish_platform_target` and `publish_package`. Run against the old
script with the build, the registry check and `npm` stubbed, all eleven
combinations of --target, --all, --force-main, --dry-run, --provenance, a
failed build and missing platforms print the same lines and exit the same.
Behaviour is unchanged: the fixtures corpus reports in every mode, the
summary, the console output and the dead-code demo are byte-identical.
basta reads JavaScript, TypeScript and Python, so `jscpd --dead-code` has no
view of the Rust crates, and the compiler does not warn about an unused `pub`
item. Every crate in the workspace is `publish = false`, though, so a `pub`
item nothing in the workspace uses is dead as surely as a private one. A scan
of every `pub` item's uses across the crates, their tests and the fuzz
targets, checked by hand, found:
- `CloneKind::is_exact`: used nowhere.
- `ConfigResult::has_diagnostics`, `basta::lang::language_of`,
`TokenizeOptions::normalizes` and `TokenizeOptions::with_code_ignore_patterns`:
called only by tests written for them. The language test now asks
`analyzer_for` directly.
- `stats_with_pct` is a reporter test fixture that was compiled into the
library; it moves inside the `#[cfg(test)]` fixtures module that re-exported it.
- Fields kept alive by `#[allow(dead_code)]`: `MarkdownFence::front_matter`
and `SfcBlock::tag` are written and never read, and `Options::list` is never
read because `main` reads `cli.list`.
- `scripts/regression_check.sh` and its 1528-line `tests/golden.json`: a gate
for the Wave 2/3 migration that nothing runs, against a report schema that
has since changed.
Behaviour is unchanged: the JSON reports for the fixtures corpus in mild, weak
and strict modes, with Type-2 normalization, and with the complexity summary,
the console output and the dead-code demo output are byte-identical before and
after.
Copilot's review of #1062 raised seven points; each checked out and each is
fixed with a test that fails without it.
- A glob keeps its pattern. `import.meta.glob('./locales/*.js')` and
import(`./pages/${name}.vue`) used to keep every file under the directory
alive; they now reach what the pattern matches, `*` staying within one
segment as it does for the bundler. On Firefly an image glob,
'../../**/*.{png,jpg}', had been keeping all of src/ alive.
- Every pattern of an `import.meta.glob([...])` array is kept, not the first.
- A declared alias is not second-guessed: when `@` points at src/ and the file
is missing there, the `@/` project-root fallback no longer finds a
different file. A catch-all `*` path claims no prefix and does not count.
- Astro client <script> blocks are read as the modules Astro bundles them
into, so what they import is used. A script with any attribute but `src` is
left to the browser, as Astro leaves it.
- import('./x') in markup is found only inside expressions, not in comments
or plain text.
- A quoted attribute value is an expression only on a directive (`:prop`,
`@click`, `#slot`, `v-*`); elsewhere only its `{...}` interpolations are,
so `class="card"` no longer credits a declaration named `card`.
- An object key in markup, `{ label: value }`, is not a read.
Re-measuring on the thirty trending projects also exposed `?raw` imports, as
in `import script from './panzoom.js?raw'`: a bundler query now is stripped
before resolving. On one snapshot of those projects the previous release
reports 1093 unused files and this branch 481, with the share some import
names falling from 83% to 62%; the CHANGELOG now quotes those numbers
instead of ones measured against a development baseline. Components used
only from .mdx content are still reported, and the docs say so.
The CX column was one path per file plus every branch keyword the token
stream showed, and on the generic tokenizer that list was partly
unreachable: it splits punctuation into single characters, so the && and ||
entries never matched outside JavaScript and ?? counted twice. Measured
against lizard on ten GitHub-trending projects in nine languages, CX ranked
files with a median Spearman correlation of 0.83 and put their size at 0.75
of lizard's.
- Adjacent punctuation is joined before matching, so short-circuit
operators count in every language.
- Per-language rules: Rust match arms count by their =>, less one per match
since N arms are N paths; Swift adds guard and Go select. Where ? also
marks an optional (TypeScript, JavaScript, Swift, Kotlin, Groovy, C#) it
counts only as a ternary, told apart from String? and x?.y by being
written in the open; ?: is Elvis only in Kotlin and Groovy.
- One path per function: by declaring keyword (def, fn, func, function, fun,
and => for arrow functions), and in C, C++, Java, Objective-C and C# by the
head(args) { shape told apart from if (...) {. Languages with neither keep
the per-file baseline.
- The body of a """ or ''' string no longer counts. The line-based tokenizer
leaves its quotes in adjacent literals, and an odd number of triple quotes
in such a run opens or closes the string, which handles the PEP 257
opener """Summary. that a pair-of-tokens check read backwards. This is
limited to languages where """ is a delimiter: where a doubled quote is an
escape (C#, VB.NET, SQL, Pascal) it swallowed a 413-branch C# file.
- A keyword-shaped word the file binds as a name - case = 3, x.case,
f(case, when) - is not a branch in that file.
On the same ten projects the median correlation rose to 0.92 and the ratio
to 1.00, and every project improved. Where CX and lizard still differ they
differ on purpose: lizard counts a match once whatever its arms, and does
not count ??.
Detection, tokenization and clone output are untouched: the fixtures corpus
reports the same 248 clones. fixtures/summary-demo has one file per language
whose complexity can be counted by hand, and every CX value matches it.
basta reported most of a real Vue, Svelte or Astro project as dead: on
thirty GitHub-trending projects in those three ecosystems it flagged 1572
unused files, and some import in the tree contradicted 78% of them. The
misses were almost never in the analysis itself; they were imports only the
project's own build knows how to resolve. This reads them the way the build
does.
- Single-file components are read whole. The script of a .vue, .svelte or
.astro file is masked in place so positions stay real, and the markup is
scanned for the names it reads: <Foo />, kebab-case spellings, {{ }}
interpolations, Svelte actions, attribute expressions, and a literal
import() written in markup.
- Aliases come from vite.config.* resolve.alias and svelte.config.* kit.alias
as well as tsconfig paths, and SvelteKit's $lib is known without the
generated .svelte-kit/tsconfig.json no repository commits. The project's
own declaration now wins over the ~/ @/ project-root fallback.
- A computed import whose static head names a directory,
import(`./pages/${name}.vue`), and import.meta.glob('./locales/*.js') are
expanded into one namespace edge per file there, the way a bundler hands
back whole module objects. Such an import no longer marks its module as
dynamic access, so the module's own findings keep their confidence.
- In a monorepo a package name resolves through that package's package.json,
exports subpath by subpath with source conditions preferred over ./dist,
anywhere in the workspace pnpm-workspace.yaml, lerna.json, turbo.json or a
workspaces field marks out.
- A path literal in a build config resolves against the project root, and
Nitro's api/, routes/ and middleware/ are file-system routes.
On the same thirty projects reported unused files fell from 1572 to 468.
fixtures/dead-code-demo gains components/, bundler/ and monorepo/. On master
the bundler demo reports five files as certainly dead, four of them wrongly;
the monorepo demo reports badge.ts, which the app imports by package name.
Both now report only what is dead. The quote-aware byte scanning the markup
and config readers share lives in one place, lang::Quotes and
lang::outside_strings, so rust/crates keeps master's 25 clones.
Two failures on the branch, neither on macOS, which is why they surfaced
only in CI.
`cargo deny` rejected `unicode_names2`, which `ruff_python_parser` pulls in
and which carries `Unicode-DFS-2016` alongside its MIT/Apache dual licence.
That licence covers the character tables and is the older wording of
`Unicode-3.0`, already on the allow list; added beside it.
The two new config-discovery tests asserted on `../project/.jscpd.json`,
and Windows prints the separator it joined with, so both failed there.
They now normalise backslashes before comparing — the assertion is about
which file was read, not about how the path is spelled.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
`cargo publish` verifies by building the packaged crate against the
*published* versions of its dependencies, not the local paths. basta uses
`cpd_core::deadcode`, `cpd_reporter::deadcode` and a public `LineIndex` —
none of which exist in the versions on crates.io — so every attempt failed
with "failed to verify package tarball", and basta could not reach crates.io
until a jscpd release happened to carry those three crates along.
That is the coupling this workflow exists to avoid. It now walks basta's own
dependency graph, publishes in topological order whatever is missing, and
waits for the index between crates. jscpd is excluded by construction: it is
a sibling, not a dependency, and releasing it is not this workflow's job.
Bumps the three crates that actually changed: cpd-core 0.1.13 -> 0.1.14
(the `deadcode` module), cpd-tokenizer 0.1.15 -> 0.1.16 (`LineIndex` made
public, `.pyi`), cpd-reporter 0.1.14 -> 0.1.15 (the dead-code reporters).
cpd-finder is untouched and stays where it is.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
`basta@0.1.0` published with `files: ["run-basta.js", "platform-map.js"]`,
so npmjs.com/package/basta shows an empty page — the one place most people
will look before installing anything. npm takes the README from the
published tarball, so a version bump is the only way to put one there.
The README is npm-facing rather than a copy of the crate's: it leads with
`npm install -g basta` and `npx basta src`, and it names the known gaps —
no Vue/Svelte/Astro, class members matched by name, PEP 420 namespace
packages — so a reader can judge whether the tool fits before running it.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
`cd rust && jscpd --basta ..` silently ignored the repository's own
`.jscpd.json`: config discovery only ever tried `.jscpd.json`,
`.config/jscpd.json` and `package.json` relative to the working
directory. The configuration is a property of the code being scanned,
not of where the shell happens to be, so a scan aimed at a project read
none of its ignores, formats or thresholds.
The cwd is still searched first and still wins, so nothing about an
existing invocation changes; each scanned path's directory is searched
after it, and a path that is also the cwd is not searched twice. A file
argument contributes the directory holding it. `ConfigSource` now
carries the path it was read from, so the "Using config from" line names
the file that was actually used rather than assuming it was the local one.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
basta is a separate product from jscpd — a different question about the
same tree — so it gets its own version, its own prebuilt packages and its
own trigger. jscpd goes out on every push to master that leaves a version
unpublished; basta goes out only when a `basta-v*` tag is pushed, or by
hand through workflow_dispatch. Neither release can drag the other along
or hold it up.
`basta-release.yml` owns the whole thing: it checks that the crate, the npm
manifest and the tag all name the same version and that the working tree is
already synced, publishes the crate, builds the eight targets and publishes
`basta-<platform>` plus the `basta` wrapper. It ends by asserting that npm's
`latest` really points at this release — the package carries 1.x versions
from an unrelated tool of the same name, and a dist-tag that quietly stayed
on those would hand every `npm i basta` the wrong program.
The one place the two cannot be separated is crates.io: `jscpd` depends on
`basta`, so cargo refuses to publish jscpd until a matching basta exists.
The jscpd crates job therefore still publishes basta when it is missing, and
that is now commented as the dependency requirement it is. Both paths skip a
version already on the registry, so whichever runs first wins.
npm needed the wrapper pattern `jscpd` and `cpd` already use: `rust/basta/`
holds the manifest, a platform map naming `basta-<platform>`, and a launcher
that falls back to a local `cargo build` when no platform package is
installed. `npm-prebuilt-package.mjs` grew a `--product` flag; its output for
jscpd is byte-identical to before. `prebuilt-targets.json` now carries a
platform label rather than a jscpd-specific blurb, so one table serves both
products. `sync-version.mjs` syncs and verifies the basta wrapper against the
basta crate version — the same independent version/pin update the jscpd block
does, for the same reason it does it.
Nothing is published by this commit.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Two defects found by checking every finding on the ten trending Python
repositories against an import graph rebuilt independently of the tool.
They accounted for 280 wrong findings out of 1,915; 4 remain, both in
cases where the checker is the generous one.
A quoted type annotation is a forward reference, not data. `def f(m:
"torch.nn.Conv1d")` reads `torch` exactly as surely as the unquoted form
does, and the `if TYPE_CHECKING:` idiom exists so that it *is* quoted —
the name may not be importable at run time. Without parsing them the
import that supplies the name looks unused, which is what made 275 of
834 unused-import findings wrong: 36% of the category, every one of them
reported at full confidence. The walk now tracks whether it is inside an
annotation and parses string literals found there with ruff's own
`parse_string_annotation`, which resolves in the source file's
coordinates so the references keep real offsets. Outside an annotation a
string stays what it was: weak evidence, never a read.
Absolute imports now also resolve against the roots the tree implies,
through a new optional `Analyzer::import_roots`. Python derives the
parent of every top-level package, which is what makes a `src/` layout
work from a file that belongs to no package: `utils/check_repo.py`
importing `transformers.testing_utils` matched no scan root and no
package of its own. Derived roots are tried last — a scan root is what
the user asked for, this is an inference.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Adds a second question about the same tree: not "what is duplicated" but
"what does nothing reach". A new `basta` crate builds the import graph from
the project's entry points and walks it, reporting unused files, unused
exports, unused module-private declarations, unused imports and — opt-in —
unused class members, across JavaScript, TypeScript, JSX, TSX and Python.
It ships two ways: `jscpd --dead-code` inside the existing binary, and a
standalone `basta` command. Both reuse the walker, the filters, all fifteen
reporters and the `--threshold` / `--exit-code` gates, so nothing new has to
be learned to run it.
Because it is a graph traversal rather than a reference count, dead code
cascades: a helper whose only caller is itself dead is reported too. Every
finding carries a confidence score and, below 100, the reasons it might be
wrong — `eval`, an unrecognised decorator, a wildcard re-export, a name seen
only in a string, a file in the scan that did not parse. `--min-confidence`
sets the floor at 60.
Entry points come from `package.json`, `pyproject.toml`, framework and
file-name conventions, and anything a shell script, CI workflow, Makefile or
Dockerfile names by path. CommonJS is read alongside ESM. A project that
renames its own import paths is read on its own terms: `paths` and `baseUrl`
from `tsconfig.json` and `jsconfig.json` are resolved, through a relative
`extends` chain and per package in a monorepo. A file named only by a path
string — a worker spawned through `new URL`, a build entry, a vitest setup
file — counts as used.
Languages plug in through one trait, `basta::lang::Analyzer`, which owns
parsing, specifier resolution, entry-point conventions, manifests, alias
configs and path traits. Nothing outside `lang/` is language-specific;
`docs/basta-extending.md` walks through adding one.
Validated against knip, fallow, vulture, skylos, fossil-mcp and repowise on
trending repositories. On a corpus of ten of them — 11k files, 3.3M lines —
every one of basta's 1,806 findings was checked against an import graph
rebuilt independently of it: no unused-file, unused-import or unused-symbol
claim survived as wrong.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
`--skip-local folder1 folder2` shows cross-folder clones only, so a
file found through a symlink inside folder1 belongs to folder1 and its
clones with other folder1 files are dropped. `--skip-isolated` also
compares walked ids; the canonical path stays as a fallback so a group
folder that is itself a symlink still matches the files found through
it.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
With `--follow-symlinks` the source id was the canonical path, so a file
reached through a link showed up as an absolute path outside the scan
root, `--ignore` (matched on the walked path) and the report disagreed
about its name, and a file reachable through two paths was two sources
(a file symlink next to its target even became a clone of itself).
The walked path, anchored at the canonical scan root, is now the id:
what reports show, what `--ignore` matches, what `--absolute` prints.
The canonical path travels alongside (`DiscoveredFile::real_path`,
`PreparedSource::real_path`) for reading the file and for the
`--skip-local` / `--skip-isolated` prefix checks. When links are
followed, files whose canonical path was already seen are dropped; the
entry that is the real file wins, else the first walked path.
Docs: `--follow-symlinks` row in the option table, the v4 default in the
migration table and README. Fixture `fixtures/follow-symlinks-demo`.
Closes#1059
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#375.
A snippet containing a NUL, an ANSI escape, a form feed or any other
control byte made the xml report unparseable: XML 1.0 has no way to
represent those characters, not even inside CDATA, so xmllint stopped at
"PCDATA invalid Char value 27" and every parser refused the file. The
reporter now replaces every character outside XML's Char production
(tab, LF, CR and U+0020 and up, minus U+FFFE/U+FFFF) with U+FFFD in
snippets and in path attributes. The existing "]]>" split is kept.
Path attributes were also escaped twice: quick-xml escapes attribute
values on its own, so the reporter's escape_xml() turned "r&d" into
"r&amp;d". The manual escape is gone and the sanitized path goes in
raw.
Tests parse the generated report to the end with quick-xml and check:
control characters become U+FFFD and none survive, a "]]>" round-trips
through the split sections, a path with & < " > is escaped exactly once
and reads back unchanged, and legal text is left borrowed.
Fixtures: fixtures/xml-report-demo with a clone whose body holds real
escape bytes, a form feed and a "]]>" literal; the README shows the
xmllint and Python checks. Verified the published 5.2.0 fails xmllint
on it and the fixed binary passes.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
The PyPI long description pointed the repo-hosted pre-commit hook at
"v5.3.0 or later"; the python hook ships with the next release, which
is 5.2.1. The PEP 440 examples use the same version.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
print_appendices gave console-full the history block it lacked; the
History section now names both console reporters and an integration
test checks console-full renders it.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Phase 3 of the DRY pass: report_result / report_to_file_timed fixtures
for the stdout and time-printing reporters, rule_ids and
first_result_properties in the sarif tests, cross_formats_with in the
cli tests, run_isolating in the skip-isolated finder tests, and a
tests/common module for the cpd crate holding the invoice/credit-note
snippets the CLI and MCP tests both used.
rust/crates: 12 clones / 0.4% left, all deliberate (fixture corpora,
use blocks, a TS file and its JS twin, impl headers, AST visitors).
971 workspace tests pass, clippy clean.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Phase 2 of the DRY pass. cpd/tests/integration.rs was 23% duplicated:
every reporter test rebuilt the same temp root, wrote the same greet
pair, ran cpd with the same assertion block and parsed the report by
hand. Now: GREET_DUP / greet_root / write_greet_pair, run_ok_in,
read_json, scan_json (json+silent scans returning the parsed report and
stderr), run_scratch (exit code + stderr for a scratch dir), config_dir
/ stderr_in_dir for config discovery, commit_dates for --history and
baseline_root_with_pair for the baseline suites. mcp_stdio.rs gains
spawn_mcp / drive and INITIALIZE / INITIALIZED constants; cpd-finder's
five integration tests share tests/common (temp_dir, duplicate_js,
fixtures); mcp.rs unit tests share scratch_dir.
rust/crates: 69 clones / 2.4% before the pass, 21 / 0.7% after.
971 workspace tests pass, clippy clean.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Phase 1 of the jscpd-guided DRY pass (69 clones, 2.4% in rust/crates):
- cpd-core: Location::new, Fragment::new/with_blame and CpdClone::exact
replace the 20-line struct literals repeated across the reporter tests
and fixtures.
- cpd-reporter: ReportContext::new instead of the copied struct literal
in every test; shared::fixtures::report_to_file replaces seven
near-identical run_<reporter>_report helpers; shared::print_appendices
prints the --summary and --history blocks for both console reporters.
- cpd-tokenizer: embedded::html_only_maps and tokenize_blocks_shifted
hold the code razor.rs and sfc.rs had copied from each other.
971 workspace tests pass, clippy clean.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
The one-line sparkline was too small to read a trend from. The console
block now draws an eight-row bar chart: a percentage axis on the left
spanning the series' own min and max (a series moving between 2.0% and
2.4% would be a flat line on a 0-based axis), one column per point built
from eighth-block characters so every point keeps at least a sliver, an
x-axis, and point numbers that match a new `#` column in the table.
Past 20 points the columns narrow to one character so 30 points still
fit an 80-column terminal. The ai reporter keeps the compact sparkline.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
CITATION.cff (CFF 1.2.0, validated with cffconvert) gives GitHub's
"Cite this repository" button BibTeX and APA output for the project.
The README gets a short Citation section with a ready-made @software
entry. sync-version.mjs keeps the file's version in step with
rust/package.json and stamps date-released whenever the version
changes, and verifies the version like the other manifests.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#1002 (the --history half; --ratchet is deliberately not
implemented, the report shows the headroom instead).
`jscpd src --history v5.0.0..HEAD` (or `--history-since 2026-01-01`)
lists the commits with `git log`, checks each one out into a temporary
detached worktree (the --baseline-from-ref machinery, now shared through
baseline_ref::map_scan_paths), scans it with the run's own configuration
and keeps the totals; the current run is the series' last point.
`--history-every N` keeps every Nth commit counted from the newest,
`--history-limit N` (default 30) samples a long series evenly keeping
both ends. Nothing runs unless one of the flags is given.
Output: the console reporter appends a sparkline of the duplication
percentage, a table (commit, date, files, lines, clones, dup lines,
dup%, change, subject) with the change column red when duplication rose
and green when it fell, the overall trend line, and, when --threshold is
set and the latest value is under it, the headroom with the value the
threshold could be tightened to. The json reporter adds a `history`
object; the ai reporter prints one compact line per point. Other
reporters are unchanged.
New cpd_core::history (model, sparkline, change/headroom helpers with
unit tests), cpd::history (git log parsing, sampling, worktree scans),
cpd_reporter::history_render. Config keys history / historySince /
historyEvery / historyLimit; action input `history`. Integration tests
build a four-commit repo with fixed dates and check the JSON series,
the console block, every/limit thinning and the error paths. 969
workspace tests pass, clippy clean.
Docs: History section and option rows in docs/rust.md, README bullet,
action input row, fixtures/history-demo/README.md with a script that
builds the demo repository and the expected output.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
Closes#1047.
Four situations used to write an empty report and exit 0, so a CI job
that checks only the exit code saw a green run:
- `--format` naming a format that is not supported (a warning since #964)
is now `Error: --format: 'x' is not a supported format`, exit 1. Names
introduced through --formats-exts or --formats-names stay accepted.
- A scan path that does not exist is `Error: path does not exist: <p>`,
exit 1, before anything runs.
- A reporter that fails to write its output still prints the existing
`Reporter '<name>' error: ...` line; the run now ends with
`ERROR: a reporter failed to write its output` and exit 1, after the
other reporters have run.
- A scan that analyzes no files (paths exist, but nothing matched the
--format/--ignore/--pattern filters or every file was below
--min-tokens) prints `Warning: jscpd analyzed no files` and still
exits 0, so an intentionally empty tree passes. The new
`--fail-on-empty` flag (config key `failOnEmpty`, action input
`fail-on-empty`) makes it exit 1. Reports are written before the check.
Docs: an "Exit codes" section in docs/rust.md, option rows in
docs/rust.md and docs/ci-and-hooks.md, a README feature bullet.
Fixtures: fixtures/fail-on-empty-demo with a `code/` clone pair, a
`nothing/` comment-only file, and a README quoting every command's
output and exit code. Tests: the #964 warning test becomes an error
test, plus integration tests for each new exit path and for the config
key; 531 tests pass, clippy clean.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
`--ignore-pattern` has no `-i` short flag (only `--ignore` does), and
ignore globs are matched against the whole file path, so bare directory
names such as `node_modules` in the config example did not exclude
anything (verified with 5.2.0: 3 of 4 files still scanned). Use the
`**/dir/**` form the other docs already use and say so in the table.
Claude-Session: https://claude.ai/code/session_01WztdFKZAwW9b51iJe9zhtV
apk