mirror of https://github.com/kucherenko/jscpd.git synced 2026-09-19 08:11:03 +08:00

Files

T

History

apk 06481d3cb7 refactor: delete wrappers that only forward

- FinderError was never constructed; run() cannot fail, so its error type
  is Infallible and the documented run(&config).unwrap() keeps compiling.
- tokenize_sfc_block_inner and tokenize_to_detection_inner forwarded to
  tokenize_format_to_detection; call it directly.
- extract_line_number was Some(error.line()).
- Timer wrapped Instant; use Instant.

2026-09-17 15:40:19 +02:00

src

refactor: delete wrappers that only forward

2026-09-17 15:40:19 +02:00

Cargo.toml

chore: drop unused dependencies

2026-09-17 15:37:40 +02:00

README.md

chore: remove defunct Universal Analytics tracking pixels from all READMEs

2026-06-13 11:08:39 +02:00

README.md

cpd-tokenizer

Source code tokenizers for cpd — a fast copy/paste detector for code.

This crate provides language-aware tokenization for 200+ programming languages, producing token streams suitable for duplicate detection. It includes:

Generic tokenizer (comments, strings, keywords, punctuation)
JavaScript/TypeScript/JSX/TSX tokenizer (Oxc-based)
Markdown tokenizer (code fences, front matter, embedded languages)
SFC tokenizer (Vue, Svelte, Astro)
Embedded language detection and cross-format tokenization

This crate is not intended to be used directly; see the jscpd crate for the full CLI.

License

MIT