Commit Graph

4 Commits

Author SHA1 Message Date
apk ad7fc6ebdc feat(tokenizer): add Markdown cross-format clone detection
Extract YAML front matter and fenced code blocks from .md files and
tokenize each segment with its resolved sub-language format (yaml,
typescript, python, javascript, etc.), enabling jscpd to detect
duplicates across languages embedded in Markdown documents.

- Implement tokenizeMarkdown() with three-tier resolveFormat() lookup
- Split markdown grammar into markdown-format.ts to break circular dep
- Dispatch markdown format through createTokenMapBasedOnCode()
- Separate .txt into its own format (removed from markdown.exts)
- Export EXT_TO_FORMAT from formats.ts for sub-language resolution
- Add 25-test suite covering all acceptance criteria (271 total)
- Add fixtures/markdown/file3.md + file4.md for integration testing
2026-05-13 11:00:29 +02:00
Josh Soref c2fefbe52d spelling: coffee
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2023-01-04 18:23:31 -05:00
Lorenzo Gabriele 9387257a3e Fix typos coma => comma 2020-10-19 16:58:28 +02:00
Andrey Kucherenko 633a854335 refactor: 💡 add new branch to git repo 2020-04-29 19:14:00 +03:00