Files
backnotprop__plannotator/packages/shared/html-to-markdown.test.ts
Michael Ramos b780739291 feat(annotate): support HTML files and URL annotation (#545)
* fix(annotate): sanitize dangerous link protocols in markdown renderer

Block javascript:, data:, and vbscript: URLs in InlineMarkdown link
rendering. Links with dangerous protocols render as plain text instead
of clickable anchors. Uses a blocklist approach so existing links with
custom protocols (obsidian://, vscode://, Windows C:\ paths) continue
to work.

For provenance purposes, this commit was AI assisted.

* feat(annotate): add HTML-to-markdown and URL-to-markdown utilities

- html-to-markdown.ts: Turndown wrapper with GFM table rule, strips
  script/style/noscript tags
- url-to-markdown.ts: Jina Reader (free, returns markdown) with
  fetch+Turndown fallback. Warns on Jina failure, auto-skips Jina for
  local/private URLs (localhost, 192.168.*, 10.*, etc.)
- config.ts: add jina setting and resolveUseJina() with priority chain
  --no-jina flag > PLANNOTATOR_JINA env > config.json > default true

For provenance purposes, this commit was AI assisted.

* feat(annotate): support HTML files and URLs in annotate command

Extend the annotate subcommand to accept .html/.htm local files
(converted via Turndown) and https:// URLs (fetched via Jina Reader
with fetch+Turndown fallback). URL content is fetched terminal-side
before opening the browser.

Add --no-jina global flag to disable Jina Reader per-invocation.
Add 10MB file size guard for local HTML files.

For provenance purposes, this commit was AI assisted.

* feat(annotate): HTML files in folder browser and on-demand conversion

- Widen file browser glob to include .html/.htm alongside markdown
- handleDoc converts HTML files via Turndown on demand when selected
- hasMarkdownFiles accepts optional extensions param for folder validation
- Add sourceInfo field to annotate server API response
- Add _site/, public/, out/, .docusaurus/, .jekyll-cache/,
  storybook-static/ to FILE_BROWSER_EXCLUDED

For provenance purposes, this commit was AI assisted.

* feat(annotate): source attribution badge for HTML/URL annotations

Show a subtle badge in DocBadges displaying the URL hostname or HTML
filename for converted content. Thread sourceInfo from API response
through App → Viewer → DocBadges.

Also update Pi extension to accept HTML-only folders in annotate mode.

For provenance purposes, this commit was AI assisted.

* test: update CLI help text assertion for HTML/URL annotate support

For provenance purposes, this commit was AI assisted.

* fix(annotate): address PR review findings

Security:
- Add project-root containment check for HTML files in /api/doc handler
  using exported isWithinProjectRoot() from resolve-file.ts
- Blocks path traversal via absolute paths or ../ escapes

isLocalUrl fixes:
- Add bracketed IPv6 loopback [::1] detection
- Replace hostname.startsWith('10.') with proper IPv4 regex to avoid
  matching public hostnames like 10.example.com

Revert Pi extension change:
- Pi server doesn't implement HTML file browsing or conversion yet
- Keep Pi folder validation markdown-only until both implementations
  are updated per CLAUDE.md guidelines

Cleanup:
- Remove dead el.children || el.childNodes fallback in table rule
- Extract hostnameOrFallback() helper to @plannotator/shared/project
  replacing duplicated try/catch IIFEs in DocBadges and index.ts

For provenance purposes, this commit was AI assisted.

* feat(annotate): Pi extension HTML annotation parity

Bring the Pi extension to full parity with the Bun server for HTML
annotation support:

- Vendor html-to-markdown and url-to-markdown via vendor.sh
- walkMarkdownFiles now scans .html/.htm alongside markdown
- handleDocRequest converts HTML files on-demand via Turndown with
  isWithinProjectRoot containment check
- serverAnnotate includes sourceInfo in /api/plan response
- index.ts supports URL detection (Jina Reader + fallback), HTML file
  detection with Turndown conversion, folder HTML validation, and 10MB
  file size guard
- openMarkdownAnnotation accepts and threads sourceInfo
- Add turndown as a Pi extension dependency

For provenance purposes, this commit was AI assisted.

* fix(pi): Obsidian vault walks stay markdown-only, add try/catch for HTML

- Add extensions param to walkMarkdownFiles (default: HTML-inclusive)
- Obsidian callers pass /\.mdx?$/i to match Bun server behavior
- Add try/catch around HTML file reads in handleDocRequest

For provenance purposes, this commit was AI assisted.

* fix(annotate): address second review — base-block traversal, metadata IP, dead code

Security:
- Add isWithinProjectRoot check to the base-relative block for HTML
  files in both Bun and Pi /api/doc handlers. Previously HTML files
  served via the base query param bypassed the containment guard.
- Add 169.254.0.0/16 (link-local / cloud metadata) to isLocalUrl
  private IP ranges

Cleanup:
- Remove dead hostname === "[::1]" check (WHATWG URL parser strips
  brackets; hostname === "::1" already handles it)
- Remove dead parent?.childNodes fallback in table cell() function

For provenance purposes, this commit was AI assisted.

* refactor(annotate): replace custom table rules with turndown-plugin-gfm

Drop ~60 lines of hand-rolled GFM table conversion that had a bug
(tables without explicit <thead> produced invalid GFM). Use the
official turndown-plugin-gfm plugin (24KB) which correctly handles
all table patterns plus adds strikethrough and task list support.

For provenance purposes, this commit was AI assisted.

* fix(annotate): handle all CommonMark backslash escapes in InlineMarkdown

Expand the backslash escape regex to cover all CommonMark-defined
escapable characters (. ) - # > + | { } &), not just the subset
the parser uses for formatting. Fixes literal backslashes appearing
in rendered output for Turndown-escaped content like "1\." → "1.".

For provenance purposes, this commit was AI assisted.

* fix(annotate): prevent SSRF via redirect to private/local URLs

Replace redirect: "follow" with redirect: "manual" in fetchViaTurndown
and validate each redirect hop against isLocalUrl. Blocks attacks where
an external URL redirects to cloud metadata endpoints (169.254.169.254)
or other private IPs. Limits redirect chain to 10 hops.

For provenance purposes, this commit was AI assisted.

* chore: update lockfile for turndown-plugin-gfm in Pi extension

bun install needed to resolve turndown-plugin-gfm in the Pi extension
workspace after adding it to apps/pi-extension/package.json.

For provenance purposes, this commit was AI assisted.

* fix(annotate): switch to @joplin/turndown-plugin-gfm, fix TS errors

Replace unmaintained turndown-plugin-gfm (2017, v1.0.2) with the
actively maintained Joplin fork (2025, v1.0.64, 16KB).

Fix TypeScript errors that broke CI:
- Add @ts-expect-error for untyped @joplin/turndown-plugin-gfm import
- Restructure fetchViaTurndown redirect loop to avoid uninitialized
  variable — first fetch before loop, loop only for redirects

For provenance purposes, this commit was AI assisted.

* fix(annotate): use proper declarations.d.ts instead of ts-expect-error

Add declarations.d.ts for @joplin/turndown-plugin-gfm with typed
function signatures, remove the ts-expect-error suppression.

For provenance purposes, this commit was AI assisted.

* fix: explicitly include declarations.d.ts in shared tsconfig

CI's tsc wasn't finding the ambient module declaration with implicit
include. Add explicit include to ensure declarations.d.ts is always
picked up regardless of environment.

For provenance purposes, this commit was AI assisted.

* fix: use ts-expect-error for @joplin/turndown-plugin-gfm types

CI's tsc does not pick up ambient declarations.d.ts files despite
local tsc finding them — likely a module resolution discrepancy
between environments. Revert to @ts-expect-error which passes in
both CI and local typecheck.

For provenance purposes, this commit was AI assisted.

* fix(annotate): body size limit for URL fetches, redirect error, file: protocol

- Add 10MB body size limit to both Jina and fetch+Turndown URL paths,
  matching the local HTML file guard. Streams response body and aborts
  if limit exceeded.
- Distinguish "Too many redirects" from a genuine 3xx response after
  redirect loop exhaustion.
- Add file: to the dangerous protocol blocklist in sanitizeLinkUrl.

For provenance purposes, this commit was AI assisted.

* fix(annotate): HTML folder outside cwd, HTML linked doc navigation

- Remove containment check from base-relative block for HTML files in
  both Bun and Pi /api/doc handlers. Matches markdown behavior so HTML
  files in annotated folders outside cwd are served correctly.
  Standalone block (no base) retains its cwd check as fallback.
- Widen isLocalMd → isLocalDoc to treat .html/.htm links as linked
  documents. Clicking [Next](next.html) in a converted page now opens
  it via /api/doc with Turndown conversion instead of a new browser tab.

For provenance purposes, this commit was AI assisted.

* fix(annotate): full loopback range, drain redirect bodies, document env vars

- Expand loopback check from just 127.0.0.1 to the full 127.0.0.0/8
  range so all loopback addresses skip Jina Reader
- Cancel redirect response body before re-fetching to avoid leaking
  TCP connections back to the pool
- Document PLANNOTATOR_JINA and JINA_API_KEY in CLAUDE.md env var table

For provenance purposes, this commit was AI assisted.

* fix(annotate): IPv6 loopback, readBodyWithLimit fallback, env var docs, comments

- Add [::1] back to isLocalUrl — WHATWG URL hostname getter preserves
  brackets for IPv6 (verified: Bun and Node both return "[::1]").
  Add comment explaining the empirical verification so future reviewers
  don't re-flag.
- Fix readBodyWithLimit null-body fallback to still enforce the 10MB
  limit via text length check instead of silently falling through.
- Document PLANNOTATOR_JINA and JINA_API_KEY in AGENTS.md env var table
  (CLAUDE.md is a symlink to AGENTS.md).
- Add comments to base-relative blocks in both Bun and Pi handleDoc
  explaining the intentional lack of containment check (matches
  pre-existing markdown behavior, base is set server-side).

For provenance purposes, this commit was AI assisted.

* fix(annotate): block IPv4-mapped IPv6 and private IPv6 ranges in isLocalUrl

Add PRIVATE_IPV6 regex matching bracketed IPv6 private/reserved ranges:
- ::ffff: (IPv4-mapped — embeds private IPv4 as hex, e.g. [::ffff:c0a8:1])
- fe80: (link-local)
- fc00::/7 (unique-local, covers fc00:: through fdff::)

Closes the redirect-SSRF bypass where a public URL redirects to a
private address expressed as IPv4-mapped IPv6, e.g.
http://[::ffff:169.254.169.254]/latest/meta-data/

For provenance purposes, this commit was AI assisted.

* fix(annotate): document IPv6 hostname verification, sourceInfo type, annotate flow

- Expand isLocalUrl comment with full empirical verification table
  showing actual hostname getter output for every IPv6 format in both
  Bun and Node — prevents false-positive review findings about brackets
- Add sourceInfo to /api/plan response type in App.tsx for type safety
- Update CLAUDE.md annotate flow diagram to reflect HTML/URL/folder
  input types

For provenance purposes, this commit was AI assisted.

* fix(annotate): escape \(, cancel response bodies on error, doc sourceInfo

- Add ( to backslash escape regex alongside existing ) — Turndown
  emits \( in link-adjacent contexts
- Cancel response body before throwing on !res.ok in both fetchViaJina
  and fetchViaTurndown error paths (redirect loop already did this)
- Document sourceInfo field in AGENTS.md annotate server API table

For provenance purposes, this commit was AI assisted.

* fix(annotate): skip base injection for URL annotations, body cleanup

- Skip dirname(filePath) base injection when filePath is a URL in both
  Bun and Pi annotate servers. dirname on a URL string produces a
  nonsensical filesystem path, causing linked doc clicks to 404.
  URL annotations now let links open normally instead.
- Cancel response body before throwing on content-type mismatch and
  content-length overflow in fetchViaTurndown/readBodyWithLimit.
- Fix double parseInt in readBodyWithLimit content-length check.
- Correct AGENTS.md flow diagram: OpenCode not yet implemented for
  HTML/URL annotation.

For provenance purposes, this commit was AI assisted.

* feat(annotate): OpenCode HTML file and URL annotation support

Add URL detection (Jina Reader + fallback), HTML file detection with
Turndown conversion, 10MB file size guard, and sourceInfo threading
to OpenCode's handleAnnotateCommand. Uses the same shared utilities
as the Bun CLI and Pi extension.

OpenCode uses the Bun server directly (startAnnotateServer from
@plannotator/server/annotate), so no server-side changes needed —
only the command handler routing was missing.

Note: folder annotation mode is not added (OpenCode didn't have it
before this PR for markdown either — separate scope).

For provenance purposes, this commit was AI assisted.

* chore(annotate): update slash command description, align fetch log messages

- OpenCode plannotator-annotate.md description now mentions HTML/URL
- Align fetch progress messages across all three clients: all now show
  "(via Jina Reader)" or "(via fetch+Turndown)" consistently

For provenance purposes, this commit was AI assisted.

* fix(annotate): skip conversion for .md URLs, wikilink HTML targets, cleanup

- URLs ending in .md/.mdx are fetched raw — no Jina, no Turndown.
  Content is already markdown. Removes text/plain from fetchViaTurndown
  content-type whitelist since .md URLs are now short-circuited.
- Wikilink regex widened to preserve .html/.htm targets instead of
  appending .md (e.g. [[page.html]] no longer becomes page.html.md)
- Remove redundant existsSync before statSync in OpenCode handler

For provenance purposes, this commit was AI assisted.

* test(annotate): add htmlToMarkdown conversion tests

Tests cover the core conversion utility that all three clients depend on:
- Basic HTML → markdown (headings, paragraphs, links, code blocks)
- Tables with and without <thead> (the GFM plugin bug that was caught)
- Script/style/noscript stripping
- Strikethrough (GFM)
- Empty HTML handling
- Dangerous links preserved (sanitization is in the renderer, not here)

For provenance purposes, this commit was AI assisted.

* fix(annotate): check content-type before treating .md URLs as raw markdown

URLs ending in .md/.mdx (e.g. GitHub's viewer page for README.md)
may return HTML instead of raw markdown. fetchRawText now checks the
response content-type — if the server returns HTML, returns null so
the caller falls through to Jina/Turndown for proper conversion.

For provenance purposes, this commit was AI assisted.

* fix(annotate): add SSRF redirect protection to fetchRawText

fetchRawText (for .md/.mdx URLs) was using default redirect: "follow"
with no isLocalUrl validation on redirect hops — a .md URL redirecting
to 169.254.169.254 would be followed and credentials returned as
"markdown". Now uses redirect: "manual" with per-hop isLocalUrl checks,
matching fetchViaTurndown's SSRF protection.

For provenance purposes, this commit was AI assisted.
2026-04-12 18:56:28 -07:00

63 lines
2.3 KiB
TypeScript

import { describe, expect, test } from "bun:test";
import { htmlToMarkdown } from "./html-to-markdown";
describe("htmlToMarkdown", () => {
test("converts basic HTML to markdown", () => {
const md = htmlToMarkdown("<h1>Hello</h1><p>World</p>");
expect(md).toContain("# Hello");
expect(md).toContain("World");
});
test("converts tables with explicit thead", () => {
const html = "<table><thead><tr><th>A</th><th>B</th></tr></thead><tbody><tr><td>1</td><td>2</td></tr></tbody></table>";
const md = htmlToMarkdown(html);
expect(md).toContain("| A");
expect(md).toContain("| ---");
expect(md).toContain("| 1");
});
test("converts tables WITHOUT thead (common HTML pattern)", () => {
const html = "<table><tr><th>Name</th><th>Value</th></tr><tr><td>foo</td><td>bar</td></tr></table>";
const md = htmlToMarkdown(html);
expect(md).toContain("| Name");
expect(md).toContain("| ---");
expect(md).toContain("| foo");
});
test("strips script, style, and noscript tags", () => {
const html = '<p>Visible</p><script>alert("xss")</script><style>.x{}</style><noscript>Hidden</noscript>';
const md = htmlToMarkdown(html);
expect(md).toContain("Visible");
expect(md).not.toContain("alert");
expect(md).not.toContain(".x{}");
expect(md).not.toContain("Hidden");
});
test("converts strikethrough (GFM)", () => {
const md = htmlToMarkdown("<p>This is <del>deleted</del> text</p>");
expect(md).toContain("~deleted~");
});
test("preserves code blocks as fenced markdown", () => {
const md = htmlToMarkdown('<pre><code>const x = 1;</code></pre>');
expect(md).toContain("```");
expect(md).toContain("const x = 1;");
});
test("converts links", () => {
const md = htmlToMarkdown('<a href="https://example.com">Click</a>');
expect(md).toContain("[Click](https://example.com)");
});
test("preserves dangerous link hrefs (sanitization is in the renderer, not here)", () => {
const md = htmlToMarkdown('<a href="javascript:alert(1)">XSS</a>');
// Turndown preserves the link — the UI renderer blocks it
expect(md).toContain("javascript:");
});
test("handles empty HTML gracefully", () => {
expect(htmlToMarkdown("")).toBe("");
expect(htmlToMarkdown("<html><body></body></html>")).toBe("");
});
});