The telemetry docs are a privacy contract, and they still described a
managed analytics store that no longer receives anything. Replace that
with what actually happens now — events land in our own D1 database on
Cloudflare, the endpoint makes no outbound requests, raw events are
purged after 90 days and only anonymous daily rollups outlive them.
This strengthens the guarantee rather than restating it: there is no
second party to share with.
- TELEMETRY.md: new "Where it is stored" section; the never-collected
IP bullet no longer leans on a vendor-side setting to hold.
- docs/design/telemetry.md: ingest section rewritten around D1 + the
nightly rollup/retention cron; volume math redone on Workers Paid and
the D1 quota (storage, not writes, is what sets the 90-day window);
new section documenting the dashboard worker and cross-linking it.
- Fixed three drifts from the worker allowlist the sweep surfaced:
schema_version was still 1, client_name/client_version was still
marked "plumbing to add" though session.ts passes it today, and the
legacy sqlite_backend field the worker still accepts was undocumented.
- telemetry-worker/README.md: step 6 claimed a repo-wide grep came back
clean, which this runbook itself falsifies. Added step 7 — deleting
the runbook is what makes that grep true, and is the completion check.
- smoke-cutover.sh: the vendor guarantee is now asserted by class
(no analytics-ingest endpoint referenced) rather than by one vendor's
name, so it keeps working once the name is gone. Verified it still
catches a planted forwarding URL. 61/61 pass.
Retention is documented as 90 days, not the 180 in the task notes: 180
days of raw events exceeds D1's 10 GB per-database cap, and the code
purges at 90.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The account-level steps of the PostHog cutover are the maintainer's to run, so
this lands the runbook they follow and the check that has to pass first.
The runbook (telemetry-worker/README.md) walks the six steps in the order that
keeps them reversible: Workers Paid → migrate → deploy → watch 24h → verify the
first rollup and the dashboard → only then delete POSTHOG_KEY and cancel the
subscription. Step 3 records the outgoing version id because `wrangler rollback`
is the escape hatch for the whole verification window, and that window is
precisely why the PostHog key is deleted last rather than first.
The new gate (scripts/smoke-cutover.sh, `npm run smoke:cutover`) covers the one
seam nothing else did. Both workers declare the same D1 database_id, so pointing
them at a single --persist-to directory runs the real chain: a client batch →
the ingest worker → D1 → the nightly rollup → the dashboard API reading the
numbers back. Every other suite stops at one link — smoke-ingest at the events
table, smoke-rollup at hand-checked SQL, smoke-api at a hand-written fixture
that the cron never touched. That left the dimension names the rollup WRITES
versus the ones the dashboard READS agreeing by convention across two branches,
where a mismatch is silent: no error, no failed request, just a panel reading
zero forever. 61 assertions, all 13 dimensions, and three deliberate traps — a
ci machine that is active but not a production user, usage_rollup counts that
must be summed rather than tallied, and an uninstall's `targets` that must not
leak into the install-scoped breakdown.
Writing it caught that the activation funnel's denominator is first-seen
machines, not install events (deliberate — a reinstall must not re-enter the
funnel), so the suite now pins that distinction rather than assuming it.
Also rewords the last PostHog reference in dashboard code: a comment justifying
the 14-day retention curve by pointing at a dashboard step 6 deletes. The
reasoning now stands on its own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds a scheduled() handler to the ingest worker that recomputes
daily_event_counts / daily_dim_counts / daily_machines for the just-completed
UTC day plus a 2-day overlap (late-arriving offline buffers), then purges raw
events past the retention window. Rollup writes are idempotent upserts, so a
re-run never double-counts. Also adds an ADMIN_TOKEN-guarded
POST /admin/rollup?day=YYYY-MM-DD for backfill/repair, and drops the PostHog
forwarding path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
First step of replacing PostHog with self-hosted telemetry on Cloudflare D1.
Creates the codegraph-telemetry database binding and the initial migration; no
worker code paths change yet (the ingest write path and the nightly rollup cron
land next).
Schema is raw events plus daily rollups: `events` holds one row per sanitized
event with the envelope broken out into columns and event-specific props as
JSON; `daily_machines`, `daily_event_counts` and `daily_dim_counts` are the
nightly rollups the dashboard reads; `machine_first_seen` and `machine_days`
carry the retention cohorts and are never purged. One generic dimension table
covers every bar and pie, so a new breakdown is a cron change rather than a
migration.
The migration is commented as an audit surface, like the rest of this worker —
every column, and which dashboard chart each rollup table serves.
Three judgment calls worth flagging, all documented in the file:
- `events` gets `(day, event)` instead of the separate `(day)` and `(event, day)`
indexes. D1 bills a row write per index touched, so a third index on the hot
table costs ~97k writes/day, and `(day, event)` is a covering index for plain
day-range scans anyway (verified with EXPLAIN QUERY PLAN).
- `daily_event_counts` and `daily_dim_counts` carry a `machines` column, and
`machine_days` a `prod` flag. The "users by ..." panels and the production-user
count are distinct-machine numbers, not event counts, and they are
unrecoverable once raw events are purged.
- No CHECK constraint on `event`: the worker's allowlist is the source of truth
and the write path is fail-silent, so a rejected INSERT would lose data
quietly instead of erroring loudly.
Volume note in the migration footer: ~30M row writes/month against the 50M
included on Workers Paid. Storage is the tighter constraint — raw events grow
~74 MB/day, so retention should start at 90 days (~6.7 GB) rather than 180,
which would exceed D1's 10 GB per-database cap.
* perf(resolution): resolve imports to definitions, not sibling import nodes (#915)
"Resolving refs" crawled (tens of minutes) on large projects — most painfully
ones mixing a big front-end and back-end. An external package or module imported
across hundreds/thousands of files (react, a shared UI package, Python
logging/typing) is re-declared as an `import` node in every importing file, so
its unresolved import ref fell through to the exact-name matcher, which scored
all K same-named import nodes via findBestMatch — K refs x K candidates = O(K^2)
per package, producing only meaningless import->import edges.
Fix: exclude `import`-kind nodes as name-match targets (they're statements, not
definitions; real import->definition resolution is the import resolver's job).
Plus two safe constant-factor wins in findBestMatch: hoist the per-candidate
ref.filePath split, and skip cross-language candidates when a same-language one
exists (provably the same winner — same-language scores >=50, cross-language
maxes at 35).
Measured: superset (Py+TS) candidates scored 7.5M -> 833K (9x), non-import edges
preserved (+1618 now resolve to real defs), ~22K useless import->import edges
removed; kubernetes (Go) computePathProximity 37.2s -> 5.0s; synthetic 8k-file
mixed repo (K=4000) resolution 16.0s -> 1.7s. Full suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: correct stale better-sqlite3/wasm references to node:sqlite
The SQLite backend has been Node's built-in node:sqlite (real SQLite, WAL + FTS5,
from the bundled runtime) for a while — there is no native build step and no
node-sqlite3-wasm fallback. README and the docs site were already updated; this
catches the stragglers:
- CLAUDE.md: the src/db/ backend description and the sqlite-backend test note.
- src/db/index.ts, src/mcp/tools.ts: two code comments that still blamed "the
wasm backend" for non-WAL behavior (reworded to "when WAL isn't in effect").
Leaves tree-sitter grammar wasm (web-tree-sitter / --liftoff-only) untouched —
that's a different, still-current use of wasm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(telemetry): drop the dead sqlite_backend field (schema v2)
node:sqlite is now the only backend, so the `index` event's `sqlite_backend`
field was a constant ("native") carrying no signal — and the `install` event
never actually sent it. Remove the field and the backendKind() helper, bump the
telemetry SCHEMA_VERSION 1 -> 2, and update TELEMETRY.md + docs/design/telemetry.md.
The ingest worker is deliberately left tolerant: `index` doesn't require the
field and schema_version validates as nonNegInt(99), so v2 events ingest fine and
old clients still sending v1 + sqlite_backend keep validating too. Added a legacy
comment there explaining it's safe to drop once old-client share is negligible.
telemetry.test.ts: the assertion pinning schema_version and a stale-claim fixture
line updated 1 -> 2. All telemetry tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds anonymous usage statistics (commands/tools used, languages indexed,
connecting agents) with a strict, auditable allowlist. Never code, paths,
file/symbol names, queries, or IPs.
- src/telemetry/: zero-dep client — consent resolution (DO_NOT_TRACK >
CODEGRAPH_TELEMETRY > stored choice > default-on), random machine UUID,
in-memory counters → capped JSONL buffer → completed-day rollups; sync
exit-append (survives process.exit) + opportunistic bounded sends; the
first-run notice gates the first SEND, never local buffering, so the
installer's consent toggle always precedes it. Off is off: no recording,
no socket, buffered data deleted.
- codegraph telemetry status|on|off; per-command counting via preAction hook.
- MCP: tool counting after the reply is on the wire (session + proxy
in-process fallback), agent attribution from initialize clientInfo,
unref'd daemon flush interval. Zero hot-path cost, zero stdout.
- Installer: visible default-on consent toggle (asked once, never re-asked),
install/index/uninstall lifecycle events.
- telemetry-worker/: public Cloudflare Worker behind telemetry.getcodegraph.com
— allowlist validation, IP stripping, per-machine rate limit, forwards to
PostHog as anonymous events. Ships nowhere with the npm package.
- TELEMETRY.md (field-by-field contract) + README section + design doc.
- 20 unit tests; suite-wide CODEGRAPH_TELEMETRY=0 guard so tests never
pollute real telemetry. Full suite: 1448 passing.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Colby McHenry