Files
2pac 8c15da7414 feat: sql command — DuckDB queries over stored documents (CLI, API, MCP, UI) (#18)
* feat: sql command — DuckDB queries over stored documents (CLI, API, MCP, UI)

Adds a sandboxed `sql` op that runs DuckDB SQL over drive documents:
csv, tsv, parquet, xlsx, json, ndjson/jsonl (text formats also .gz),
sqlite (.db/.sqlite/.sqlite3, tables exposed as <name>.<table>), and
.duckdb. Documents are referenced by quoted drive-path literals
(SELECT * FROM '/data/sales.csv'), bound as named tables, or format-
overridden to make any document queryable (-t logs=/raw/data.txt:csv).

Security: the sqlite extension bypasses DuckDB's allowed_directories
sandbox (own file I/O), so all inputs are materialized into in-memory
tables during setup — sqlite via an isolated bridge instance converted
to parquet — then the local filesystem is disabled, external access cut,
and configuration locked before any user SQL runs. Interrupt-based
timeout, per-file size cap, and memory limit are env-tunable.

- core: sql op + registry/RBAC (viewer) + data-file mime types
- cli: `agent-fs sql` with repeatable -t bindings, stdin, --max-rows,
  aligned table output; @duckdb/node-api marked external in the bundle
- server/mcp: exposed automatically via ops dispatch + tool registry
- live: SQL workbench at /sql/~/:orgId/:driveId — DuckDB-WASM in the
  browser for csv/tsv/parquet/json, automatic server fallback for
  xlsx/sqlite/duckdb; results table, SVG charts, CSV/JSON export,
  "Query" button on queryable files
- e2e: 15 new tests (CLI + raw API + MCP) against MinIO — suite 96/96
- docs: docs/sql.md + landing registration + SKILL.md; versions synced
  to 0.8.2

* feat(live): shared virtualized DataGrid + tabular/db file previews + SQL UX

Adds a shared DataGrid (TanStack Table + react-virtual) used by both the SQL
results panel and a new default preview for tabular/data documents, plus a
batch of SQL workbench UX fixes from QA.

DataGrid & previews:
- live/src/components/data-grid/DataGrid.tsx — headless, virtualized, themed
  grid (sort, resize, type tooltips, dimmed NULLs); replaces the hand-rolled
  results table.
- TablePreviewViewer — csv/tsv/ndjson/parquet/xlsx render as a grid by default
  via the SQL engine (server-side, so previews work even when the raw-bytes
  presigned fetch is CORS-blocked); csv/tsv/ndjson keep a Source toggle.
- DatabasePreviewViewer — sqlite/duckdb files list their tables and preview the
  selected one in the grid.
- FileViewer: route tabular/db files to these previews; office binaries
  (doc/docx/ppt/pptx/xls/odt…) no longer render as garbled text — they fall
  back to the download view.

SQL workbench:
- Engine control is now a Browser/Server toggle with tooltips, reflecting the
  effective engine; Browser disabled (with reason) when unsupported.
- Browser-load size estimate; sizes on doc chips and preview footers.
- sqlite/duckdb seeding introspects tables → seeds a working `schema.table`
  query (fixes the path-literal error on database files).
- Cmd/Ctrl+Enter runs via Monaco addAction (reliable keybinding).
- Editor autocompletes bound table and `schema.table` names.

Misc:
- Switching connection/org/drive resets to `/`.
- Correct the MinIO-gated op-registry count assertion (now 29, includes sql).

* chore: bump version to 0.9.0 (sql feature is a minor release)

* docs(landing): SEO/GEO for the SQL docs page

- sitemap.xml + sitemap.md: add /docs/sql (IndexNow submits it on deploy, since
  it parses sitemap.xml).
- DocsPage: per-document SEO meta — unique <title>, description, canonical,
  Open Graph, Twitter, and TechArticle JSON-LD — so every doc page (including
  sql) is independently indexable and answer-engine citable.
- Re-sync public/docs/openapi.json to the 0.9.0 spec.

llms.txt already lists the SQL page (Reference + core commands); public/docs/sql.md
is synced from docs/sql.md by generate:md.

* fix(sql): address PR review — path-literal scoping, gzip, dup columns, cache, engine routing

Resolves the 6 P2 review findings:

core (packages/core/src/ops/sql.ts):
- Only rewrite path literals in FROM/JOIN table position, never value-position
  strings — `WHERE x = '/data/sales.csv'` is no longer clobbered.
- Honor a real .gz suffix even when the format is explicitly overridden, so the
  temp file keeps its .gz extension and DuckDB decompresses it.
- Uniquify duplicate result column names (e.g. two projected `id`s) so building
  row objects never drops data by key collision.

live (sql-engine):
- formatForPath strips an optional .gz suffix (and deriveTableName too); the
  document picker adds .gz globs — gzipped text docs are pickable/seedable.
- Browser-engine file registration keys on the current file revision (stat),
  so an edited doc re-registers instead of serving a stale buffer.
- canRunInBrowser is query-aware: a FROM/JOIN path literal that isn't a bound
  doc routes to the server (only it auto-binds bare path literals).

+ tests for path-literal scoping, gzip override, and duplicate columns.

* fix(sql): apply table-position rewrite + stale-view cleanup to the WASM engine

Second review pass flagged the browser engine still had two issues the server
fix didn't cover:

- rewritePathLiterals now only rewrites FROM/JOIN path literals (matching the
  server), so a value-position string like `WHERE source = '/data/sales.csv'`
  is no longer clobbered in the browser engine.
- The persistent WASM db kept views from earlier runs; drop views that aren't
  bound in the current run (removed docs, org/drive switch) before querying, so
  a stale table name can't be selected out of scope.

Verified in headless Chrome: a seeded `FROM '/qa/sales.csv'` query runs in the
wasm engine (badge "wasm", 5 rows) with the new rewrite/view logic.

* fix(sql): WASM duplicate columns + preview stale-response guards

Third review pass:
- duckdb.ts: the browser engine built rows keyed by Arrow field name, so
  duplicate output columns collided. Uniquify names and read cells by column
  index (getChildAt) so a query projecting two `id`s keeps both — matching the
  server-side fix.
- DatabasePreviewViewer / TablePreviewViewer: guard async loads with a monotonic
  request token so a slow response for a previously-selected file/table can't
  overwrite the current preview (switching files or db tables quickly).

* fix(sql): apply the query timeout across the whole operation, not just the query

Fourth review pass: the deadline was armed only for the query phase, leaving
downloads, extension loads, the sqlite->parquet bridge, and table
materialization unbounded. Arm the timer before any work so it covers the full
operation — it interrupts the active DuckDB connection (extension load /
materialization / query) and a checkTimeout() guard stops between phases
(downloads and the bridge, where there's no connection to interrupt). The
sqlite bridge also checks the deadline between tables.

* fix(sql): drop stale WASM files + guard workbench runs against stale results

Fifth review pass:
- duckdb.ts: ensureRegistered now also drops registered file buffers that aren't
  bound in the current run (removed docs, org/drive switch), not just stale
  views — so an out-of-scope buffer can't be read.
- SqlPage: the workbench run() guards its result/error/running updates with a
  monotonic run token, so a slow earlier run can't clobber a newer one (rapid
  re-runs or an edited query), matching the preview viewers.

* fix(sql): interrupt the sqlite bridge connection on timeout

Sixth review pass: the sqlite->parquet bridge runs on its own connection, so the
deadline could interrupt the main connection but not an in-flight bridge COPY.
Register the bridge connection with the timer so a timeout interrupts it too
(unregistered before close), completing timeout coverage across every phase.

* fix(sql): abort in-flight S3 downloads when the deadline expires

Seventh review pass: downloads were the last phase the timeout couldn't
interrupt — only checked between files. Thread an AbortSignal from the deadline
timer into AgentS3Client.getObject so an in-flight download is aborted on
timeout. Every phase (download, extension load, sqlite bridge, materialization,
query) is now interruptible.
2026-06-11 19:14:22 +02:00

5.8 KiB

SQL Queries

Run DuckDB SQL directly against documents stored in agent-fs — CSV, TSV, Parquet, Excel, JSON, NDJSON, SQLite databases, and DuckDB databases. Available as a CLI command (agent-fs sql), an API op (sql), and an MCP tool.

agent-fs sql "SELECT category, sum(amount) FROM '/finance/2026.csv' GROUP BY category"

Supported formats

Format Extensions Notes
CSV .csv, .csv.gz Delimiter, header, and types auto-detected
TSV .tsv, .tab, .tsv.gz Tab-delimited
Parquet .parquet
Excel .xlsx First sheet; header auto-detected
JSON .json, .json.gz Arrays of objects or single objects
NDJSON .ndjson, .jsonl (+ .gz) Newline-delimited JSON
SQLite .db, .sqlite, .sqlite3 All tables exposed — see below
DuckDB .duckdb Tables from the main schema

Referencing documents

There are two ways to make a document queryable:

1. Path literals (file formats)

Reference any stored CSV/TSV/Parquet/Excel/JSON/NDJSON document by its drive path as a quoted string, exactly like DuckDB's own file syntax:

agent-fs sql "SELECT * FROM '/data/sales.csv' WHERE amount > 100"
agent-fs sql "SELECT a.id FROM '/data/a.parquet' a JOIN '/data/b.ndjson' b USING (id)"

Only string literals that resolve to an existing document in the drive are bound — other strings are left untouched, so WHERE name = 'report.csv' works as expected.

2. Named table bindings

Bind documents to table names with --table (repeatable). Required for SQLite/DuckDB databases, and useful for readable multi-file queries:

agent-fs sql "SELECT s.name, t.tag FROM sales s JOIN tags t ON s.id = t.id" \
  -t sales=/data/sales.csv -t tags=/data/tags.parquet

SQLite and DuckDB databases expose every table under the binding name as a schema:

# /backups/app.db has tables `users` and `orders`
agent-fs sql "SELECT u.email, count(*) FROM app.users u JOIN app.orders o ON o.user_id = u.id GROUP BY 1" \
  -t app=/backups/app.db

Making any document SQL-able (format override)

If a document doesn't have a recognized extension, override the format with a :format suffix (CLI) or { path, format } (API/MCP):

agent-fs sql "SELECT sum(a) FROM logs" -t logs=/raw/data.txt:csv

Formats: csv, tsv, parquet, xlsx, json, ndjson, sqlite, duckdb.

CLI usage

agent-fs sql [query] [options]

  -t, --table <name=path[:format]>  Bind a document as a named table (repeatable)
  --max-rows <n>                    Max rows to return (default: 1000, max: 10000)
  --json                            Output the raw result object

The query can also be piped via stdin:

echo "SELECT count(*) FROM '/data/events.ndjson'" | agent-fs sql

Multi-statement queries are supported — the result of the last statement is returned:

agent-fs sql "CREATE TABLE big AS SELECT * FROM '/data/sales.csv' WHERE amount > 100; SELECT count(*) FROM big"

API usage

sql is a standard op on the dispatch endpoint:

curl -X POST http://localhost:7433/orgs/{orgId}/ops \
  -H "Authorization: Bearer <api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "op": "sql",
    "driveId": "<driveId>",
    "query": "SELECT s.name, t.tag FROM sales s JOIN tags t ON s.id = t.id",
    "tables": {
      "sales": "/data/sales.csv",
      "tags": { "path": "/data/tags.parquet", "format": "parquet" }
    },
    "maxRows": 1000
  }'

Response:

{
  "columns": [{ "name": "name", "type": "VARCHAR" }, { "name": "tag", "type": "VARCHAR" }],
  "rows": [{ "name": "alpha", "tag": "aa" }],
  "rowCount": 1,
  "truncated": false,
  "files": [
    { "table": "sales", "path": "/data/sales.csv", "format": "csv" },
    { "table": "tags", "path": "/data/tags.parquet", "format": "parquet" }
  ],
  "elapsedMs": 42
}

Value encoding: BIGINTs outside JavaScript's safe integer range and timestamps/dates/UUIDs are returned as strings; decimals as numbers; lists and structs as JSON arrays/objects; blobs as <blob N bytes> placeholders. The op requires the viewer role.

The same op is exposed as the sql MCP tool with identical arguments.

Web UI

The live UI has a SQL workbench at /sql/~/{orgId}/{driveId} — pick documents, write queries (Cmd+Enter to run), and view results as a table or quick chart. CSV/TSV/Parquet/JSON/NDJSON queries run client-side via DuckDB WASM; Excel/SQLite/DuckDB documents automatically run on the server. Open it from any queryable file via the Query button in the file detail view.

Sandboxing & limits

User SQL never touches the host. Referenced documents are downloaded and materialized into an in-memory DuckDB instance during setup; before any user SQL executes, external access is disabled, the local filesystem is turned off, and the configuration is locked. SQLite files are converted in an isolated bridge instance — the instance that runs your query never loads the sqlite extension, closing its direct-I/O escape hatch. Attempts to read host paths, fetch URLs, ATTACH, COPY ... TO, or change settings fail with a VALIDATION_ERROR.

Server-side knobs (environment variables on the daemon/server):

Variable Default Meaning
AGENT_FS_SQL_TIMEOUT_MS 30000 Query timeout (the query is interrupted)
AGENT_FS_SQL_MAX_FILE_BYTES 268435456 (256 MB) Per-document size cap
AGENT_FS_SQL_MEMORY_LIMIT 512MB DuckDB memory limit

maxRows caps the returned rows (default 1000, max 10000); truncated: true signals there were more.

Note: the first query that touches an .xlsx or SQLite document downloads the DuckDB excel/sqlite extension on the server (cached in ~/.duckdb afterwards), so the server needs outbound network access once.