* feat: sql command — DuckDB queries over stored documents (CLI, API, MCP, UI) Adds a sandboxed `sql` op that runs DuckDB SQL over drive documents: csv, tsv, parquet, xlsx, json, ndjson/jsonl (text formats also .gz), sqlite (.db/.sqlite/.sqlite3, tables exposed as <name>.<table>), and .duckdb. Documents are referenced by quoted drive-path literals (SELECT * FROM '/data/sales.csv'), bound as named tables, or format- overridden to make any document queryable (-t logs=/raw/data.txt:csv). Security: the sqlite extension bypasses DuckDB's allowed_directories sandbox (own file I/O), so all inputs are materialized into in-memory tables during setup — sqlite via an isolated bridge instance converted to parquet — then the local filesystem is disabled, external access cut, and configuration locked before any user SQL runs. Interrupt-based timeout, per-file size cap, and memory limit are env-tunable. - core: sql op + registry/RBAC (viewer) + data-file mime types - cli: `agent-fs sql` with repeatable -t bindings, stdin, --max-rows, aligned table output; @duckdb/node-api marked external in the bundle - server/mcp: exposed automatically via ops dispatch + tool registry - live: SQL workbench at /sql/~/:orgId/:driveId — DuckDB-WASM in the browser for csv/tsv/parquet/json, automatic server fallback for xlsx/sqlite/duckdb; results table, SVG charts, CSV/JSON export, "Query" button on queryable files - e2e: 15 new tests (CLI + raw API + MCP) against MinIO — suite 96/96 - docs: docs/sql.md + landing registration + SKILL.md; versions synced to 0.8.2 * feat(live): shared virtualized DataGrid + tabular/db file previews + SQL UX Adds a shared DataGrid (TanStack Table + react-virtual) used by both the SQL results panel and a new default preview for tabular/data documents, plus a batch of SQL workbench UX fixes from QA. DataGrid & previews: - live/src/components/data-grid/DataGrid.tsx — headless, virtualized, themed grid (sort, resize, type tooltips, dimmed NULLs); replaces the hand-rolled results table. - TablePreviewViewer — csv/tsv/ndjson/parquet/xlsx render as a grid by default via the SQL engine (server-side, so previews work even when the raw-bytes presigned fetch is CORS-blocked); csv/tsv/ndjson keep a Source toggle. - DatabasePreviewViewer — sqlite/duckdb files list their tables and preview the selected one in the grid. - FileViewer: route tabular/db files to these previews; office binaries (doc/docx/ppt/pptx/xls/odt…) no longer render as garbled text — they fall back to the download view. SQL workbench: - Engine control is now a Browser/Server toggle with tooltips, reflecting the effective engine; Browser disabled (with reason) when unsupported. - Browser-load size estimate; sizes on doc chips and preview footers. - sqlite/duckdb seeding introspects tables → seeds a working `schema.table` query (fixes the path-literal error on database files). - Cmd/Ctrl+Enter runs via Monaco addAction (reliable keybinding). - Editor autocompletes bound table and `schema.table` names. Misc: - Switching connection/org/drive resets to `/`. - Correct the MinIO-gated op-registry count assertion (now 29, includes sql). * chore: bump version to 0.9.0 (sql feature is a minor release) * docs(landing): SEO/GEO for the SQL docs page - sitemap.xml + sitemap.md: add /docs/sql (IndexNow submits it on deploy, since it parses sitemap.xml). - DocsPage: per-document SEO meta — unique <title>, description, canonical, Open Graph, Twitter, and TechArticle JSON-LD — so every doc page (including sql) is independently indexable and answer-engine citable. - Re-sync public/docs/openapi.json to the 0.9.0 spec. llms.txt already lists the SQL page (Reference + core commands); public/docs/sql.md is synced from docs/sql.md by generate:md. * fix(sql): address PR review — path-literal scoping, gzip, dup columns, cache, engine routing Resolves the 6 P2 review findings: core (packages/core/src/ops/sql.ts): - Only rewrite path literals in FROM/JOIN table position, never value-position strings — `WHERE x = '/data/sales.csv'` is no longer clobbered. - Honor a real .gz suffix even when the format is explicitly overridden, so the temp file keeps its .gz extension and DuckDB decompresses it. - Uniquify duplicate result column names (e.g. two projected `id`s) so building row objects never drops data by key collision. live (sql-engine): - formatForPath strips an optional .gz suffix (and deriveTableName too); the document picker adds .gz globs — gzipped text docs are pickable/seedable. - Browser-engine file registration keys on the current file revision (stat), so an edited doc re-registers instead of serving a stale buffer. - canRunInBrowser is query-aware: a FROM/JOIN path literal that isn't a bound doc routes to the server (only it auto-binds bare path literals). + tests for path-literal scoping, gzip override, and duplicate columns. * fix(sql): apply table-position rewrite + stale-view cleanup to the WASM engine Second review pass flagged the browser engine still had two issues the server fix didn't cover: - rewritePathLiterals now only rewrites FROM/JOIN path literals (matching the server), so a value-position string like `WHERE source = '/data/sales.csv'` is no longer clobbered in the browser engine. - The persistent WASM db kept views from earlier runs; drop views that aren't bound in the current run (removed docs, org/drive switch) before querying, so a stale table name can't be selected out of scope. Verified in headless Chrome: a seeded `FROM '/qa/sales.csv'` query runs in the wasm engine (badge "wasm", 5 rows) with the new rewrite/view logic. * fix(sql): WASM duplicate columns + preview stale-response guards Third review pass: - duckdb.ts: the browser engine built rows keyed by Arrow field name, so duplicate output columns collided. Uniquify names and read cells by column index (getChildAt) so a query projecting two `id`s keeps both — matching the server-side fix. - DatabasePreviewViewer / TablePreviewViewer: guard async loads with a monotonic request token so a slow response for a previously-selected file/table can't overwrite the current preview (switching files or db tables quickly). * fix(sql): apply the query timeout across the whole operation, not just the query Fourth review pass: the deadline was armed only for the query phase, leaving downloads, extension loads, the sqlite->parquet bridge, and table materialization unbounded. Arm the timer before any work so it covers the full operation — it interrupts the active DuckDB connection (extension load / materialization / query) and a checkTimeout() guard stops between phases (downloads and the bridge, where there's no connection to interrupt). The sqlite bridge also checks the deadline between tables. * fix(sql): drop stale WASM files + guard workbench runs against stale results Fifth review pass: - duckdb.ts: ensureRegistered now also drops registered file buffers that aren't bound in the current run (removed docs, org/drive switch), not just stale views — so an out-of-scope buffer can't be read. - SqlPage: the workbench run() guards its result/error/running updates with a monotonic run token, so a slow earlier run can't clobber a newer one (rapid re-runs or an edited query), matching the preview viewers. * fix(sql): interrupt the sqlite bridge connection on timeout Sixth review pass: the sqlite->parquet bridge runs on its own connection, so the deadline could interrupt the main connection but not an in-flight bridge COPY. Register the bridge connection with the timer so a timeout interrupts it too (unregistered before close), completing timeout coverage across every phase. * fix(sql): abort in-flight S3 downloads when the deadline expires Seventh review pass: downloads were the last phase the timeout couldn't interrupt — only checked between files. Thread an AbortSignal from the deadline timer into AgentS3Client.getObject so an in-flight download is aborted on timeout. Every phase (download, extension load, sqlite bridge, materialization, query) is now interruptible.
5.8 KiB
SQL Queries
Run DuckDB SQL directly against documents stored in agent-fs — CSV, TSV, Parquet, Excel, JSON, NDJSON, SQLite databases, and DuckDB databases. Available as a CLI command (agent-fs sql), an API op (sql), and an MCP tool.
agent-fs sql "SELECT category, sum(amount) FROM '/finance/2026.csv' GROUP BY category"
Supported formats
| Format | Extensions | Notes |
|---|---|---|
| CSV | .csv, .csv.gz |
Delimiter, header, and types auto-detected |
| TSV | .tsv, .tab, .tsv.gz |
Tab-delimited |
| Parquet | .parquet |
|
| Excel | .xlsx |
First sheet; header auto-detected |
| JSON | .json, .json.gz |
Arrays of objects or single objects |
| NDJSON | .ndjson, .jsonl (+ .gz) |
Newline-delimited JSON |
| SQLite | .db, .sqlite, .sqlite3 |
All tables exposed — see below |
| DuckDB | .duckdb |
Tables from the main schema |
Referencing documents
There are two ways to make a document queryable:
1. Path literals (file formats)
Reference any stored CSV/TSV/Parquet/Excel/JSON/NDJSON document by its drive path as a quoted string, exactly like DuckDB's own file syntax:
agent-fs sql "SELECT * FROM '/data/sales.csv' WHERE amount > 100"
agent-fs sql "SELECT a.id FROM '/data/a.parquet' a JOIN '/data/b.ndjson' b USING (id)"
Only string literals that resolve to an existing document in the drive are bound — other strings are left untouched, so WHERE name = 'report.csv' works as expected.
2. Named table bindings
Bind documents to table names with --table (repeatable). Required for SQLite/DuckDB databases, and useful for readable multi-file queries:
agent-fs sql "SELECT s.name, t.tag FROM sales s JOIN tags t ON s.id = t.id" \
-t sales=/data/sales.csv -t tags=/data/tags.parquet
SQLite and DuckDB databases expose every table under the binding name as a schema:
# /backups/app.db has tables `users` and `orders`
agent-fs sql "SELECT u.email, count(*) FROM app.users u JOIN app.orders o ON o.user_id = u.id GROUP BY 1" \
-t app=/backups/app.db
Making any document SQL-able (format override)
If a document doesn't have a recognized extension, override the format with a :format suffix (CLI) or { path, format } (API/MCP):
agent-fs sql "SELECT sum(a) FROM logs" -t logs=/raw/data.txt:csv
Formats: csv, tsv, parquet, xlsx, json, ndjson, sqlite, duckdb.
CLI usage
agent-fs sql [query] [options]
-t, --table <name=path[:format]> Bind a document as a named table (repeatable)
--max-rows <n> Max rows to return (default: 1000, max: 10000)
--json Output the raw result object
The query can also be piped via stdin:
echo "SELECT count(*) FROM '/data/events.ndjson'" | agent-fs sql
Multi-statement queries are supported — the result of the last statement is returned:
agent-fs sql "CREATE TABLE big AS SELECT * FROM '/data/sales.csv' WHERE amount > 100; SELECT count(*) FROM big"
API usage
sql is a standard op on the dispatch endpoint:
curl -X POST http://localhost:7433/orgs/{orgId}/ops \
-H "Authorization: Bearer <api-key>" \
-H "Content-Type: application/json" \
-d '{
"op": "sql",
"driveId": "<driveId>",
"query": "SELECT s.name, t.tag FROM sales s JOIN tags t ON s.id = t.id",
"tables": {
"sales": "/data/sales.csv",
"tags": { "path": "/data/tags.parquet", "format": "parquet" }
},
"maxRows": 1000
}'
Response:
{
"columns": [{ "name": "name", "type": "VARCHAR" }, { "name": "tag", "type": "VARCHAR" }],
"rows": [{ "name": "alpha", "tag": "aa" }],
"rowCount": 1,
"truncated": false,
"files": [
{ "table": "sales", "path": "/data/sales.csv", "format": "csv" },
{ "table": "tags", "path": "/data/tags.parquet", "format": "parquet" }
],
"elapsedMs": 42
}
Value encoding: BIGINTs outside JavaScript's safe integer range and timestamps/dates/UUIDs are returned as strings; decimals as numbers; lists and structs as JSON arrays/objects; blobs as <blob N bytes> placeholders. The op requires the viewer role.
The same op is exposed as the sql MCP tool with identical arguments.
Web UI
The live UI has a SQL workbench at /sql/~/{orgId}/{driveId} — pick documents, write queries (Cmd+Enter to run), and view results as a table or quick chart. CSV/TSV/Parquet/JSON/NDJSON queries run client-side via DuckDB WASM; Excel/SQLite/DuckDB documents automatically run on the server. Open it from any queryable file via the Query button in the file detail view.
Sandboxing & limits
User SQL never touches the host. Referenced documents are downloaded and materialized into an in-memory DuckDB instance during setup; before any user SQL executes, external access is disabled, the local filesystem is turned off, and the configuration is locked. SQLite files are converted in an isolated bridge instance — the instance that runs your query never loads the sqlite extension, closing its direct-I/O escape hatch. Attempts to read host paths, fetch URLs, ATTACH, COPY ... TO, or change settings fail with a VALIDATION_ERROR.
Server-side knobs (environment variables on the daemon/server):
| Variable | Default | Meaning |
|---|---|---|
AGENT_FS_SQL_TIMEOUT_MS |
30000 |
Query timeout (the query is interrupted) |
AGENT_FS_SQL_MAX_FILE_BYTES |
268435456 (256 MB) |
Per-document size cap |
AGENT_FS_SQL_MEMORY_LIMIT |
512MB |
DuckDB memory limit |
maxRows caps the returned rows (default 1000, max 10000); truncated: true signals there were more.
Note: the first query that touches an .xlsx or SQLite document downloads the DuckDB excel/sqlite extension on the server (cached in ~/.duckdb afterwards), so the server needs outbound network access once.