128 Commits

Author SHA1 Message Date
Developers Digest 8cc098c3fe Add multi-page examples and clarify flag
Add examples showing how to use --scrape-urls to monitor multiple pages (e.g. pricing, docs, changelog) in README.md and skills/firecrawl-cli/SKILL.md. Also update the CLI help text in src/commands/monitor.ts to clarify the --scrape-urls description to "Comma-separated page URLs to scrape on each check."
2026-05-25 14:01:16 -04:00
Developers Digest d13a7afaa4 Update monitor.ts 2026-05-25 13:47:24 -04:00
Developers Digest 45f7987618 Merge branch 'jonathan/monitoring-2' of https://github.com/firecrawl/cli into jonathan/monitoring-2 2026-05-25 13:45:05 -04:00
Developers Digest 45690534ed Add monitor goal/page flags, tests, and docs
Export and update monitor creation helpers to accept a --goal and single --page shorthand (constructs urls from page when provided) and include goal in the request body. Add CLI options and help text for --goal and --page, support goal on monitor update, and adjust example usage. Add unit tests for buildCreateBody covering goal and page behavior. Update README and SKILL docs to document monitor usage and guidance. Bump package version to 1.18.1.
2026-05-25 13:43:23 -04:00
Developers Digest 3ed98eec7c Add monitor goal/page flags, tests, and docs
Export and update monitor creation helpers to accept a --goal and single --page shorthand (constructs urls from page when provided) and include goal in the request body. Add CLI options and help text for --goal and --page, support goal on monitor update, and adjust example usage. Add unit tests for buildCreateBody covering goal and page behavior. Update README and SKILL docs to document monitor usage and guidance. Bump package version to 1.18.1.
2026-05-25 13:20:48 -04:00
Developers Digest 623574bad3 monitor create: add --goal flag 2026-05-22 17:32:25 -04:00
Nicolas f873e5bfd2 Merge branch 'main' into monitoring 2026-05-17 12:48:54 -04:00
Eric Ciarla 32309be1c8 Add workflows to cli! 2026-05-15 09:44:34 -04:00
Nicolas 4e0337677d Merge branch 'main' into nsc/search-feedback 2026-05-14 14:50:31 -04:00
Nicolas 60925e5e9f Nick: search feedback 2026-05-14 14:33:29 -04:00
Developers Digest e251e6f012 bump @mendable/firecrawl-js to 4.22.2; document why monitor still uses raw fetch 2026-05-12 11:22:33 -04:00
Developers Digest 84140d9af1 monitor: rename --urls to --scrape-urls to pair with --crawl-url 2026-05-12 11:13:18 -04:00
Developers Digest 19ae52cc73 monitor: accept JSON via positional file or piped stdin; document in skill 2026-05-12 11:06:00 -04:00
Developers Digest 933930b1fe monitor: accept JSON file/stdin and remove --body
Allow monitor create/update commands to consume a JSON payload from a positional file argument or piped stdin (use '-' to force stdin). Introduce async readJsonPayload helper, refactor buildCreateBody to no longer take a raw body string, and try JSON input first before falling back to flag-built payloads. Update CLI signatures, help text, and examples accordingly, and add the monitor entry to the SKILL.md command list.
2026-05-12 11:05:40 -04:00
Developers Digest 6890170328 rename monitor --status to --state to avoid global flag collision 2026-05-12 10:59:59 -04:00
Developers Digest e576ca35bc add monitor command (create/list/get/update/delete/run/checks) 2026-05-12 10:57:48 -04:00
Developers Digest db445ae1f6 remove unsupported scrape webhook flag 2026-05-04 11:08:10 -04:00
Developers Digest f737873e9d Add webhook/cancel support and JSON parsing
Add support for webhook configuration and canceling jobs across agent, crawl, and scrape commands. Introduce JSON parsing helpers and CLI options for schema, actions, scrape-options, proxy, and webhook (inline or file), and validate payload shapes. Extend types to include webhook/cancel fields and normalize agent statuses to include "cancelled", updating polling and status handling accordingly. Propagate webhook/scrapeOptions through command handlers and improve CLI behavior for global --status handling and format resolution.
2026-05-04 10:56:23 -04:00
Gergő Móricz 3c6ac28a0c Merge pull request #103 from firecrawl/mog/parse-api
feat: parse command (ENG-4830)
2026-04-27 17:24:38 +02:00
Abimael Martell d0145f4e62 feat: add --lockdown flag to scrape (#104)
* feat: add --lockdown-mode flag to scrape

* rename --lockdown-mode to --lockdown
2026-04-22 15:19:02 -07:00
mogery 16e09f6339 feat: parse command 2026-04-21 18:15:39 +01:00
Developers Digest bc2c6d93a8 Switch agent scaffold to web-agent repo
Update vendored scaffold references and default remote to use firecrawl/web-agent. Adjusted header comments in create.ts, create-flow.ts, credentials.ts, manifest.ts, scaffold.ts, and ui.ts, and changed DEFAULT_REMOTE in manifest.ts from 'firecrawl/firecrawl-agent' to 'firecrawl/web-agent' so the scaffolder clones the web-agent repo for templates.
2026-04-16 12:16:44 -04:00
Developers Digest 47af8daf5c chore: friendlier error when default clone fails 2026-04-15 13:58:19 -04:00
Developers Digest b7c16d0df6 feat: vendor agent scaffolder into firecrawl-cli
Replaces the `npx firecrawl-agent-cli` delegator with an in-process
scaffolder vendored from firecrawl/firecrawl-agent. No separate npm
package for the agent CLI — the root firecrawl-cli clones the public
agent repo at runtime to fetch templates + agent-core.

- Vendor under src/utils/agent-scaffold/ (manifest, scaffold,
  credentials, ui, create-flow). Upstream-tracked; keep in sync.
- manifest.ts: strip the "bundled next to CLI" branch (doesn't apply
  here), always clone the public repo. Nested-manifest fallback
  handles the current `.internal/cli/agent-manifest.json` location.
- Extract handleCreate from upstream init.ts; drop the commander
  wrapper since the root CLI owns the command surface.
- create.ts now lazy-imports the scaffolder so non-create paths stay
  snappy.

Smoke-tested end-to-end against a local agent repo via --from: next
template scaffolds correctly with agent-core, .env.local, and
expected file layout.
2026-04-15 13:56:17 -04:00
Developers Digest 39bd5b7e48 feat: hidden firecrawl create agent command
Wires a hidden `firecrawl create <kind>` command tree into the root CLI.
Only the `agent` kind is registered today; it delegates to
`npx -y firecrawl-agent-cli create` with full flag passthrough so the
agent repo remains the single source of truth for templates.

The command is hidden from --help until `firecrawl-agent-cli` is on
npm. Flip to visible by removing `{ hidden: true }` in index.ts.

Surface:
  firecrawl create agent [name] [-t next|express|library]
  firecrawl create agent [name] --provider <p> --model <m>
  firecrawl create agent [name] --api-key <fc-key> --key anthropic=sk-...
  firecrawl create agent [name] --from user/repo
2026-04-15 13:23:23 -04:00
Nicolas 3193905aa4 Nick: fixes when using -k 2026-04-13 17:58:57 -03:00
Developers Digest 1b48a69d99 swap interact example to action-oriented search + filter flow 2026-04-10 11:58:53 -04:00
Developers Digest 9d6cc65c4c split next-steps into web vs build sections + friendly install labels
During install, show "Installing core firecrawl skills..." and
"Installing skills to build with firecrawl..." instead of the raw repo
names firecrawl/cli and firecrawl/skills. SKILL_REPO_LABELS map drives
the friendly text.

In the next-steps summary, split into two distinct groupings:
  1. "Connect & interact with the web (direct or in your AI agent)"
     with scrape/search/interact examples showing the natural language
     prompt next to the equivalent CLI command.
  2. "Building with firecrawl?" at the bottom, pointing at the build
     skills with a concrete business example.

MCP + --help entries sit between the two sections as general utilities.
Bumped to 1.14.7.
2026-04-10 11:52:13 -04:00
Developers Digest 6da95bba24 use firecrawl-specific business example in next-steps tip 2026-04-10 11:04:36 -04:00
Developers Digest f91f069382 rephrase build-skills tip with concrete example, drop em dash 2026-04-10 11:01:50 -04:00
Developers Digest 6ef1aa565c add 'start building' tip to next-steps summary 2026-04-10 11:00:19 -04:00
Developers Digest a7aa071f7a tighten next-steps summary and surface skill count
Parse '(Found|Installed) N skills' from captured npx skills output and
sum across both repos so the post-install summary shows
"✓ Installed 14 skills across your AI coding agents" instead of vague
text.

Collapse the next-steps block from 18 lines to 7: drop the 4 example
prompts in favor of one of each (AI prompt + direct CLI), plus the
MCP install hint and the --help pointer. Each entry uses padded labels
("Ask your AI:", "Run direct: ", "Add MCP:    ", "All commands:") so
the commands align in a column.

stepIntegrations and runNonInteractive both thread the count through
to printNextSteps. Tests still mock execSync as undefined, so
parseSkillCount handles the empty case. Bumped to 1.14.3.
2026-04-10 10:25:14 -04:00
Developers Digest c2e4856ef5 add mcp install hint to next-steps summary 2026-04-10 10:21:21 -04:00
Developers Digest fa735ff336 quiet skill install output and add next-steps summary
Replace the noisy 'npx skills add' inherited stdout (banner, +/symlink
table, security report, ~80 lines per repo) with a single line per repo:
"  ✓ firecrawl/cli" / "  ✓ firecrawl/skills". TTY runs get a transient
"↓ <repo>" indicator while the install is in flight; non-TTY runs (CI,
piped) get only the final line.

Replace the trailing "Setup complete!" line with a printNextSteps()
block that surfaces:
  - what just happened (skills installed globally)
  - what firecrawl is for (search, scrape, crawl, interact)
  - 4 example prompts (2 build-skill triggers, 2 direct CLI calls)
  - pointer to firecrawl --help

Both interactive and non-interactive init paths use the same helpers,
so output is consistent regardless of how users enter the flow.
Bumped to 1.14.2.
2026-04-10 10:17:42 -04:00
Developers Digest f5d8c635c9 strip inherited npm_* env vars before nested npx calls
when this cli is launched via 'npx -y firecrawl-cli@VERSION', npm injects
npm_command/npm_lifecycle_event/npm_execpath/INIT_CWD into the process
env. those vars leak into nested execSync('npx -y skills add ...') calls
and cause the loop's second iteration to silently exit after the first
install completes — published 1.14.0 only installs from firecrawl/cli
even though SKILL_REPOS contains both repos.

cleanNpmEnv() in skills-install.ts strips those vars; init.ts and
setup.ts pass it as the env to every nested execSync. added a
regression test that primes process.env with the leaking vars and
asserts every install call sees a clean env. bumped to 1.14.1.
2026-04-10 09:58:30 -04:00
Developers Digest a9748148a8 install skills from firecrawl/skills in addition to firecrawl/cli
onboarding (init) and setup skills now loop over a SKILL_REPOS list
so both the core cli skills and the build skills repo get installed
through the same npx/native paths. buildSkillsInstallArgs and
installSkillsNative take a repo parameter (defaulting to firecrawl/cli
for back compat). tests updated to assert both install invocations.
2026-04-10 09:34:36 -04:00
Developers Digest a7e6712eb5 remove legacy browser references from shared prompts and skill metadata 2026-04-08 22:20:49 -04:00
Developers Digest 2618c20d8d clean up cli: rename instruct to interact, hide browser, group experimental commands
- rename skills/firecrawl-instruct → firecrawl-interact + update all cross-references
- hide deprecated browser command from --help (still works when called directly)
- group download, claude, codex, opencode under `firecrawl experimental` (alias: `firecrawl x`)
- remove browser references from passthrough system prompt quick reference
2026-04-08 22:17:47 -04:00
Developers Digest 8bb27a5ce5 Merge pull request #77 from firecrawl/install-sh
add native skill installer and post-install setup flow
2026-03-25 20:24:10 +00:00
Developers Digest 49c5484f12 add native skill installer and post-install setup flow
- new skills-native.ts: replicates `npx skills add` without requiring
  Node.js — clones repo, copies skills to ~/.agents/skills/, creates
  relative symlinks to detected agent dirs (claude, cursor, etc.)
- setup.ts + init.ts: try npx first, fall back to native installer
- install.sh: after binary installs, prompt to continue with
  `firecrawl init --skip-install` for login + skills + integrations
2026-03-25 16:20:59 -04:00
Nicolas 40b3fdd9a3 Nick: improvements 2026-03-23 17:17:03 -03:00
Nicolas 726d90f677 Nick: 2026-03-23 12:56:44 -03:00
Nicolas c084d98486 Nick: 2026-03-23 12:54:22 -03:00
Nicolas 954d3830e4 Nick: 2026-03-22 21:23:48 -03:00
Nicolas 0bdf2b7426 Nick: 2026-03-22 20:25:44 -03:00
Nicolas 5abd341e30 Nick: updates 2026-03-19 15:44:35 -07:00
Nicolas 8607c7797d Nick: fixes 2026-03-18 16:46:50 -07:00
Nicolas be2547b96f Nick: 2026-03-18 16:26:04 -07:00
Nicolas af20982297 Nick: 2026-03-18 15:55:05 -07:00
Developers Digest 7ba546f5ce add --query flag for query format support 2026-03-12 11:39:46 -04:00