skills/firecrawl__cli
1
0
Fork 0
mirror of https://github.com/firecrawl/cli.git synced 2026-09-14 17:30:55 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
add-interactiveLiveView
firecrawl__cli/README.md
T

773 lines
26 KiB
Markdown
Raw Permalink Blame History

# 🔥 Firecrawl CLI
Command-line interface for Firecrawl. Scrape, crawl, and extract data from any website directly from your terminal.
## Installation
```bash
npm install -g firecrawl-cli
```
Or set up everything in one command (install CLI globally, authenticate, and add skills):
```bash
npx -y firecrawl-cli@latest init --all --browser
```
- `--all` installs the firecrawl skill to every detected AI coding agent
- `--browser` opens the browser for Firecrawl authentication automatically
### Setup Skills and MCP
If you are using an AI coding agent like Claude Code, you can also install the skill individually with:
```bash
firecrawl setup skills
```
To install the Firecrawl MCP server into your editors (Cursor, Claude Code, VS Code, etc.):
```bash
firecrawl setup mcp
```
Or directly via npx:
```bash
npx skills add firecrawl/cli
npx add-mcp "npx -y firecrawl-mcp" --name firecrawl
```
## Quick Start
Just run a command - the CLI will prompt you to authenticate if needed:
```bash
firecrawl https://example.com
```
## Authentication
On first run, you'll be prompted to authenticate:
```
🔥 firecrawl cli
Turn websites into LLM-ready data
Welcome! To get started, authenticate with your Firecrawl account.
1. Login with browser (recommended)
2. Enter API key manually
Tip: You can also set FIRECRAWL_API_KEY environment variable
Enter choice [1/2]:
```
### Authentication Methods
```bash
# Interactive (prompts automatically when needed)
firecrawl
# Browser login
firecrawl login
# Direct API key
firecrawl login --api-key fc-your-api-key
# Environment variable
export FIRECRAWL_API_KEY=fc-your-api-key
# Per-command API key
firecrawl scrape https://example.com --api-key fc-your-api-key
```
### Self-hosted / Local Development
For self-hosted Firecrawl instances or local development, use the `--api-url` option:
```bash
# Use a local Firecrawl instance (no API key required)
firecrawl --api-url http://localhost:3002 scrape https://example.com
# Or set via environment variable
export FIRECRAWL_API_URL=http://localhost:3002
firecrawl scrape https://example.com
# Self-hosted with API key
firecrawl --api-url https://firecrawl.mycompany.com --api-key fc-xxx scrape https://example.com
```
When using a custom API URL (anything other than `https://api.firecrawl.dev`), authentication is automatically skipped, allowing you to use local instances without an API key.
---
## Commands
### `scrape` - Scrape URLs
Extract content from any webpage. Pass multiple URLs to scrape them concurrently -- each result is saved to `.firecrawl/` automatically.
```bash
# Basic usage (outputs markdown)
firecrawl https://example.com
firecrawl scrape https://example.com
# Get raw HTML
firecrawl https://example.com --html
firecrawl https://example.com -H
# Multiple formats (outputs JSON)
firecrawl https://example.com --format markdown,links,images
# Save to file
firecrawl https://example.com -o output.md
firecrawl https://example.com --format json -o data.json --pretty
# Multiple URLs (scraped concurrently, each saved to .firecrawl/)
firecrawl scrape https://firecrawl.dev https://firecrawl.dev/blog https://docs.firecrawl.dev
```
#### Scrape Options
| Option | Description |
| -------------------------- | ------------------------------------------------------- |
| `-f, --format <formats>` | Output format(s), comma-separated |
| `-H, --html` | Shortcut for `--format html` |
| `-S, --summary` | Shortcut for `--format summary` |
| `--only-main-content` | Extract only main content (removes navs, footers, etc.) |
| `--wait-for <ms>` | Wait time before scraping (for JS-rendered content) |
| `--screenshot` | Take a screenshot |
| `--full-page-screenshot` | Take a full page screenshot |
| `--include-tags <tags>` | Only include specific HTML tags |
| `--exclude-tags <tags>` | Exclude specific HTML tags |
| `--max-age <milliseconds>` | Maximum age of cached content in milliseconds |
| `-o, --output <path>` | Save output to file |
| `--json` | Output as JSON format |
| `--pretty` | Pretty print JSON output |
| `--timing` | Show request timing info |
#### Available Formats
| Format | Description |
| ---------------- | ---------------------------- |
| `markdown` | Clean markdown (default) |
| `html` | Cleaned HTML |
| `rawHtml` | Original HTML |
| `links` | All links on the page |
| `images` | All images on the page |
| `screenshot` | Screenshot as base64 |
| `summary` | AI-generated summary |
| `json` | Structured JSON extraction |
| `changeTracking` | Track changes on the page |
| `attributes` | Page attributes and metadata |
| `branding` | Brand identity extraction |
#### Examples
```bash
# Extract only main content as markdown
firecrawl https://blog.example.com --only-main-content
# Wait for JS to render, then scrape
firecrawl https://spa-app.com --wait-for 3000
# Get all links from a page
firecrawl https://example.com --format links
# Screenshot + markdown
firecrawl https://example.com --format markdown --screenshot
# Extract specific elements only
firecrawl https://example.com --include-tags article,main
# Exclude navigation and ads
firecrawl https://example.com --exclude-tags nav,aside,.ad
```
---
### `search` - Search the web
Search the web and optionally scrape content from search results.
```bash
# Basic search
firecrawl search "firecrawl web scraping"
# Limit results
firecrawl search "AI news" --limit 10
# Search news sources
firecrawl search "tech startups" --sources news
# Search images
firecrawl search "landscape photography" --sources images
# Multiple sources
firecrawl search "machine learning" --sources web,news,images
# Filter by category (GitHub, research papers, PDFs)
firecrawl search "web scraping python" --categories github
firecrawl search "transformer architecture" --categories research
firecrawl search "machine learning" --categories github,research
# Time-based search
firecrawl search "AI announcements" --tbs qdr:d # Past day
firecrawl search "tech news" --tbs qdr:w # Past week
# Location-based search
firecrawl search "restaurants" --location "San Francisco,California,United States"
firecrawl search "local news" --country DE
# Search and scrape results
firecrawl search "firecrawl tutorials" --scrape
firecrawl search "API documentation" --scrape --scrape-formats markdown,links
# Output as pretty JSON
firecrawl search "web scraping"
```
#### Search Options
| Option | Description |
| ---------------------------- | ------------------------------------------------------------------------------------------- |
| `--limit <n>` | Maximum results (default: 5, max: 100) |
| `--sources <sources>` | Comma-separated: `web`, `images`, `news` (default: web) |
| `--categories <categories>` | Comma-separated: `github`, `research`, `pdf` |
| `--tbs <value>` | Time filter: `qdr:h` (hour), `qdr:d` (day), `qdr:w` (week), `qdr:m` (month), `qdr:y` (year) |
| `--location <location>` | Geo-targeting (e.g., "Germany", "San Francisco,California,United States") |
| `--country <code>` | ISO country code (default: US) |
| `--timeout <ms>` | Timeout in milliseconds (default: 60000) |
| `--ignore-invalid-urls` | Exclude URLs invalid for other Firecrawl endpoints |
| `--scrape` | Enable scraping of search results |
| `--scrape-formats <formats>` | Scrape formats when `--scrape` enabled (default: markdown) |
| `--only-main-content` | Include only main content when scraping (default: true) |
| `-o, --output <path>` | Save to file |
| `--json` | Output as compact JSON |
#### Examples
```bash
# Research a topic with recent results
firecrawl search "React Server Components" --tbs qdr:m --limit 10
# Find GitHub repositories
firecrawl search "web scraping library" --categories github --limit 20
# Search and get full content
firecrawl search "firecrawl documentation" --scrape --scrape-formats markdown --json -o results.json
# Find research papers
firecrawl search "large language models" --categories research --json
# Search with location targeting
firecrawl search "best coffee shops" --location "Berlin,Germany" --country DE
# Get news from the past week
firecrawl search "AI startups funding" --sources news --tbs qdr:w --limit 15
```
---
### `map` - Discover all URLs on a website
Quickly discover all URLs on a website without scraping content.
```bash
# List all URLs (one per line)
firecrawl map https://example.com
# Output as JSON
firecrawl map https://example.com --json
# Search for specific URLs
firecrawl map https://example.com --search "blog"
# Limit results
firecrawl map https://example.com --limit 500
```
#### Map Options
| Option | Description |
| --------------------------- | --------------------------------- |
| `--limit <n>` | Maximum URLs to discover |
| `--search <query>` | Filter URLs by search query |
| `--sitemap <mode>` | `include`, `skip`, or `only` |
| `--include-subdomains` | Include subdomains |
| `--ignore-query-parameters` | Dedupe URLs with different params |
| `--timeout <seconds>` | Request timeout |
| `--json` | Output as JSON |
| `-o, --output <path>` | Save to file |
#### Examples
```bash
# Find all product pages
firecrawl map https://shop.example.com --search "product"
# Get sitemap URLs only
firecrawl map https://example.com --sitemap only
# Save URL list to file
firecrawl map https://example.com -o urls.txt
# Include subdomains
firecrawl map https://example.com --include-subdomains --limit 1000
```
---
### `crawl` - Crawl an entire website
Crawl multiple pages from a website.
```bash
# Start a crawl (returns job ID)
firecrawl crawl https://example.com
# Wait for crawl to complete
firecrawl crawl https://example.com --wait
# With progress indicator
firecrawl crawl https://example.com --wait --progress
# Check crawl status
firecrawl crawl <job-id>
# Limit pages
firecrawl crawl https://example.com --limit 100 --max-depth 3
```
#### Crawl Options
| Option | Description |
| --------------------------- | ---------------------------------------- |
| `--wait` | Wait for crawl to complete |
| `--progress` | Show progress while waiting |
| `--limit <n>` | Maximum pages to crawl |
| `--max-depth <n>` | Maximum crawl depth |
| `--include-paths <paths>` | Only crawl matching paths |
| `--exclude-paths <paths>` | Skip matching paths |
| `--sitemap <mode>` | `include`, `skip`, or `only` |
| `--allow-subdomains` | Include subdomains |
| `--allow-external-links` | Follow external links |
| `--crawl-entire-domain` | Crawl entire domain |
| `--ignore-query-parameters` | Treat URLs with different params as same |
| `--delay <ms>` | Delay between requests |
| `--max-concurrency <n>` | Max concurrent requests |
| `--timeout <seconds>` | Timeout when waiting |
| `--poll-interval <seconds>` | Status check interval |
#### Examples
```bash
# Crawl blog section only
firecrawl crawl https://example.com --include-paths /blog,/posts
# Exclude admin pages
firecrawl crawl https://example.com --exclude-paths /admin,/login
# Crawl with rate limiting
firecrawl crawl https://example.com --delay 1000 --max-concurrency 2
# Deep crawl with high limit
firecrawl crawl https://example.com --limit 1000 --max-depth 10 --wait --progress
# Save results
firecrawl crawl https://example.com --wait -o crawl-results.json --pretty
```
---
### `credit-usage` - Check your credits
```bash
# Show credit usage
firecrawl credit-usage
# Output as JSON
firecrawl credit-usage --json --pretty
```
---
### `agent` - AI-powered web data extraction
Run an AI agent that autonomously browses and extracts structured data from the web based on natural language prompts.
> **Note:** Agent tasks typically take **2 to 5 minutes** to complete, and sometimes longer for complex extractions. Use sparingly and consider `--max-credits` to limit costs.
```bash
# Basic usage (returns job ID immediately)
firecrawl agent "Find the pricing plans for Firecrawl"
# Wait for completion
firecrawl agent "Extract all product names and prices from this store" --wait
# Focus on specific URLs
firecrawl agent "Get the main features listed" --urls https://example.com/features
# Use structured output with JSON schema
firecrawl agent "Extract company info" --schema '{"type":"object","properties":{"name":{"type":"string"},"employees":{"type":"number"}}}'
# Load schema from file
firecrawl agent "Extract product data" --schema-file ./product-schema.json --wait
# Check status of an existing job
firecrawl agent <job-id>
firecrawl agent <job-id> --wait
```
#### Agent Options
| Option | Description |
| --------------------------- | ------------------------------------------------------------- |
| `--urls <urls>` | Comma-separated URLs to focus extraction on |
| `--model <model>` | `spark-1-mini` (default, cheaper) or `spark-1-pro` (accurate) |
| `--schema <json>` | JSON schema for structured output (inline JSON string) |
| `--schema-file <path>` | Path to JSON schema file for structured output |
| `--max-credits <number>` | Maximum credits to spend (job fails if exceeded) |
| `--status` | Check status of existing agent job |
| `--wait` | Wait for agent to complete before returning results |
| `--poll-interval <seconds>` | Polling interval in seconds when waiting (default: 5) |
| `--timeout <seconds>` | Timeout in seconds when waiting (default: no timeout) |
| `-o, --output <path>` | Save output to file |
| `--json` | Output as JSON format |
| `--pretty` | Pretty print JSON output |
#### Examples
```bash
# Research task with timeout
firecrawl agent "Find the top 5 competitors of Notion and their pricing" --wait --timeout 300
# Extract data with cost limit
firecrawl agent "Get all blog post titles and dates" --urls https://blog.example.com --max-credits 100 --wait
# Use higher accuracy model for complex extraction
firecrawl agent "Extract detailed technical specifications" --model spark-1-pro --wait --pretty
# Save structured results to file
firecrawl agent "Extract contact information" --schema-file ./contact-schema.json --wait -o contacts.json --pretty
# Check job status without waiting
firecrawl agent abc123-def456-... --json
# Poll a running job until completion
firecrawl agent abc123-def456-... --wait --poll-interval 10
```
---
### `browser` - Browser sandbox sessions (Beta)
Launch and control cloud browser sessions. By default, commands are sent to agent-browser (pre-installed in every sandbox). Use `--python` or `--node` to run Playwright code directly instead.
```bash
# 1. Launch a session
firecrawl browser launch --stream
# 2. Execute agent-browser commands (default)
firecrawl browser execute "open https://example.com"
firecrawl browser execute "snapshot"
firecrawl browser execute "click @e5"
firecrawl browser execute "scrape"
# 3. Execute Playwright Python or JavaScript
firecrawl browser execute --python "await page.goto('https://example.com'); print(await page.title())"
firecrawl browser execute --node "await page.goto('https://example.com'); await page.title()"
# 4. List sessions
firecrawl browser list
# 5. Close
firecrawl browser close
```
#### Launch Options
| Option | Description |
| ---------------------------- | ------------------------------------------- |
| `--ttl <seconds>` | Total session TTL in seconds (default: 300) |
| `--ttl-inactivity <seconds>` | Inactivity TTL in seconds |
| `--stream` | Enable live view streaming |
| `-o, --output <path>` | Save output to file |
| `--json` | Output as JSON format |
#### Execute Options
| Option | Description |
| --------------------- | ------------------------------------------------------------------ |
| `--python` | Execute as Playwright Python code |
| `--node` | Execute as Playwright JavaScript code |
| `--bash` | Execute bash commands in the sandbox (agent-browser pre-installed) |
| `--session <id>` | Target a specific session (auto-saved on launch) |
| `-o, --output <path>` | Save output to file |
| `--json` | Output as JSON format |
By default (no flag), commands are sent to agent-browser. `--python`, `--node`, and `--bash` are mutually exclusive.
#### Examples
```bash
# agent-browser commands (default mode)
firecrawl browser execute "open https://example.com"
firecrawl browser execute "snapshot"
firecrawl browser execute "click @e5"
firecrawl browser execute "fill @e3 'search query'"
firecrawl browser execute "scrape"
# Playwright Python
firecrawl browser execute --python "await page.goto('https://example.com'); print(await page.title())"
# Playwright JavaScript
firecrawl browser execute --node "await page.goto('https://example.com'); await page.title()"
# Bash (arbitrary commands in the sandbox)
firecrawl browser execute --bash "ls /tmp"
# Launch with extended TTL
firecrawl browser launch --ttl 900 --ttl-inactivity 120
# JSON output
firecrawl browser execute --json "snapshot"
```
---
### `config` - Configure settings
```bash
# Configure with custom API URL
firecrawl config --api-url https://firecrawl.mycompany.com
firecrawl config --api-url http://localhost:3002 --api-key fc-xxx
```
### `view-config` - View current configuration
```bash
# View current configuration and authentication status
firecrawl view-config
```
Shows authentication status and stored credentials location.
---
### `login` / `logout`
```bash
# Login
firecrawl login
firecrawl login --method browser
firecrawl login --method manual
firecrawl login --api-key fc-xxx
# Login to self-hosted instance
firecrawl login --api-url https://firecrawl.mycompany.com
firecrawl login --api-url http://localhost:3002 --api-key fc-xxx
# Logout
firecrawl logout
```
---
## Global Options
These options work with any command:
| Option | Description |
| --------------------- | ------------------------------------------------------ |
| `--status` | Show version, auth, concurrency, and credits |
| `-k, --api-key <key>` | Use specific API key |
| `--api-url <url>` | Use custom API URL (for self-hosted/local development) |
| `-V, --version` | Show version |
| `-h, --help` | Show help |
### Check Status
```bash
firecrawl --status
```
```
🔥 firecrawl cli v1.4.0
● Authenticated via stored credentials
Concurrency: 0/100 jobs (parallel scrape limit)
Credits: 500,000 / 1,000,000 (50% left this cycle)
```
---
## Output Handling
### Stdout vs File
```bash
# Output to stdout (default)
firecrawl https://example.com
# Pipe to another command
firecrawl https://example.com | head -50
# Save to file
firecrawl https://example.com -o output.md
# JSON output
firecrawl https://example.com --format links --pretty
```
### Format Behavior
- **Single format**: Outputs raw content (markdown text, HTML, etc.)
- **Multiple formats**: Outputs JSON with all requested data
```bash
# Raw markdown output
firecrawl https://example.com --format markdown
# JSON output with multiple formats
firecrawl https://example.com --format markdown,links,images
```
---
## Tips & Tricks
### Scrape multiple URLs
```bash
# Just pass multiple URLs -- results are saved to .firecrawl/
firecrawl scrape https://firecrawl.dev https://firecrawl.dev/blog https://docs.firecrawl.dev
```
### Combine with other tools
```bash
# Extract links and process with jq
firecrawl https://example.com --format links | jq '.links[].url'
# Convert to PDF (with pandoc)
firecrawl https://example.com | pandoc -o document.pdf
# Search within scraped content
firecrawl https://example.com | grep -i "keyword"
```
### CI/CD Usage
```bash
# Set API key via environment
export FIRECRAWL_API_KEY=${{ secrets.FIRECRAWL_API_KEY }}
firecrawl crawl https://docs.example.com --wait -o docs.json
# Use self-hosted instance
export FIRECRAWL_API_URL=${{ secrets.FIRECRAWL_API_URL }}
firecrawl scrape https://example.com -o output.md
```
---
## `download` - Bulk Site Download
A convenience command that combines `map` + `scrape` to save a site as local files. Maps the site first to discover pages, then scrapes each one into nested directories under `.firecrawl/`. All [scrape options](#scrape-options) work with download. Run without flags for an interactive wizard that walks you through format, screenshot, and path selection.
```bash
# Interactive wizard (picks format, screenshots, paths for you)
firecrawl download https://docs.firecrawl.dev
# Download with screenshots
firecrawl download https://docs.firecrawl.dev --screenshot --limit 20 -y
# Full page screenshots
firecrawl download https://docs.firecrawl.dev --full-page-screenshot --limit 20 -y
# Multiple formats (each saved as its own file per page)
firecrawl download https://docs.firecrawl.dev --format markdown,links --screenshot --limit 20 -y
# Creates per page: index.md + links.txt + screenshot.png
# Download as HTML
firecrawl download https://docs.firecrawl.dev --html --limit 20 -y
# Main content only
firecrawl download https://docs.firecrawl.dev --only-main-content --limit 50 -y
# Filter to specific paths
firecrawl download https://docs.firecrawl.dev --include-paths "/features,/sdks"
# Skip localized pages
firecrawl download https://docs.firecrawl.dev --exclude-paths "/zh,/ja,/fr,/es,/pt-BR"
# Include subdomains
firecrawl download https://firecrawl.dev --allow-subdomains
# Combine everything
firecrawl download https://docs.firecrawl.dev \
--include-paths "/features,/sdks" \
--exclude-paths "/zh,/ja,/fr,/es,/pt-BR" \
--only-main-content \
--screenshot \
-y
```
#### Download Options
| Option | Description |
| ------------------------- | ---------------------------------------------- |
| `--limit <number>` | Max pages to download |
| `--search <query>` | Filter pages by search query |
| `--include-paths <paths>` | Only download matching paths (comma-separated) |
| `--exclude-paths <paths>` | Skip matching paths (comma-separated) |
| `--allow-subdomains` | Include subdomains when mapping |
| `-y, --yes` | Skip confirmation prompt and wizard |
All [scrape options](#scrape-options) also work with download (formats, screenshots, tags, geo-targeting, etc.)
#### Output Structure
Each format is saved as its own file per page:
```
.firecrawl/
docs.firecrawl.dev/
features/
scrape/
index.md # markdown content
links.txt # one link per line
screenshot.png # actual PNG image
crawl/
index.md
screenshot.png
sdks/
python/
index.md
```
---
## Telemetry
The CLI collects anonymous usage data during authentication to help improve the product:
- CLI version, OS, and Node.js version
- Detect development tools (e.g., Cursor, VS Code, Claude Code)
**No command data, URLs, or file contents are collected via the CLI.**
To disable telemetry, set the environment variable:
```bash
export FIRECRAWL_NO_TELEMETRY=1
```
---
## Documentation
For more details, visit the [Firecrawl Documentation](https://docs.firecrawl.dev).
Reference in New Issue View Git Blame Copy Permalink