Spider MCP Server
The fastest web crawling, scraping, and browser automation server for AI agents. Gives Claude direct access to the web through 22 tools — crawl sites at 100K+ pages/sec, extract structured data with AI, and control remote browsers with built-in anti-bot bypass.
Why Spider
- Speed — Crawl 100K+ pages per second. Smart request routing picks HTTP or headless Chrome automatically. Streaming responses start delivering data immediately.
- Cost — Pay-per-use credits with no subscription required for core tools. Check your balance anytime with
spider_get_credits. AI tools available with an AI subscription. - Reliability — Anti-bot bypass with fingerprinting and proxy rotation. Browser fleet with automatic fallback across Chrome, Firefox, and more. Built-in retry and stealth escalation.
Quick Start
Claude Code
claude mcp add spider -- npx -y spider-cloud-mcpSet your API key:
export SPIDER_API_KEY="your-api-key"Claude Desktop
Add to claude_desktop_config.json:
{
"mcpServers": {
"spider": {
"command": "npx",
"args": ["-y", "spider-cloud-mcp"],
"env": {
"SPIDER_API_KEY": "your-api-key"
}
}
}
}Cursor
Add to .cursor/mcp.json:
{
"mcpServers": {
"spider": {
"command": "npx",
"args": ["-y", "spider-cloud-mcp"],
"env": {
"SPIDER_API_KEY": "your-api-key"
}
}
}
}Windsurf
Add to ~/.codeium/windsurf/mcp_config.json:
{
"mcpServers": {
"spider": {
"command": "npx",
"args": ["-y", "spider-cloud-mcp"],
"env": {
"SPIDER_API_KEY": "your-api-key"
}
}
}
}Configuration
| Variable | Required | Description | |---|---|---| | SPIDER_API_KEY | Yes | Your Spider API key. Get one at spider.cloud/api-keys |
Tools
Core Tools (8)
These work on pay-per-use credits with no subscription required.
| Tool | Description | |---|---| | spider_crawl | Crawl a website and extract content from multiple pages. Follows links up to a depth/limit. | | spider_scrape | Scrape a single page. Faster and cheaper than crawling when you need one URL. | | spider_search | Search the web. Optionally fetch full page content from results. | | spider_links | Extract all links from a page without fetching content. | | spider_screenshot | Capture a page screenshot as base64 PNG. | | spider_unblocker | Access bot-protected content with advanced anti-bot bypass. | | spider_transform | Convert HTML to markdown or text without making web requests. | | spider_get_credits | Check your API credit balance. |
AI Tools (5)
Natural language web interaction. Describe what you want in plain English.
Requires an AI subscription.
| Tool | Description | |---|---| | spider_ai_crawl | AI-guided crawling — describe what content to find. | | spider_ai_scrape | Extract structured data with plain English — no CSS selectors. | | spider_ai_search | AI-enhanced search with intent understanding. | | spider_ai_browser | Automate browser actions with natural language. | | spider_ai_links | Find and filter links by description. |
Browser Automation Tools (9)
Direct browser control via spider-browser. Browsers run in Spider's cloud with smart retry, browser switching, anti-bot protection, proxy rotation, and automatic stealth escalation. Supports CDP and BiDi protocols.
| Tool | Description | |---|---| | spider_browser_open | Open a remote browser session. Returns a session_id. | | spider_browser_navigate | Navigate to a URL and wait for load. | | spider_browser_click | Click an element by CSS selector. | | spider_browser_fill | Fill a form field with text. | | spider_browser_screenshot | Take a screenshot of the current page. Returns base64 PNG. | | spider_browser_content | Get page HTML or visible text. | | spider_browser_evaluate | Execute JavaScript in the page context. | | spider_browser_wait_for | Wait for an element, navigation, or network idle. | | spider_browser_close | Close the session and stop billing. |
Browser sessions auto-close after 5 minutes of inactivity. Always call spider_browser_close when done.
Examples
Research and RAG
"Crawl the React documentation and summarize the hooks API"
Uses spider_crawl to fetch 50+ pages in seconds and return clean markdown ready for context.
spider_crawl: {
url: "https://react.dev/reference/react",
limit: 50,
return_format: "markdown",
filter_output_main_only: true
}Structured Data Extraction
"Get all product names and prices from this e-commerce page"
Uses spider_ai_scrape to extract structured JSON with zero CSS selectors.
spider_ai_scrape: {
url: "https://example-store.com/products",
prompt: "Extract every product name, price, and availability status as JSON"
}Multi-Step Browser Automation
"Log into the dashboard, go to reports, and screenshot the monthly summary"
Uses spider_browser_* tools to drive a remote browser with full anti-bot protection.
1. spider_browser_open: { browser: "auto" }
2. spider_browser_navigate: { url: "https://app.example.com/login" }
3. spider_browser_fill: { selector: "input[name='email']", value: "user@example.com" }
4. spider_browser_fill: { selector: "input[name='password']", value: "..." }
5. spider_browser_click: { selector: "button[type='submit']" }
6. spider_browser_wait_for: { navigation: true }
7. spider_browser_navigate: { url: "https://app.example.com/reports/monthly" }
8. spider_browser_screenshot: {}
9. spider_browser_close: {}Competitive Intelligence
"Search for recent AI startup funding rounds and get the details"
Uses spider_search with time filtering, then spider_scrape for details.
spider_search: {
search: "AI startup Series A funding 2025",
num: 10,
fetch_page_content: true,
return_format: "markdown",
tbs: "qdr:m"
}API Reference
All tools map directly to the Spider API. Core tools accept the same parameters as their API counterparts:
- Crawl/Scrape:
url,return_format,request,readability,root_selector,proxy_enabled,cache, and many more - Search:
search,num,fetch_page_content,country,language,tbs - AI tools: Add a
promptparameter describing what you want in natural language - Browser tools: Use
session_idfromspider_browser_openfor all operations
Full parameter reference: spider.cloud/docs/api
Links
License
MIT
