@aegis-ai/mcp-scraper
MCP server for intelligent web scraping. Supports static pages, JavaScript-rendered SPAs (via Playwright), and natural language data extraction.
Install
# Add to Claude Desktop config (~/.claude/claude_desktop_config.json):
{
"mcpServers": {
"scraper": {
"command": "bunx",
"args": ["@aegis-ai/mcp-scraper"]
}
}
}
# Or run directly:
bunx @aegis-ai/mcp-scraperTools
scrape
Fetch a web page and return clean text content.
{
"url": "https://news.ycombinator.com",
"js": false,
"selector": ".titleline",
"headers": { "Cookie": "session=abc" }
}extract
Scrape a page and extract structured data using natural language.
{
"url": "https://example.com/products",
"prompt": "Get all product names, prices, and ratings",
"js": true
}batch_scrape
Scrape multiple URLs with rate limiting (1s between requests).
{
"urls": ["https://example.com/page1", "https://example.com/page2"],
"js": false
}Features
- Static page fetching (fast, no browser needed)
- JavaScript rendering via Playwright (for SPAs)
- CSS selector targeting
- Custom headers support
- Automatic HTML stripping
- Content truncation to avoid context overflow
- Rate-limited batch scraping
License
MIT - AEGIS AI Cooperative
