Portable MCP Web Scraper

🚀 Portable MCP Web Scraper

A drop-in MCP (Model Context Protocol) server for efficient web scraping with two-step workflow optimization.

✨ Features

• Two-Step Workflow: Get HTML structure preview first, then scrape with targeted filters
• Token Efficient: Minimal token usage for AI analysis and decision-making
• Clean Output: Automatically removes navigation, ads, and UI elements
• Portable: Drop anywhere and add to your MCP servers
• Multiple Tools: Single page, multi-page, and documentation site scraping

🚀 Quick Start

Option 1: Automated Installation

python install.py

Option 2: Manual Setup

1. Install dependencies:

   pip install -r requirements.txt

1. Copy to your MCP servers directory:

   cp portable_mcp_scraper.py /path/to/your/mcp/servers/

1. Add to your MCP configuration (e.g., ~/.cursor/mcp.json):

   {
     "mcpServers": {
       "web-scraper": {
         "command": "python",
         "args": ["/path/to/your/mcp/servers/portable_mcp_scraper.py"]
       }
     }
   }

1. Restart your MCP client

🛠️ Available Tools

1. preview_html_structure

Get a clean HTML structure preview for AI analysis.

Parameters:

• url (string): The URL to analyze
• max_elements (int, optional): Maximum elements to include (default: 50)

Returns: Structured HTML preview with minimal text content

2. scrape_web_content

Scrape web content with custom filtering.

Parameters:

• url (string): The URL to scrape
• include_tags (list, optional): HTML tags to include
• exclude_tags (list, optional): HTML tags to exclude
• save_to_file (bool, optional): Save content to file (default: false)
• output_dir (string, optional): Directory to save files (default: "./scraped_content")

Returns: Clean Markdown content

3. scrape_documentation_site

Scrape an entire documentation site with intelligent crawling.

Parameters:

• base_url (string): Base URL of the documentation site
• max_pages (int, optional): Maximum pages to scrape (default: 10)
• include_tags (list, optional): HTML tags to include
• exclude_tags (list, optional): HTML tags to exclude
• save_to_files (bool, optional): Save each page to separate file (default: true)
• output_dir (string, optional): Directory to save files (default: "./documentation")

Returns: Summary of scraped content with file paths

💡 Two-Step Workflow Benefits

1. Step 1: Get HTML structure preview (~500-1000 tokens)
2. Step 2: Scrape with AI-determined filters (clean, focused content)

Efficiency Gains:

• For 100-page documentation: ~90% token reduction
• AI only analyzes structure, not full content
• Clean, focused output without manual filtering
• Massive cost savings for large documentation sites

🧪 Testing

Run the test script to verify everything works: ``bash python test_mcp_server.py ``

📋 Example Usage

Basic Single Page Scraping

# Get structure preview first
preview = preview_html_structure("https://cursor.com/docs")

# Then scrape with filters
content = scrape_web_content(
    url="https://cursor.com/docs",
    include_tags=["h1", "h2", "h3", "p", "div"],
    exclude_tags=["nav", "footer", "aside"],
    save_to_file=True
)

Documentation Site Scraping

# Scrape entire documentation site
summary = scrape_documentation_site(
    base_url="https://cursor.com/docs",
    max_pages=20,
    save_to_files=True,
    output_dir="./cursor_docs"
)

🔧 Requirements

• Python 3.8+
• Google Chrome browser
• ChromeDriver (automatically managed by webdriver-manager)

📁 Files

• portable_mcp_scraper.py - The main MCP server
• fastmcp.json - FastMCP configuration
• install.py - Automated installation script
• test_mcp_server.py - Test script
• requirements.txt - Python dependencies
• USAGE_GUIDE.md - 📚 Complete usage guide with examples
• PORTABLE_PACKAGE.md - Detailed package information
• SUCCESS_SUMMARY.md - What this package provides

🎯 Perfect For

• AI Agents that need to scrape documentation on-demand
• Documentation Analysis with minimal token usage
• Content Extraction from complex websites
• Multi-page Scraping with intelligent crawling
• Cost-Effective web scraping for AI workflows

📄 License

MIT License - Feel free to use and modify as needed.

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

📞 Support

If you encounter any issues:

1. Check that Google Chrome is installed
2. Verify all dependencies are installed
3. Run the test script to diagnose problems
4. Check the documentation files for troubleshooting

---

Ready to drop and use! 🎉