scpr

scpr

scpr is a simple and straightforward webscraping CLI tool made to scrape page as markdown content, and developed to be used both by humans and by coding agents (either as an MCP server or as a skill).

scpr is written in Go and based on colly for web scraping and html-to-markdown for converting HTML pages to markdown.

Installation

Install with Go (v1.24+ required):

go install github.com/AstraBert/scpr

Install with NPM:

npm install @cle-does-things/scpr

Extra instructions for Windows installation

If you are on Windows, scpr might not be available right after global installation with npm. In that case, you might need to take extra steps:

1. Find where the node executable is stored on your machine:

Get-Command node

This will print the directory where node.exe is stored: scpr will be installed at .\bin\scpr.exe in that folder.

[!NOTE] _If you are using nvm for Windows, node.exe will be at C:\Users\nvm4w\nodejs_

1. Add {NODE_FOLDER}\bin (in the case of nvm: C:\Users\nvm4w\nodejs\bin) to the PATH environment variables. Follow this guide for instructions on how to set PATH env variables.
2. Restart your computer
3. Test scpr --help from your terminal. The execution might be challenged by your antivirus, but, since the executable does not contain any harmful code, the antivirus will eventually allow it

Usage

As a CLI tool

Basic usage (scrape a single page):

scpr --url https://example.com --output ./scraped

This will scrape the page and save it as a markdown file in the ./scraped folder.

Recursive scraping

To scrape a page and all linked pages within the same domain:

scpr --url https://example.com --output ./scraped --recursive --allowed example.com --max 3

Parallel scraping

Speed up recursive scraping with multiple threads:

scpr --url https://example.com --output ./scraped --recursive --allowed example.com --max 2 --parallel 5

Additional options

• --log - Set logging level (info, debug, warn, error)
• --max - Maximum depth of pages to follow (default: 1)
• --parallel - Number of concurrent threads (default: 1)
• --allowed - Allowed domains for recursive scraping (can be specified multiple times)

For more details, run:

scpr --help

As a stdio MCP server

Start the MCP server with:

scpr mcp

And configure it in agents using:

{
  "mcpServers": {
    "web-scraping": {
      "type": "stdio",
      "command": "scpr",
      "args": [
        "mcp"
      ],
      "env": {}
    }
  }
}

_The above JSON snippet is reported as used by Claude Code, adapt it to your agent before using it_

Contributing

Contributions are welcome! Please read the Contributing Guide to get started.

License

This project is licensed under the MIT License