myscrape

myscrape (Go)


A self-contained web-research MCP server for local agents — web_search, web_fetch, web_research — shipped as a compiled binary (no interpreter needed).

This is the Go port of the Python reference implementation. The port follows the spec, behavior, and test suite of the original; see specs/PORTING_GO.md in the Python repo for the module map and the empirically-tuned operating points that carry over verbatim. The Python version is kept only as the differential-test oracle/fallback until this reaches parity, then retired.

Status — Phase 1 complete (functional parity, GPU-free core)

All three tools work over stdio and streamable-HTTP (official modelcontextprotocol/go-sdk):

• web_search — round-robin over many engines (DDG + Marginalia always on;

Tavily/Exa/SerpApi/Serper/Mojeek/Google CSE join when their key is set), with a per-engine circuit breaker.

• web_fetch — static fetch + main-content extraction (go-trafilatura).
• web_research — search → fetch → chunk → BM25 rank → synthesize. Default

synthesis is extractive (no LLM/GPU); set synthesis=llm (and an LLM endpoint) to opt into model-written answers.

The HTTP stack carries the Python reference's stability behavior verbatim: Tier-0 cache, Tier-1 per-host pacing (8s/2s), Tier-3 retry/backoff with Retry-After.

go build -o myscrape ./cmd/myscrape
./myscrape                                                     # stdio
MYSCRAPE_MCP_TRANSPORT=http MYSCRAPE_MCP_PORT=8000 ./myscrape  # -> http://127.0.0.1:8000/mcp

# Docker (static image, ~30 MB, GPU-free)
docker compose up -d --build

Provider keys and the optional LLM endpoint are configured via MYSCRAPE_* env vars — see .env.example.

Docs

• specs/SPEC.md — tool contracts + error taxonomy
• specs/PROVIDERS.md — search engines + how to add one
• specs/EXPERIMENTS.md — the empirical operating points
• specs/DEVLOG.md — Go port decisions (G-001…)
• docs/INTEGRATION.md — consuming myscrape from another product
• CONTRIBUTING.md — workflow, hooks, conventions

Layout

cmd/myscrape        entrypoint (stdio | http)
internal/config     MYSCRAPE_* settings
internal/httpx      HTTP primitive: cache -> rate-limit -> retry (the Doer seam)
internal/cache      Tier-0 TTL cache
internal/ratelimit  Tier-1 per-host pacer
internal/retry      Tier-3 backoff (Retryable, Retry-After)
internal/search     Provider interface, engine adapters, round-robin/breaker
internal/fetch      static fetch + go-trafilatura extraction
internal/research   chunk, BM25 rank, extractive + LLM synthesis, pipeline
internal/llm        OpenAI-compatible chat client (opt-in synthesis)
internal/mcpserver  tool registration + handlers

Beyond the core

• Dynamic (JS) fetch — internal/fetch escalates thin/JS pages to a headless

Chrome render via chromedp (MYSCRAPE_DYNAMIC_ENABLED, needs Chrome on PATH).

• cmd/bench — live stability gate (go run ./cmd/bench); passes when

ratelimit_rate == 0 at the 8s/2s operating point.

• CI/release — GitHub Actions runs the offline gate + Docker build; tagging

v* builds cross-OS binaries and publishes a multi-arch image to GHCR.