!Python 3.11+ !License: MIT !Code style: ruff

Surf MCP

MCP server for visual browser automation via Fara.

Overview

Surf provides browser automation through visual grounding - you describe what you see, and it clicks, types, and navigates based on that description. No CSS selectors, no DOM traversal, just natural language.

The core insight: an AI that can see the page doesn't need to parse HTML.

Features

Visual grounding: Click/type by natural language description ("the blue Submit button")
Direct Fara execution: Fara decides the action, we execute it
Autonomous mode: Multi-step goal completion with progress tracking
Multi-server LM Studio: Auto-discovery and failover across GPU servers
Session persistence: Storage state (cookies, localStorage) round-trips through tool calls
Security controls: Domain allowlists, rate limiting, audit logging

Security

surf-mcp is designed for LOCAL use only via stdio transport.

Not Suitable For

Multi-tenant environments - trust boundary is the machine
Untrusted networks without SSH tunneling
Compliance-sensitive contexts - no formal security audit
Untrusted MCP clients - surf-mcp trusts its client completely

Residual Risks

No encryption at MCP protocol level
LLM responses (Fara/Gemini) executed without verification
Browser automation can click/type anything visible

Remote Execution

Use SSH as the transport - surf-mcp sees normal stdio:

{
  "mcpServers": {
    "surf-remote": {
      "command": "ssh",
      "args": ["-i", "~/.ssh/key", "user@gpu-box", "surf-mcp"]
    }
  }
}

See SECURITY.md for the full threat model and security controls.

How It Works

Surf uses Fara-7B (Microsoft's agentic vision model) to understand web pages:

sequenceDiagram
    participant Client as MCP Client
    participant Surf as surf-mcp
    participant PW as Playwright
    participant Fara as Fara-7B

    Client->>Surf: act("click the search button")
    activate Surf
    Surf->>PW: screenshot()
    PW-->>Surf: PNG image
    Surf->>Fara: analyze(image, goal)
    Note right of Fara: Visual grounding
    Fara-->>Surf: FaraToolCall{left_click, [624,280]}
    Surf->>PW: click(624, 280)
    PW-->>Surf: done
    deactivate Surf
    Surf-->>Client: Result + new screenshot

Supported Actions

| Action | Description | |--------|-------------| | left_click | Click at coordinates | | double_click | Double-click at coordinates | | type | Type text (optionally at coordinates) | | scroll | Scroll page up/down | | key | Press keyboard keys | | visit_url | Navigate to URL | | terminate | Task complete signal (agent mode) | | wait | Wait for page to load |

Installation

# Install from source
pip install -e .

# Install Playwright browsers
playwright install chromium

# Optional: Install harness dependencies
pip install -e ".[harness]"

Quick Start

As MCP Server

Add to your MCP client configuration:

{
  "mcpServers": {
    "surf": {
      "command": "surf-mcp"
    }
  }
}

Docker

# Recommended: use docker compose (reads .env automatically)
cp .env.example .env
# Edit .env with your settings
docker compose up

# Or build and run directly (note: --env-file doesn't strip quotes)
docker build -t surf-mcp .
docker run -it --rm \
  --add-host=host.docker.internal:host-gateway \
  -e LMSTUDIO_SERVERS=default=http://host.docker.internal:1234/v1 \
  surf-mcp

Fara Test Harness

Interactive UI for testing visual grounding:

cd tools/fara-harness
./run.sh    # Linux/Mac
run.bat     # Windows

See tools/fara-harness/CHEATSHEET.md for command reference.

Usage Examples

Browser Navigation with Visual Grounding

# Create session
session = await mcp.call("session_create", {
    "drivers": {
        "web": {
            "type": "browser",
            "headless": False,
            "storage_state": saved_state  # Optional: restore cookies
        }
    }
})

# Navigate to page
await mcp.call("goto", {
    "session_id": session["session_id"],
    "driver": "web",
    "location": "https://example.com"
})

# Click element by description
await mcp.call("click", {
    "session_id": session["session_id"],
    "driver": "web",
    "description": "the blue Submit button"
})

# Direct Fara execution (recommended)
await mcp.call("act", {
    "session_id": session["session_id"],
    "driver": "web",
    "goal": "type 'hello world' into the search box"
})

# Autonomous multi-step execution
await mcp.call("act_autonomous", {
    "session_id": session["session_id"],
    "driver": "web",
    "goal": "log in with username 'demo' and password 'demo123'"
})

# Destroy session and capture storage_state
result = await mcp.call("session_destroy", {"session_id": session["session_id"]})
saved_state = result["summary"]["web"]["storage_state"]

Configuration

Environment Variables

# Multi-server LM Studio (visual grounding)
LMSTUDIO_SERVERS="rtx3090=http://localhost:1234/v1,rtx8000=http://192.168.1.100:1234/v1"
FARA_MODEL_IDS="microsoft_fara-7b,fara-7b-gguf,gao-zijian/fara-7b"
FARA_MAX_FAILURES=2
FARA_PROBE_TIMEOUT=2.0

# Confidence and Agent Mode
FARA_MIN_CONFIDENCE=0.7
FARA_CONFIDENCE_RETRIES=2
FARA_MAX_AGENT_STEPS=20

# Alternative: Single OpenAI-compatible endpoint
OPENAI_API_KEY=lm-studio
OPENAI_BASE_URL=http://localhost:1234/v1
SURF_LLM_MODEL=microsoft_fara-7b

# Alternative: Gemini
GOOGLE_API_KEY=...
SURF_LLM_PROVIDER=gemini
SURF_LLM_MODEL=gemini-2.0-flash

# Browser defaults
SURF_BROWSER_HEADLESS=true
SURF_BROWSER_VIEWPORT_WIDTH=1920
SURF_BROWSER_VIEWPORT_HEIGHT=1080

# Session management
SURF_MAX_SESSIONS=10
SURF_SESSION_TIMEOUT_SECONDS=3600

Multi-Server LM Studio

Surf supports multiple LM Studio instances for redundancy:

LMSTUDIO_SERVERS="gpu1=http://localhost:1234/v1,gpu2=http://192.168.1.50:1234/v1"

Behavior:

Auto-discovery: Probes each server's /v1/models to find loaded Fara model
Prefer loaded: Prioritizes servers with Fara already in VRAM
Failover: Automatically retries on another server if one fails

MCP Tools

Session Lifecycle

| Tool | Description | |------|-------------| | session_create | Create browser session | | session_destroy | Cleanup session, returns storage_state | | session_list | List active sessions |

Navigation

| Tool | Description | |------|-------------| | goto | Navigate to URL | | current | Get current URL | | back / forward | Navigate history | | history | Get navigation history |

Content

| Tool | Description | |------|-------------| | list | Extract page links | | read | Read page content | | snapshot | Capture screenshot |

Visual Grounding

| Tool | Description | |------|-------------| | locate | Find element by description, return coordinates | | click | Click element by description | | type | Type into element by description | | scroll | Scroll page up/down | | wait | Wait for element or delay | | act | Direct Fara execution - Fara decides the action | | act_autonomous | Multi-step autonomous execution until task complete |

Architecture

See docs/ARCHITECTURE.md for detailed architecture documentation.

Design decisions are recorded in docs/adr/.

Development

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest                    # All tests
pytest -m "not live"      # Skip LLM tests (for CI)
pytest -m live            # Only live LLM tests

# Type checking
mypy src/

# Linting
ruff check src/

License

MIT

---

surf-mcp