pagemap

<!-- mcp-name: io.github.Retio-ai/pagemap -->

PageMap

PageMap converts raw HTML (100K+ tokens) into structured, AI-readable page maps (2-5K tokens) — a 97% token reduction. It works as an MCP server, Python SDK, and CLI, supporting 16 page types and 30+ e-commerce sites. Agents can read, click, type, and navigate any web page.

"Give your agent eyes and hands on the web."

     

---

<!-- ============================================================ --> <!-- HUMAN GUIDE --> <!-- ============================================================ -->

Why PageMap?

Playwright MCP dumps 50-540KB accessibility snapshots per page, overflowing context windows after 2-3 navigations. Firecrawl and Jina convert HTML to markdown — read-only, no interaction.

PageMap gives your agent a compressed, actionable view of any web page:

| | PageMap | Playwright MCP | Firecrawl | Jina Reader | |--|:------:|:---------:|:-----------:|:--------:| | Tokens / page | 2-5K | 6-50K | 10-50K | 10-50K | | Interaction | click / type / select / hover | Raw tree parsing | Read-only | Read-only | | Multi-page sessions | Unlimited | Breaks at 2-3 pages | N/A | N/A | | Task success (94 tasks) | 84.7% | 61.5% | 64.5% | 57.8% | | Avg tokens / task | 2,710 | 13,737 | 13,888 | 11,424 | | Cost / 94 tasks | $1.06 | $4.09 | $3.98 | $2.26 |

Benchmarked across 11 e-commerce sites, 94 static tasks, 7 conditions. 8,100+ tests passing.

---

Quick Start

Chromium is auto-installed on first use — no manual playwright install needed.

Install

pip install retio-pagemap

MCP Client Config

Add to Claude Code, Cursor, Windsurf, or Claude Desktop:

{
  "mcpServers": {
    "pagemap": {
      "command": "uvx",
      "args": ["retio-pagemap"]
    }
  }
}

Claude Desktop (macOS): Use the absolute path to uvx — run which uvx (e.g. /opt/homebrew/bin/uvx).

VS Code (Copilot): Use "servers" instead of "mcpServers" in .vscode/mcp.json.

Docker

docker run -p 8000:8000 retio1001/pagemap --transport http

---

Features

13 MCP Tools — Read + Interact

Not just reading — your agent can click buttons, fill forms, select options, manage tabs, and navigate across pages. 13 tools cover the full browsing workflow:

get_page_map · execute_action · fill_form · scroll_page · wait_for · take_screenshot · get_page_state · navigate_back · batch_get_page_map · open_tab · switch_tab · list_tabs · close_tab

16 Page Types, Auto-Detected

PageMap automatically classifies pages and applies optimized extraction for each type:

product_detail · listing · search_results · article · news · video · login · form · checkout · dashboard · help_faq · settings · error · documentation · landing · blocked

E-Commerce Deep Coverage

Built-in support for 30+ major e-commerce sites across 4 tiers:

• Global mega-platforms — Amazon, eBay, AliExpress, SHEIN, Walmart, Rakuten
• Global fashion — Zara, H&M, Nike, Uniqlo, ASOS, Zalando, SSENSE, Farfetch, COS
• Korea — Coupang, Naver Shopping, Musinsa, 29CM, W Concept, SSG, 11st
• Japan/China — ZOZO, Tmall, JD.com, Taobao

Structured extraction of prices, options (size/color), ratings, availability — with automatic cookie consent handling and login barrier detection.

Smart Recovery

PageMap detects problems and tells your agent what to do:

• Barrier detection — Login required? Bot blocked? Out of stock? Age verification? Popup overlay? PageMap adds a barrier field with the diagnosis and suggested next steps
• Cookie consent auto-dismiss — 7 CMP providers auto-detected (Cookiebot, OneTrust, TrustArc, Didomi, Quantcast, Usercentrics, generic fallback). 5-tier dismiss cascade: CMP JS API → Reject → Accept → Dismiss → Close symbol. GDPR reject-first default policy
• Popup overlay detection — AX tree role="dialog" + HTML regex 2-phase detection. Promotional popups (newsletter, exit-intent) auto-dismissed
• Bot detection awareness — Detects Cloudflare, Turnstile, reCAPTCHA, hCaptcha, and Akamai. Reports the provider and suggests wait/retry strategies
• Stale ref recovery — When DOM changes invalidate refs, PageMap returns clear guidance to re-fetch

Content Intelligence

• 8 JSON-LD schemas — Product, NewsArticle, VideoObject, FAQPage, Event, LocalBusiness, BreadcrumbList, and ItemList
• Metadata extraction — Prices, ratings, reviews, descriptions, images from structured data and DOM fallbacks
• 2-layer caching — Cache hit (~10ms), content refresh (~500ms), full rebuild (~1.5s). Diff-based updates for unchanged sections
• Delta evidence packet output - Optional to_delta_packet() serializer emits digest-bound evidence units, claim candidates, provenance, and authority flags for downstream memory/review systems without changing the default MCP output

10 Languages

Locale auto-detected from URL. Token budgets adjusted for CJK scripts.

| Language | Locale | Language | Locale | |----------|:------:|----------|:------:| | English | en | Chinese | zh | | Korean | ko | Spanish | es | | Japanese | ja | Italian | it | | French | fr | Portuguese | pt | | German | de | Dutch | nl |

---

Deployment

Local (STDIO)

Default mode. Runs as a local MCP server — no server setup needed.

retio-pagemap

Docker

docker run -p 8000:8000 retio1001/pagemap --transport http

Multi-architecture images (amd64/arm64) available on Docker Hub and GitHub Container Registry.

---

Python API

import asyncio
from pagemap.browser_session import BrowserSession
from pagemap.delta_serializer import to_delta_packet
from pagemap.page_map_builder import build_page_map_live
from pagemap.serializer import to_agent_prompt, to_json

async def main():
    async with BrowserSession() as session:
        page_map = await build_page_map_live(session, "https://example.com/product/123")
        print(to_agent_prompt(page_map))   # Agent-optimized text format
        print(to_json(page_map))           # Structured JSON
        print(to_delta_packet(page_map))   # Digest-bound evidence packet
        print(page_map.page_type)          # "product_detail"
        print(page_map.interactables)      # [Interactable(ref=1, role="button", ...)]
        print(page_map.metadata)           # {"name": "...", "price": "..."}

asyncio.run(main())

For offline processing (no browser):

from pagemap.page_map_builder import build_page_map_offline

page_map = build_page_map_offline(open("page.html").read(), url="https://example.com/product/123")

---

Security

PageMap treats all web content as untrusted input:

• SSRF defense — Multi-layer protection against server-side request forgery
• Prompt injection defense — Content boundaries, role-prefix stripping, suspicious content flagging
• robots.txt compliance — RFC 9309 compliant. --ignore-robots opt-out flag
• Resource guards — DOM node limit, HTML size limit, response size limit
• Session isolation — Each session has independent cookies and storage, automatically cleaned up

Local development: Private IPs are blocked by default. Use --allow-local or PAGEMAP_ALLOW_LOCAL=1.

Disclaimer

Users are responsible for complying with the terms of service of target websites and all applicable laws when using PageMap.

---

Troubleshooting

"spawn uvx ENOENT" (Claude Desktop on macOS) — Claude Desktop does not inherit your shell PATH. Run which uvx and use the absolute path in your config.

First page takes a long time — Chromium cold start takes ~10-30s on first navigation. Subsequent pages load in 1-3 seconds.

Localhost blocked — Use --allow-local flag or set PAGEMAP_ALLOW_LOCAL=1.

Chromium not found — Run pip install retio-pagemap && playwright install chromium to install manually.

---

Requirements

• Python 3.11+
• Chromium (auto-installed on first use)

Community

Have a question or idea? Join the conversation in GitHub Discussions.

Development


git clone https://github.com/Retio-ai/Retio-pagemap.git
cd Retio-pagemap
uv sync --group dev
playwright install chromium
uv run pytest --tb=short -q

Pricing

Local (STDIO) — Free forever. Self-hosted, open source under AGPL-3.0.

Cloud API — Hosted multi-tenant server with auth, rate limiting, and credit-based billing. Contact retio1001@retio.ai for access.

License

AGPL-3.0-only — see LICENSE for the full text.

For commercial licensing options, contact retio1001@retio.ai.

---

<!-- ============================================================ --> <!-- AGENT REFERENCE --> <!-- ============================================================ -->

For Agents

This section is written for AI agents using PageMap as an MCP tool.

Tools

| Tool | When to use | |------|-------------| | get_page_map | Start here. Navigate to a URL and get a full structured map with numbered refs. | | execute_action | Click, type, select, or hover using a ref number from the last get_page_map. | | fill_form | Fill multiple form fields in one call. More efficient than sequential execute_action calls. | | get_page_state | Check current URL and title without a full rebuild. Use after actions that may navigate. | | scroll_page | Scroll to reveal lazy-loaded content before calling get_page_map again. | | wait_for | Wait for dynamic content to appear (e.g. after a search or form submit). | | take_screenshot | Capture the visual state when the PageMap alone is ambiguous. | | navigate_back | Go back one step in browser history. | | open_tab | Open a new browser tab and navigate to a URL. | | switch_tab | Switch to a different open tab by index. | | list_tabs | List all open tabs with their URLs and titles. | | close_tab | Close a tab by index. | | batch_get_page_map | Fetch multiple URLs in parallel. Use for comparison tasks. |

Output Format

URL: https://example.com/product/123
Title: Product Name
Type: product_detail          # auto-detected page type

## Actions
[1] button: Add to cart (click)
[2] select: Size (select) — options: S, M, L, XL
[3] link: See all reviews (click)
...

## Info
Price: $49.99
Rating: 4.5 / 5 (128 reviews)
Description: ...

## Images
  [1] https://cdn.example.com/product.jpg

## Meta
Tokens: ~1,800 | Interactables: 24 | Generation: 380ms

• ## Actions — Every interactive element on the page with a stable ref number.
• ## Info — Key page content extracted from HTML: prices, titles, ratings, descriptions.
• ## Images — Product/content image URLs.
• ## Meta — Token count, interactable count, generation time.

Barrier Detection

When PageMap encounters a page-level obstacle, it includes a barrier field in the response:

State:
  barrier: login_required
  barrier_hint: "Login form detected with email + password fields. Use fill_form to authenticate."

Possible barriers: cookie_consent, login_required, bot_blocked, out_of_stock, empty_results, error_page, age_verification, region_restricted, popup_overlay.

When you see a barrier: follow the barrier_hint guidance. For bot_blocked, wait and retry. For login_required, use fill_form with credentials.

Ref Lifecycle

Refs are assigned by get_page_map and remain valid until the page state changes.

Refs are invalidated when:

• The page navigates to a new URL
• A DOM mutation occurs (modal opens, SPA navigation, accordion toggles)
• execute_action causes a page-level change

When you get a stale ref error: call get_page_map again to get fresh refs before retrying.

Token Budget Behavior

When a page exceeds the token budget, content is pruned in this order:

1. Navigation menus, footers, sidebars removed first
2. Secondary body content trimmed
3. ## Actions and ## Info are always preserved

If key content seems missing, try scroll_page to load lazy content, then get_page_map again.

Recommended Workflow

1. get_page_map(url)          → read Actions + Info, pick refs
2. execute_action(ref, ...)   → interact
3. get_page_state()           → confirm navigation occurred
4. get_page_map(new_url)      → get fresh refs for next step

For pages with dynamic content (search results, filters): ```

1. get_page_map(url)
2. execute_action(ref, "click") → trigger search/filter
3. wait_for(text="results") → wait for content
4. get_page_map(url) → get updated map

### Known Limitations

- **Login-gated pages** — PageMap does not manage sessions or cookies. Authentication must be handled externally.
- **Heavy bot detection** (Cloudflare, Akamai) — May block automated access. PageMap detects the provider and suggests strategies, but cannot bypass active bot mitigation.
- **Private network access** — Blocked by default. Requires `--allow-local` flag.
- **iframes** — Cross-origin iframes are not accessible due to browser security policies.

---

*PageMap — Structured Web Intelligence for the Agent Era.*