autopilot-mcp

autopilot-mcp

A browser-automation MCP server. It hands an LLM a generic set of browser tools — navigate, screenshot, read text, run JS, click, type — that work on any URL, with each site backed by its own persistent Camoufox browser profile. Logins are filled straight from a Bitwarden vault so passwords never enter the model's context, and any sequence of steps that works can be saved as a "playbook" for one-call replay next time.

Tools

Browser (free-roam)

Each registrable domain (eTLD+1) gets its own persistent browser profile under data/profiles/<profile>/. navigate(url) auto-derives the profile from the URL; everything else takes profile explicitly.

| Tool | Description | |------|-------------| | navigate | Open a URL. Auto-derives profile from eTLD+1 (overridable). Returns visible text. | | screenshot | PNG screenshot of the profile's current page. | | get_text | Visible text only — cheaper than a screenshot. | | get_url | Current URL for the profile. | | run_js | Preferred for form fills / button clicks. Selector-based. | | click | Click at (x, y). Use when run_js can't target the element. | | type_text | Type into the focused element. | | attach_file | Attach a local file to a <input type="file"> (incl. hidden inputs). | | scroll | Scroll up or down. |

For parallel work on the same site, use isolated browser instances — each clones the site's base profile so concurrent sessions don't collide:

| Tool | Description | |------|-------------| | spawn_instance | Clone a base profile into a temporary isolated browser profile and open a URL. Returns instance_id. | | list_instances | List live spawned instances and TTLs. | | close_instance | Close an instance and delete its temporary profile. | | instance_navigate / instance_screenshot / instance_get_text / instance_get_url | Browser navigation/inspection scoped to one instance_id. | | instance_run_js / instance_click / instance_type_text / instance_scroll | Page interaction scoped to one instance_id. | | instance_attach_file / instance_fill_login | Upload/login helpers scoped to one instance_id. |

Example: spawn_instance(url="https://accounts.google.com/...", clone_from_profile="google.com") lets each Gmail cleanup branch use its own cloned Google session. Always call close_instance(instance_id) when the branch is finished; timed-out instances are also cleaned up automatically.

Credentials (Bitwarden, fill-don't-reveal)

| Tool | Description | |------|-------------| | list_logins | Search the vault. Returns id/name/urls/username — never passwords. | | fill_login | Inject creds from Bitwarden straight into form fields. Password never returns. | | get_totp | Current 6-digit TOTP from Bitwarden (single source of truth). | | create_login | New vault entry. Refuses name collision. | | update_login | Patch fields on an existing entry. | | upsert_login | Create-or-update by (url, username). The signup convenience path. | | delete_login | Send to Bitwarden trash. Requires confirm=True. | | reveal_credentials | ESCAPE HATCH — returns plaintext. Requires reason, audited. |

Local file server (uploads)

For sites that ask the user to upload a local file. Two paths:

1. Standard <input type="file"> — use `attach_file(profile, selector,

path)`. Works even when the input is hidden inside a custom dropzone widget; target the input itself, not the visible drop area.

1. Pure-JS uploader (no real input element) — use the local CORS file

server below. The MCP publishes the file at an unguessable URL on 127.0.0.1; the LLM uses run_js to fetch() it inside the page, wrap the Blob in a File, and dispatch a synthetic drop event (or set it on a hidden input via DataTransfer).

| Tool | Description | |------|-------------| | serve_local_file | Publish a local file at http://127.0.0.1:<port>/file/<token> with CORS. Returns url, token, content_type, size, expires_at. TTL default 30 min. | | list_served_files | List currently-published files. | | unserve_local_file | Revoke a token immediately. |

Security envelope: server binds 127.0.0.1 only; tokens are uuid4 hex (122 bits of entropy); one token = one file path (no directory traversal); idle entries reaped on every request. Override the bind via AUTOPILOT_FILE_SERVER_HOST / AUTOPILOT_FILE_SERVER_PORT env vars.

Playbooks

| Tool | Description | |------|-------------| | list_playbooks | List saved playbooks (filter by start_url substring). | | run_playbook | Execute a playbook. Returns screenshots/text from observation steps. | | save_playbook | Save a step sequence. Call after a successful task. | | delete_playbook | Remove a broken playbook. | | playbook_run_list | List run-ledger entries (one record per execution), newest first; filter by name/success. | | playbook_run_get | Fetch one run ledger's full JSON by run_id. |

Workflow

1. list_playbooks(url_match) — is there already a playbook for this task?
2. run_playbook(name) — if yes, run it. Done.
3. Otherwise: navigate(url) → screenshot / get_text → run_js / click / type_text.
4. On a login page: fill_login(url) — Bitwarden injects creds directly. If the form needs 2FA: get_totp(vault_item) then type_text(profile, code).
5. For SMS 2FA: navigate("https://messages.google.com/web/") and read the code from Google Messages.
6. After the task succeeds, save_playbook(...) so next time is one call.
7. Just signed up somewhere new? upsert_login(url, username, password) stores it and Bitwarden sync pushes to your other devices.

Credentials setup (Bitwarden)

The MCP unlocks Bitwarden with a master password stashed in the OS keyring (DPAPI-encrypted on Windows, scoped to your user). On each start it pulls the master password from the keyring, runs bw unlock --raw, and caches the session token in RAM only — idle-expires after 15 minutes, re-locks on shutdown. The master password never lands on disk outside the OS keyring, and never enters the model's context.

One-time setup for a fresh machine, top to bottom:

1. Install the Bitwarden CLI

winget install --id Bitwarden.CLI --accept-source-agreements --accept-package-agreements

winget puts bw.exe on PATH via a shim — open a new shell afterward so the update takes effect. (No winget? npm install -g @bitwarden/cli, or grab a binary from <https://bitwarden.com/download/>.) Verify:

bw --version   # e.g. 2026.3.0
bw status      # {"status":"unauthenticated", ...}

2. Log in

Interactive — only your terminal sees the master password.

bw login

Prompts for email, master password, and a two-step token. On success bw status reports "status":"locked" — leave it locked; the MCP unlocks on demand.

3. Stash the master password in the OS keyring

Keep it out of .env and off the command line. After uv sync, stash it at the hidden prompt:

uv run python -c "import keyring, getpass; keyring.set_password('autopilot-mcp', 'bw_master', getpass.getpass('Master password: ')); print('stored')"

This writes to service autopilot-mcp, username bw_master. Confirm without printing the value:

uv run python -c "import keyring; v = keyring.get_password('autopilot-mcp', 'bw_master'); print(f'present={v is not None} length={len(v) if v else 0} backend={keyring.get_keyring().__class__.__name__}')"
# present=True length=<your pw length> backend=WinVaultKeyring

4. Smoke-test the unlock loop

Runs the real path — keyring read, bw unlock, list, lock — without printing the password:

uv run python -c "
import json, os, subprocess, keyring
pw = keyring.get_password('autopilot-mcp', 'bw_master')
assert pw, 'keyring empty'
subprocess.run(['bw', 'sync'], check=True)
u = subprocess.run(['bw', 'unlock', '--raw', '--passwordenv', 'BW_PW'],
                   env={**os.environ, 'BW_PW': pw}, capture_output=True, text=True, check=True)
session = u.stdout.strip()
items = json.loads(subprocess.run(['bw', 'list', 'items', '--search', 'example',
                                   '--session', session],
                                  capture_output=True, text=True, check=True).stdout)
print(f'vault items matching \"example\": {len(items)}')
subprocess.run(['bw', 'lock', '--session', session], check=True)
"

If it completes without errors, setup is done.

Maintenance

• Rotate the master password — re-stash; the entry is overwritten in place:

  uv run python -c "import keyring, getpass; keyring.set_password('autopilot-mcp', 'bw_master', getpass.getpass('New master password: '))"

• Remove the keyring entry (the MCP then fails at startup until restored):

  uv run python -c "import keyring; keyring.delete_password('autopilot-mcp', 'bw_master')"

• bw fell off PATH — open a new shell (winget's PATH update doesn't reach

already-open shells); if still missing, re-run the install from step 1.

• Force a full re-sync — bw sync --force. The MCP runs bw sync after

every write, so this is only needed if the vault was edited elsewhere and you want the in-RAM cache to refresh before idle expiry.

• Log out — bw logout drops the account from local bw state; repeat

steps 2–3 to restore.

Initial browser session setup

Each profile gets one persistent browser profile the first time it's opened. For sites where you want the session pre-established (to handle 2FA challenges / "remember me" outside the MCP flow):

uv run python scripts/manual_login.py <url>

A visible Camoufox window opens at the URL. Log in, complete 2FA, check "remember me", close the window. The profile at data/profiles/<eTLD+1>/ persists across headless MCP invocations.

Environment variables

All optional — defaults are sane for local use.

| Variable | Default | Description | |----------|---------|-------------| | HEADLESS | true | Set "false" to show the browser window for debugging. | | BROWSER_TIMEOUT | 30000 | Per-page navigation/action timeout, in ms. | | AUTOPILOT_TOOL_TIMEOUT_SECONDS | 60 | Wall-clock cap on a single tool call. | | AUTOPILOT_PLAYBOOK_TIMEOUT_SECONDS | 300 | Wall-clock cap on a run_playbook call. | | AUTOPILOT_BW_TIMEOUT_SECONDS | 45 | Timeout for a single bw CLI invocation. | | AUTOPILOT_FILE_SERVER_HOST | 127.0.0.1 | Bind interface for the local file server. | | AUTOPILOT_FILE_SERVER_PORT | 0 | Bind port for the local file server (0 = ephemeral). | | AUTOPILOT_LOG_JSON | false | "true" for JSON logs; otherwise human-readable console output. | | AUTOPILOT_LOG_LEVEL | INFO | Root log level for all autopilot.* loggers. | | BITWARDENCLI_APPDATA_DIR | — | Override the Bitwarden CLI data directory (standard bw variable). |

Credentials are pulled from Bitwarden — there are no per-site username/password environment variables.

Development

uv sync --extra dev
uv run camoufox fetch
uv run ruff check .
uv run pytest
uv run python server.py    # stdio mode