ML Experiment Tracker MCP

Query your MLflow experiments in plain English using Claude Desktop. No dashboards, no SQL — just ask.

---

What It Does

Connect Claude Desktop to your MLflow experiment tracker via an MCP server. Ask natural language questions and Claude queries your experiments automatically.

Example questions you can ask Claude:

"List all my ML experiments"
"Which run had the best val_f1 in pothole-detector?"
"What hyperparameters did the best model use?"
"Compare the top 5 runs in road-crack-detection by accuracy"
"Show me the metric history for run ID xyz"

---

Tools Exposed

| Tool | Description | |---|---| | list_experiments | Lists all experiments in MLflow | | compare_runs | Ranks top N runs by a given metric | | get_best_model | Returns the single best run for a metric | | fetch_metric_history | Returns epoch-by-epoch metric history for a run |

---

Prerequisites

Before starting, make sure you have:

Python 3.11+ installed
uv package manager (pip install uv)
Claude Desktop downloaded from https://claude.ai/download
Git installed

---

Installation

1. Clone the repository

git clone https://github.com/yourusername/experiments-mcp.git
cd experiments-mcp

2. Create virtual environment with Python 3.11

uv venv --python /opt/homebrew/opt/python@3.11/bin/python3.11
source .venv/bin/activate

Windows: use .venv\Scripts\activate instead

3. Install dependencies

uv pip install -r requirements.txt

requirements.txt contains: `` fastmcp mlflow pandas ``

4. Verify installation

python --version    # Should show Python 3.11.x
mlflow --version    # Should show mlflow, version 3.x.x

---

Seed Dummy Experiment Data To Test MCP (Optional)

If you don't have real MLflow experiments, run the seed script to populate fake but realistic data:

python seed_mlflow.py

This creates 2 experiments with 35 total runs:

pothole-detector — 20 runs
road-crack-detection — 15 runs

Each run has params (learning_rate, batch_size, backbone, optimizer) and metrics (val_f1, val_accuracy, val_loss, train_f1).

---

Connect to Claude Desktop

1. Find your Python path

which python
# Example output: /Users/yourname/experiments-mcp/.venv/bin/python

2. Find your project path

pwd
# Example output: /Users/yourname/experiments-mcp

3. Edit Claude Desktop config

Open the config file:

# Mac
open ~/Library/Application\ Support/Claude/

# Windows
# %APPDATA%\Claude\claude_desktop_config.json

Add the mcpServers block to claude_desktop_config.json:

{
  "mcpServers": {
    "ml-experiment-tracker": {
      "command": "/Users/yourname/experiments-mcp/.venv/bin/python",
      "args": [
        "/Users/yourname/experiments-mcp/server.py"
      ]
    }
  }
}

Replace /Users/yourname/experiments-mcp with your actual paths from steps 1 and 2.

4. Restart Claude Desktop

Cmd + Q  →  reopen Claude Desktop

5. Verify connection

Click the + button in the chat input → Add plugins → your server should appear as connected.

Or simply type in chat: `` List all my ML experiments ``

---

Project Structure

experiments-mcp/
├── server.py           # MCP server — all 4 tools
├── mlflow_client.py    # MLflow SDK wrapper
├── seed_mlflow.py      # Fake data generator
├── requirements.txt    # Dependencies
└── README.md

---

Using Your Own MLflow Data

By default the server points to a SQLite DB at ~/mcp/experiments-mcp/mlflow.db.

To point it to your own MLflow instance, edit the top of mlflow_client.py:

# Local SQLite (default)
mlflow.set_tracking_uri("sqlite:///path/to/your/mlflow.db")

# Remote MLflow server
mlflow.set_tracking_uri("http://your-mlflow-server:5000")

---

Common Errors & Fixes

`PermissionError: Operation not permitted: .venv/pyvenv.cfg`

Cause: Project is inside ~/Desktop — Claude Desktop cannot access Desktop on Mac due to macOS security.

Fix: Move the project out of Desktop: ``bash mv ~/Desktop/experiments-mcp ~/experiments-mcp cd ~/experiments-mcp ` Update the paths in claude_desktop_config.json` accordingly.

---

`ModuleNotFoundError: No module named 'mlflow'`

Cause: Wrong Python being used — system Python instead of venv Python.

Fix: ```bash

Check which python is active

which python

If it shows /usr/bin/python or /opt/homebrew/bin/python — wrong one

Re-activate your venv:

source /full/path/to/experiments-mcp/.venv/bin/activate

Then reinstall

uv pip install -r requirements.txt ```

---

`MlflowException: filesystem tracking backend is in maintenance mode`

Cause: MLflow 3.x dropped file-based storage (mlruns/ folder). Requires SQLite.

Fix: Make sure mlflow_client.py uses SQLite URI: ``python import os mlflow.set_tracking_uri( f"sqlite:///{os.path.expanduser('~/experiments-mcp/mlflow.db')}" ) ` Also add the same line to seed_mlflow.py` before running it.

---

`sqlite3.OperationalError: unable to open database file`

Cause: The .db file doesn't exist yet — seed script hasn't been run, or was run from a different directory.

Fix: ``bash cd ~/experiments-mcp source .venv/bin/activate python seed_mlflow.py ``

---

`NameError: name 'os' is not defined`

Cause: Missing import os at the top of mlflow_client.py.

Fix: Add to the very first line of mlflow_client.py: ``python import os ``

---

`Server disconnected` in Claude Desktop

Cause: Could be any Python error in server.py or mlflow_client.py.

Fix: Check the logs: ``bash tail -f ~/Library/Logs/Claude/mcp-server-ml-experiment-tracker.log `` The last error in the log will tell you exactly what went wrong.

---

`zsh: command not found: python`

Cause: On newer Macs, python is not aliased — use python3 or activate venv properly.

Fix: ```bash

Option 1: use python3

python3 seed_mlflow.py

Option 2: recreate venv with correct python

uv venv --python /opt/homebrew/opt/python@3.11/bin/python3.11 source .venv/bin/activate

now 'python' works


---

## Do I Need MLflow UI Running?

**No.** The MCP server reads directly from the SQLite database. You do NOT need to keep `mlflow ui` running for Claude to query your experiments.

---

## Tech Stack

- **FastMCP** — MCP server framework
- **MLflow 3.x** — Experiment tracking (SQLite backend)
- **pandas** — DataFrame processing for run results
- **Claude Desktop** — MCP client

---

## License

MIT

ML Experiment Tracker MCP