data-aggregator-mcp

🔎 data-aggregator-mcp

One MCP server to find and fetch research data across archives, omics registries, and literature — behind a single normalized model.

      

search one query across 17 sources — Zenodo, DataCite (Dryad / Figshare / Dataverse / OSF / OpenNeuro / Mendeley), NCBI omics (GEO / SRA / BioProject), BioStudies (EBI, incl. ArrayExpress), literature (PubMed / OpenAIRE), HuggingFace datasets, DataONE (eco / environmental), OmicsDI (proteomics / metabolomics), DANDI (neurophysiology), CZ CELLxGENE (single-cell), OpenML (ML datasets), RCSB PDB (structures), UniProtKB (proteins), the GWAS Catalog, GBIF (biodiversity), data.gov (US federal open data), and NASA CMR (Earth science) — deduplicated, normalized, and cross-linked. resolve any hit to its file manifest, citation, trust signals, and the data it points at. fetch it to disk with checksum verification.

mcp-name: io.github.musharna/data-aggregator-mcp

<p align="center"> <img src="https://raw.githubusercontent.com/musharna/data-aggregator-mcp/main/examples/assets/demo.svg" alt="data-aggregator-mcp stdio demo — initialize, tools/list (search, resolve, fetch, operate, relate, list_sources), and a live list_sources call showing the wired sources across archives, omics, and literature" width="820"> </p>

✨ Why this

Most data MCPs wrap a single source. This one unifies them behind six tools and one DataResource model, so an agent searches once and gets back comparable records:

• Multi-domain, one model — generalist archives + raw omics + literature,

deduplicated by DOI (the fetchable record wins over bare metadata).

• Taxonomy synonym expansion — organism="Orobanche aegyptiaca" also matches

Phelipanche aegyptiaca (NCBI Taxonomy), so a species rename doesn't cost you results.

• Paper → data bridge — resolve a paper and get links to the GEO / SRA /

BioProject / DataCite records it produced.

• Verified fetch — streams to disk with md5 verification where the source

exposes a checksum, optional archive unpacking, and a fail-loud integrity sniff that rejects an HTML paywall page served as a "PDF".

• Citations, access & full text — render a citation in any CSL style, get

normalized access/license, and pull open-access full text — all in one resolve.

• Trust signals — usage metrics (citations / views / downloads / likes),

version status (is_latest / superseded_by), and last_updated freshness, surfaced wherever the source exposes them.

• Interop exports — resolve(format="croissant") or "ro-crate" hands a

dataset to an ML or research-packaging pipeline as standard JSON-LD.

• Operate on data in place — operate reads the schema, previews rows, or

runs a read-only SQL SELECT against a remote Parquet/CSV/TSV without downloading it (Parquet footer + DuckDB httpfs range reads). Optional [operate] extra; base install is unchanged.

• Relate across records — relate takes a handful of resolved ids and

reports how they connect — shared accession, shared cross-identifier, an explicit link, or version lineage — naming the literal shared value as evidence. Metadata hints only: it never reads files or executes a join.

→ Full rationale and a comparison vs. single-source servers, breadth gateways, and ML-dataset tools: docs/POSITIONING.md.

<p align="center"> <img src="https://raw.githubusercontent.com/musharna/data-aggregator-mcp/main/docs/assets/architecture.svg" alt="Architecture: an MCP client speaks stdio to data-aggregator-mcp's six tools, which fan out through one router (DOI dedup, ontology expansion, ranking) to archives (Zenodo, DataCite, HuggingFace, DataONE, OpenML, RCSB PDB), omics (GEO, SRA, BioProject, OmicsDI, DANDI, CELLxGENE, GWAS Catalog), and literature (PubMed, OpenAIRE, EuropePMC, Unpaywall)" width="760"> </p>

⚡ Quickstart

Run with no install:

uvx data-aggregator-mcp

Register with Claude Code:

claude mcp add data-aggregator -- uvx data-aggregator-mcp

A typical agent flow:

search("drought stress RNA-seq", organism="Sorghum bicolor")
  → [ geo:GSE..., sra:SRX..., zenodo:..., pubmed:... ]   # deduped, taxa-normalized

resolve("sra:SRX079566")
  → DataResource{ files: [ENA FASTQ urls…], access: "open", taxa: [...] }

fetch("sra:SRX079566", dest="./data")
  → ["./data/SRX079566_1.fastq.gz", …]                   # md5-verified

<details> <summary>Other ways to run (pip, python -m, raw client config)</summary>

pip install data-aggregator-mcp
data-aggregator-mcp        # or: python -m data_aggregator_mcp

To use the operate tool (query remote tabular files in place), install the optional extra:

pip install "data-aggregator-mcp[operate]"

Add to a client's MCP config (e.g. Claude Desktop claude_desktop_config.json):

{
  "mcpServers": {
    "data-aggregator": {
      "command": "uvx",
      "args": ["data-aggregator-mcp"],
      "env": { "NCBI_API_KEY": "your-optional-key" }
    }
  }
}

</details>

🌐 Transports

stdio (default) — the server runs as a child of the client, so fetch() writes to your own disk. Nothing to configure; every command above uses it.

Streamable HTTP — the same six tools, prompts, and resources over HTTP:

data-aggregator-mcp --transport http     # → http://127.0.0.1:8000/mcp/

| flag | default | notes | | -------------------------- | ---------------- | ----------------------------------------------------------------- | | --transport {stdio,http} | stdio | | | --host | 127.0.0.1 | this machine only; any non-loopback value requires --allow-host | | --port | 8000 | | | --allow-host HOST:PORT | auto on loopback | permitted Host header, repeatable — required off loopback | | --allow-origin ORIGIN | derived | permitted browser Origin header, repeatable | | --stateless | off | fresh transport per request, no session affinity | | --json-response | off | plain JSON responses instead of SSE streams |

The endpoint is served at /mcp/ — with the trailing slash. /mcp answers 307 redirecting there, which is fine for any client that follows redirects (a 307 preserves the POST body); point one that doesn't straight at /mcp/. In stateful mode, sessions idle for 30 minutes are reaped.

DNS-rebinding protection is always on. A loopback bind derives its own host/origin allowlist, so the default needs no configuration. A non-loopback bind (--host 0.0.0.0, a LAN address, a container interface) refuses to start without at least one explicit --allow-host — guessing an allowlist there is precisely the hole the protection exists to close, so it fails loud instead of open:

data-aggregator-mcp --transport http --host 0.0.0.0 \
  --allow-host data.example.org:8000

Once running, a request whose Host header is outside the allowlist is refused with 421 Invalid Host header.

⚠️ fetch(dest=…) writes to the _server's_ filesystem, not the client's. Over stdio those are the same disk; over HTTP they may be different machines, and the caller gets back paths it cannot read. Treat dest on an HTTP deployment as server-side staging, or use stdio when you need the bytes locally. search, resolve, operate, relate, and list_sources are unaffected — they return data, not paths.

🗂️ Sources

| Source | Discover | Fetch | Checksum | | ---------------------------- | :------: | :---------------: | :--------------: | | Zenodo | ✅ | ✅ | md5 | | DataCite → Figshare | ✅ | ✅ | md5 | | DataCite → Dataverse | ✅ | ✅ | md5 | | DataCite → OSF | ✅ | ✅ | md5 | | DataCite → Dryad | ✅ | manifest only¹ | sha-256 (listed) | | DataCite → Mendeley & others | ✅ | — | — | | NCBI SRA | ✅ | ✅ (ENA FASTQ) | md5 | | NCBI GEO | ✅ | ✅ (suppl/) | none² | | NCBI BioProject | ✅ | → SRA links | — | | PubMed / OpenAIRE | ✅ | ✅ (OA full text) | none² | | HuggingFace datasets | ✅ | ✅ (resolve URL) | none | | DataONE (eco/env) | ✅ | ✅ (Member Node) | md5 / sha-256 | | OmicsDI → PRIDE | ✅ | ✅ (HTTPS FTP) | size only | | OmicsDI → MetaboLights | ✅ | ✅ (HTTPS FTP) | none | | OmicsDI → other MS repos | ✅ | — | — | | DataCite → OpenNeuro | ✅ | ✅ (snapshot) | none² | | DANDI (neurophysiology) | ✅ | ✅ (302→S3) | none² | | CZ CELLxGENE (single-cell) | ✅ | ✅ (H5AD/RDS) | none² | | OpenML (ML datasets) | ✅ | ✅ (ARFF) | md5 | | RCSB PDB (structures) | ✅ | ✅ (.cif/.pdb) | none² | | GWAS Catalog | ✅ | → PMID bridge | — |

¹ Dryad downloads are token / bot-challenge gated, so fetch fails loud; resolve still lists the files. ² No upstream checksum — fetch verifies content-type instead (rejects an HTML page served in place of a binary).

🛠️ Tools

search(query?, size?, sources?, organism?, disease?, tissue?, chemical?, assay?, kind?, published_after?, published_before?, rank?, cursor?, collapse_mirrors?, understand?, multi_query?, provenance?)

Fan out across all wired sources in parallel and return compact DataResource records, deduped by DOI. Per-source failures land in errors{} — never silently dropped.

• organism — expand the query with NCBI-Taxonomy synonyms; the expansion is

echoed in taxon_expansion, and results carry normalized taxa[] ({taxid, name}) plus a described_in link to plant-genomics-mcp for plant taxa.

• sources — restrict the fan-out, e.g. ["omics"].
• size — max results (1–50).
• kind — keep only dataset / sequencing_run / study / publication /

software.

• published_after / published_before — filter by publication year.
• rank — relevance (default) or semantic (re-rank the fetched page by

embedding similarity to the query; needs EMBEDDING_API_BASE, degrades to relevance order otherwise).

• understand — opt into LLM query understanding (default false). A free-text

query is normalized into a focused keyword query: conversational fluff ("I'm looking for…", "where can I find…") is stripped while the scientific and entity terms are kept so they still match by text. The LLM also detects structured entities (organism/disease/tissue/chemical/assay, kind) — these are echoed in query_understanding.extracted for transparency but not auto-applied, because ANDing LLM-_inferred_ facets across free-text keyword upstreams over-constrains and hurts recall. Only the cleaned keyword_core and explicit year scopes are applied; the ontology resolvers still run on the facets you pass (the LLM proposes, you dispose). Needs an LLM endpoint (LLM_API_BASE); with none configured the search runs unchanged and notes it in errors['understand']. Effectiveness is query- and model-dependent — opt-in / default-off; validate the recall lift on your own corpus and LLM (see the eval harness below). On our small verified set multi_query= is the stronger, always-safe recall lever; understand= is approximately neutral with a weak local model.

• multi_query — opt into diverse multi-query recall expansion (default false).

An LLM generates up to a few deliberately-diverse reformulations of your query (different facets/synonyms/framings, not paraphrases), each is fanned out across every source, and the deduped union is re-ranked against your original query — surfacing relevant records a single keyword query would miss. Bounded at MAX_QUERY_VARIANTS (4, incl. the original, which is always kept so recall never drops below baseline), so it costs at most N× the upstream calls. Composes with understand= (which structures variant 0). The variants used are echoed in query_expansion. Needs an LLM endpoint (LLM_API_BASE); with none configured the search runs as a normal single query and notes it in errors['multi_query'].

• cursor — opaque token from a prior result's next_cursor; pages forward

across every source. In cursor mode the other params are read from the token, so query is optional.

resolve(id, cite?, format?, trust?, fair?, use?)

Full record + files manifest. Routes by id shape — zenodo:7654321, a bare DOI, datacite:10.5061/dryad.x, an omics id (sra:SRX079566, geo:GSE332789, bioproject:PRJNA1468572), a literature id (pubmed:34320281, openaire:<id>), a HuggingFace id (hf:owner/name), a DataONE id (dataone:doi:10.5063/F1HT2M7Q), or an OmicsDI id (omicsdi:pride:PXD000001). Attaches, where available:

• files[] — ENA FASTQ manifest (SRA), GEO suppl/, or the host repo's

native manifest (Figshare / Dataverse / OSF / Dryad).

• links[] — paper → data: pubmed: → sra: / geo: / bioproject: (NCBI

elink); openaire: → datacite: (ScholeXplorer Scholix).

• access / license — normalized status

(open / embargoed / restricted / closed / unknown) and license where the source exposes it.

• identifiers — normalized {pmid, pmcid, doi}, plus an open-access

full-text FileEntry (EuropePMC XML, or an Unpaywall PDF fallback) for papers.

• citation — pass cite=<format>: bibtex, ris, csl-json, or any CSL

style name (apa, mla, vancouver, …). DOI records use content negotiation; others render CSL-JSON from metadata. Off by default; failures degrade quietly.

• trust signals — metrics (citations / views / downloads / likes),

is_latest / superseded_by (derived from version links), and last_updated freshness, where the source provides them.

• trust=true — attach retraction status (via Crossref) under trust{}.

One extra Crossref call; meaningful for DOI-bearing records only.

• fair=true — attach an RDA-grounded FAIRness score (0–100 + F/A/I/R

sub-scores + actionable gaps) computed from the record metadata under fair{}. Pure/local — no extra network call.

• use=<intent> — attach a licence-compatibility advisory under

license_compat{} for the intended use (commercial / redistribute / modify / ml-training). Returns ALLOW/REVIEW/DENY with the governing clause. Metadata-derived advisory, not legal advice; an absent/unrecognized licence yields REVIEW.

• format — pass format="croissant" (file-level Croissant JSON-LD),

"ro-crate" (minimal RO-Crate 1.1), or "provenance" (one-call RO-Crate 1.1 data-availability dossier bundling version-currency, licence+SPDX, FAIR score, and retraction status) to attach a standard manifest under the matching field.

fetch(id, dest?, files?, max_bytes?, force?, extract?)

Download files to disk and return their paths. Streams under a max_bytes guard (force to override) with md5 verification wherever a checksum exists.

• files — restrict to a subset of the resolved manifest.
• extract — unpack downloaded zip / tar archives in place, guarded against

path traversal and runaway extracted size. Off by default.

• Unverified fetches (GEO suppl/, literature full text) get a content-type

sniff that fails loud if a declared binary is actually an HTML page.

• Fetchable: Zenodo, SRA, GEO, DataONE (Member-Node objects,

md5/sha-256 verified), DataCite-hosted Figshare / Dataverse / OSF, HuggingFace datasets, PRIDE / MetaboLights (via OmicsDI, unverified), and literature open-access full text. Dryad, other DataCite repos, and other OmicsDI repos (MassIVE / GNPS / ...) are discovery-only and raise FetchNotSupportedError.

list_sources()

Wired sources with their capabilities — layer, kinds, supported filters, fetchability, operable flag, id examples, auth, and rate limits.

operate(op, id, file?, query?, n?, columns?)

Inspect or query a remote tabular file (Parquet / CSV / TSV) without downloading it. Addresses a file by catalog id + file name (defaults to the first tabular file on the resolved record). Ops:

• schema — column names + types (reads the Parquet footer / sniffs the CSV

header; no full load).

• preview — a small sample of rows.
• head — the first n rows (default 20), optionally restricted to columns.
• sql — a read-only SELECT (the file is the view data), e.g.

SELECT col, count(*) FROM data GROUP BY 1.

• peek — per-column profile via DuckDB SUMMARIZE (type, null-rate,

approximate distinct count, min/max, numeric quartiles) without downloading the file. Like head/sql, reads the whole file and honors the source-size ceiling.

Backed by the Parquet footer reader + DuckDB httpfs range reads. sql runs in a locked-down DuckDB (read-only, local filesystem disabled, single-SELECT validation, row / wall-clock caps). Requires the optional [operate] extra (pip install data-aggregator-mcp[operate]); without it, operate returns a clear install-the-extra message and the other four tools are unaffected.

Any HuggingFace dataset with a datasets-server converted view is operable (schema / preview / head / sql): resolve surfaces the auto-converted Parquet files (source="hf-datasets-server") even for datasets stored as JSON/JSONL/arrow, so pass file=<config>/<split>/...parquet to pick a split when there are several.

relate(ids)

Cross-resource join/harmonization hints. Given 2–10 resource ids, relate resolves each (TTL-cached) and reports how they relate and on what key they could be joined:

• shared_accession — same BioProject/SRA/GEO accession on ≥2 records → joinable key.
• shared_identifier — same doi/pmid/pmcid across records → same work / paper↔data link.
• explicit_link — one record's links[] points at another input record.
• version_lineage — one record supersedes another (dedupe, don't join, those).

Hints only. relate never reads file columns, fetches files, or executes a join/merge/conversion — every hint names the shared value as evidence. Per-id resolve failures are reported in errors, not fatal; an empty result carries an explanatory note.

Prompts

Three workflow prompts surface in clients (e.g. /mcp__data_aggregator__* in Claude Code):

• find_data — find datasets for a topic, optionally scoped to an organism.
• data_behind_paper — find the datasets / accessions behind a paper.
• search_resolve_fetch — walk the end-to-end search → resolve → fetch flow.

⚙️ Configuration

All optional, set via environment variables:

• NCBI_API_KEY — raises the NCBI E-utilities rate limit (3 → 10 req/s) used by

the omics, literature, and taxonomy lookups.

• DATA_GOV_API_KEY — a free api.data.gov key for

the data.gov source. Absent ⇒ requests fall back to the shared public DEMO_KEY, rate-limited to roughly 30 requests/hour per IP — fine for light discovery, worth setting for anything heavier.

• UNPAYWALL_EMAIL — enables the Unpaywall fallback leg of literature full-text

retrieval (the EuropePMC leg works without it).

• NCBI_EMAIL — contact address sent to NCBI's ID converter; falls back to

UNPAYWALL_EMAIL when unset.

• DATAVERSE_BASE_URL — resolve Dataverse DOIs against a different installation

(default https://dataverse.harvard.edu).

• CACHE_TTL_SECONDS — resolve-cache lifetime in seconds (default 3600; an

unparseable value falls back to that default).

• EMBEDDING_API_BASE / EMBEDDING_API_KEY / EMBEDDING_MODEL — an

OpenAI-compatible embeddings endpoint enabling rank=semantic. Absent ⇒ semantic re-rank degrades to relevance order. Key is optional (keyless local servers supported); model defaults to text-embedding-3-small.

• LLM_API_BASE / LLM_API_KEY / LLM_MODEL — an OpenAI-compatible

/chat/completions endpoint enabling search(understand=true) (NL→structured query rewriting) and search(multi_query=true) (diverse multi-query recall expansion). Absent ⇒ both run the raw query unchanged and note it in errors['understand'] / errors['multi_query']. Key is optional (keyless local servers supported); model defaults to gpt-4o-mini (a passthrough string — set it to whatever your endpoint serves). multi_query fans out at most MAX_QUERY_VARIANTS (4, incl. the original) variants, bounding the N× cost.

To measure the recall lift of understand=true / multi_query=true on a small labeled set, run the gated eval harnesses (need a live LLM endpoint):

DATA_AGGREGATOR_MCP_LIVE=1 LLM_API_BASE=... python scripts/eval_understand.py
DATA_AGGREGATOR_MCP_LIVE=1 LLM_API_BASE=... python scripts/eval_multi_query.py

They print per-query and mean recall@20 (understand / multi-query off vs. on). See the fixtures at scripts/eval_understand_fixture.json and scripts/eval_multi_query_fixture.json.

🧪 Develop

uv venv && uv pip install -e ".[dev]"
uv run pytest -q
uv run ruff check src tests
DATA_AGGREGATOR_MCP_LIVE=1 uv run pytest -k live -q   # real-API probes

The README demo (examples/assets/demo.svg) is recorded network-free from examples/_demo_stdio.py — see the header of that file to re-record.

License

MIT — see LICENSE.