confluence

Hybrid Agentic RAG — LangChain + LangGraph + Evaluation

This project implements hybrid agentic RAG over Confluence using Python, LangChain, LangGraph, FAISS, BM25Retriever, Cohere reranker, OpenAI embeddings, RAGAS, BERTScore, LangSmith, and FastMCP. An MCP server and Streamlit chatbot are implemented as core functionalities, with a full evaluation suite and LangSmith monitoring.

---

Table of Contents

• How it Differs from Hybrid RAG
• Setup
• Pipeline
• Testing Without a Confluence Instance
• Monitoring: LangSmith
• Evaluation: RAGAS + Custom Metrics
• MCP Server

---

How it differs from hybrid RAG

The retrieval quality and pipeline structure are identical. What changes is the underlying library stack and what you get on top of it.

| Component | Hybrid RAG | This project (LangChain) | |-----------|------------|--------------------------| | BM25 | bm25s | BM25Retriever (langchain-community) | | Dense index | numpy .npy | FAISS vectorstore (langchain-community) | | RRF fusion | custom _rrf() | EnsembleRetriever (same c=60 constant) | | Reranking | cohere.Client() | CohereRerank + ContextualCompressionRetriever | | Agent | pydantic-ai | create_react_agent (LangGraph) | | Tracing | none | LangSmith (automatic, zero code) | | Evaluation | none | RAGAS + custom MRR/NDCG/BERTScore |

---

Setup

uv pip install -r requirements.txt
cp .env.example .env
# Fill in CONFLUENCE_*, OPENAI_API_KEY, COHERE_API_KEY, ANTHROPIC_API_KEY
# Optionally add LANGCHAIN_API_KEY for LangSmith monitoring

---

Pipeline

1-fetch-confluence.py   Fetch Confluence pages → chunks/*.json
1-fetch-k8s.py          Fetch public K8s docs  → chunks/*.json  (no account needed)
        ↓
2-build-index.py        Build indexes/bm25.pkl + indexes/faiss/ + indexes/meta.json
        ↓
3-hybrid-search.py      Interactive CLI to test the four-stage retrieval chain
        ↓
4-agent.py              LangGraph ReAct agent with structured citations
        ↓
5-evaluate.py           Full evaluation suite (retrieval + generator + end-to-end)
        ↓
6-mcp-server.py         FastMCP server for Claude Desktop / Cursor / Claude Code
7-chatbot.py            Streamlit chatbot with LangSmith trace links

Run each step

uv run 1-fetch-confluence.py
uv run 2-build-index.py
uv run 3-hybrid-search.py
uv run 4-agent.py "What is our deployment process?"
uv run 5-evaluate.py
uv run 6-mcp-server.py
uv run streamlit run 7-chatbot.py

---

Testing without a Confluence instance

1-fetch-k8s.py scrapes the public Kubernetes documentation (kubernetes.io/docs) and saves chunks in the exact same JSON format that 2-build-index.py expects, so the full pipeline — indexes, agent, evaluation, MCP server, chatbot — runs unchanged. No Confluence account or API token needed.

Prerequisites

# beautifulsoup4 is the only extra dependency
uv pip install beautifulsoup4

Running the scraper

# Scrape ~190 Kubernetes docs pages (≈ 2 min at 0.5 s/request)
uv run 1-fetch-k8s.py

# Then continue with the normal pipeline
uv run 2-build-index.py
uv run 3-hybrid-search.py
uv run 6-mcp-server.py              # terminal 1
uv run streamlit run 7-chatbot.py   # terminal 2

Expected output:

Fetching sitemap: https://kubernetes.io/en/sitemap.xml
Found 192 pages to index

[  1/192] Concepts                                           3 chunk(s)
[  2/192] Kubernetes Components                              4 chunk(s)
...
Done: 189 pages → 847 chunks  (2 empty, 1 errors)
Chunks saved to: .../chunks/

Configuration

Edit the constants at the top of 1-fetch-k8s.py:

| Constant | Default | What it controls | |---|---|---| | INCLUDE_SECTIONS | concepts/, tasks/, tutorials/, setup/, reference/glossary/, reference/kubectl/ | Sections of kubernetes.io/docs to crawl | | SKIP_PATTERNS | reference/kubernetes-api/, contribute/ | Sub-paths excluded even within included sections | | REQUEST_DELAY | 0.5 s | Pause between HTTP requests | | MAX_CHUNK_CHARS | 1500 | Maximum characters per chunk | | OVERLAP_CHARS | 150 | Overlap between consecutive chunks |

Example queries once running

• "What is the difference between a Deployment and a StatefulSet?"
• "How do I configure resource limits for a Pod?"
• "What happens when a node fails?"
• "How does the Kubernetes scheduler decide where to place a Pod?"

For evaluation, add K8s questions to eval_set.json and run uv run 5-evaluate.py to get MRR, NDCG@10, RAGAS faithfulness, and BERTScore against this corpus.

---

Monitoring: LangSmith

Add three lines to your .env:

LANGCHAIN_TRACING_V2=true
LANGCHAIN_API_KEY=ls_...
LANGCHAIN_PROJECT=hybrid-agentic-rag

That's it. Every LangChain and LangGraph call — embedding lookups, retriever invocations, LLM completions, tool calls — is automatically traced. Open smith.langchain.com to see latency breakdowns, token usage, and full input/output for every step.

Free tier: 5,000 traces/month.

---

Evaluation: RAGAS + custom metrics

Edit eval_set.json to add your own questions:

[
  {
    "question": "What is our deployment process?",
    "ground_truth": "...",
    "relevant_chunk_ids": ["page123_c0", "page456_c2"]
  }
]

• ground_truth: needed for ROUGE, BERTScore, context precision/recall
• relevant_chunk_ids: needed for MRR, NDCG@10, Precision@10, Recall@10

(chunk IDs are visible in 3-hybrid-search.py output)

If fields are omitted, those metric groups are skipped gracefully.

Metrics covered

| Category | Metric | Implementation | |----------|--------|---------------| | Retrieval | MRR | utils/evaluation.py | | Retrieval | NDCG@10 | utils/evaluation.py | | Retrieval | Precision@10 | utils/evaluation.py | | Retrieval | Recall@10 | utils/evaluation.py | | Generator | ROUGE-1/2/L | rouge-score library | | Generator | BERTScore F1 | bert-score library | | End-to-End | Faithfulness | RAGAS (LLM-as-judge) | | End-to-End | Answer Relevancy | RAGAS (LLM-as-judge) | | End-to-End | Context Precision | RAGAS (needs ground_truth) | | End-to-End | Context Recall | RAGAS (needs ground_truth) | | Monitoring | Latency, tokens, errors | LangSmith (automatic) |

RAGAS uses OpenAI gpt-4o-mini as judge by default. Override with RAGAS_JUDGE_MODEL=gpt-4o in .env for higher accuracy.

---

MCP server

uv run 6-mcp-server.py   # starts on http://localhost:8051/sse

Add to Claude Desktop config (%APPDATA%\Claude\claude_desktop_config.json):

{
  "mcpServers": {
    "confluence": { "url": "http://localhost:8051/sse" }
  }
}

Or to .mcp.json in any repo root for Claude Code:

{
  "mcpServers": {
    "confluence": { "type": "sse", "url": "http://localhost:8051/sse" }
  }
}