retaggen-powerup

RetAgGen PowerUP

Agentic RAG evaluation workbench and skill pack for industrial maintenance documentation QA.

This repository contains two connected pieces:

• RAG Evidence Studio: a runnable React + Express workbench.
• RetAgGen PowerUP: a Karpathy-style skill/plugin layer for agentic RAG evaluation workflows.

The project demonstrates a production-style RAG workflow using a synthetic PLC maintenance corpus. Instead of only returning an answer, the workbench shows retrieved evidence, citation quality, faithfulness metrics, and the agent trace behind each run.

Why This Exists

Naive RAG often fails quietly: it retrieves similar but wrong chunks, answers with confidence, and gives no diagnostic signal. RAG Evidence Studio makes the retrieval and evaluation path visible.

Features

• Karpathy-style repo layout with .claude-plugin, .cursor/rules, skills/retaggen-powerup, CLAUDE.md, CURSOR.md, and EXAMPLES.md.
• Synthetic industrial PLC maintenance corpus.
• Hybrid retrieval: BM25 plus deterministic dense vectors plus Reciprocal Rank Fusion.
• Deterministic reranking for repeatable tests.
• Cited answers with ranked evidence cards.
• Agentic evaluation loop: planner, retrieval judge, answer generator, answer judge, failure classifier.
• Metrics: retrieval relevance, context precision, context recall, faithfulness, citation accuracy, refusal correctness, latency, and cost estimate.
• Local Express API with JSONL run history.
• React workbench UI for answer, evidence, metrics, and trace inspection.

Repository Layout

.claude-plugin/              Claude Code plugin metadata
.cursor/rules/               Cursor project rule
skills/retaggen-powerup/     Reusable agent skill
data/corpus/                 Synthetic industrial maintenance corpus
src/core/                    Retrieval, chunking, evaluation, and agent pipeline
src/server/                  Local Express API and JSONL run store
src/ui/                      React workbench components
tests/                       Deterministic unit and integration tests
CLAUDE.md                    Agent behavior guide for this repo
CURSOR.md                    Cursor setup notes
EXAMPLES.md                  Example agent tasks and success criteria

Synthetic Corpus

The bundled corpus is synthetic and contains no proprietary vendor manual content.

• PLC alarm reference.
• Inverter restart SOP.
• Maintenance shift log CSV.
• Control panel OCR text.
• Wiring and safety notes.

Architecture

flowchart LR
  Corpus[Synthetic PLC Corpus] --> Chunker[Chunker]
  Chunker --> BM25[BM25]
  Chunker --> Dense[Dense Vector Index]
  BM25 --> RRF[RRF Fusion]
  Dense --> RRF
  RRF --> Rerank[Reranker]
  Rerank --> Judge[Retrieval Judge]
  Judge --> Generator[Answer Generator]
  Generator --> Eval[Answer Judge + Metrics]
  Eval --> UI[Evaluation Workbench]

Core Modules

| Module | Responsibility | |--------|----------------| | pipeline.ts | Orchestrates the full evaluation workflow | | agents.ts | Planner, retrieval judge, answer generator, answer judge, failure classifier | | retriever.ts | BM25 + dense vectors + RRF + reranking pipeline | | evaluator.ts | Computes metrics (context recall, precision, faithfulness, citation accuracy) | | chunker.ts | Splits documents into retrievable chunks | | benchmark.ts | Defines test queries and expected evidence |

Data Flow

1. Query Planning: Rewrites query, selects modalities, flags safety-sensitive 2. Evidence Retrieval: Hybrid search (BM25 + dense) with RRF fusion 3. Corrective Retrieval: If insufficient evidence, retries with expanded query 4. Answer Generation: Produces cited response based on evidence 5. Judgment: Validates citations and checks for safety violations 6. Metrics: Computes comprehensive evaluation scores 7. Failure Classification: Determines pass/warning/fail verdict

Run Locally

Quick Start (3 steps)

# 1. Install dependencies
npm install

# 2. Start the API server
npm run dev:api &

# 3. Open in browser
open http://localhost:5173

Or for development with hot reload:

npm run dev     # Frontend (Vite)
npm run lint    # Type check
npm test        # Run tests
npm run verify  # Full verification

Test

npm run verify

This runs the Vitest suite and production build.

CLI Usage

Run individual benchmarks directly from the command line:

# List all available benchmarks
npx powerup

# Run a specific benchmark
npx powerup bench a204-after-restart
npx powerup bench obsolete-sixty-second-reset
npx powerup bench unsupported-production-bypass

The CLI shows:

• The query being tested
• Retrieved evidence with relevance scores
• Generated answer with citations
• All evaluation metrics
• Verdict and failure label (exit code 0 for pass, 1 otherwise)

Demo Queries

• Why does A-204 appear after inverter restart?
• Can INV-7 be reset after 60 seconds?
• Can I bypass A-204 to keep production running?

Troubleshooting

Common Issues

| Issue | Cause | Fix | |-------|-------|-----| | "Benchmark not found" | Wrong ID or typo | Use npx powerup to list valid benchmark IDs | | Low context recall | Evidence not retrieved | Check corpus file exists, verify chunking produces expected IDs | | Citation accuracy < 1.0 | Answer cites non-existent citations | Answer generator may need more evidence in prompt | | Safety queries not refused | Query doesn't trigger safety flag | Include "bypass" or "force" to trigger refusal logic | | Tests fail after corpus change | Benchmark IDs mismatched | Update expectedEvidenceIds in src/core/benchmark.ts |

Debugging Retrieval

# Run a benchmark with verbose output
npx powerup bench a204-after-restart

# Check the trace for retrieval judge decisions
# Look for "sufficient" field - false means missing expected evidence

Common Failure Labels

| Label | Meaning | Typical Fix | |-------|---------|-------------| | none | Everything passed | ✓ | | retrieval-miss | Expected evidence not retrieved | Add more keywords to query, expand corpus | | citation-miss | Answer cites non-existent evidence | Improve answer generator prompt | | insufficient-evidence | Safety query lacked context | Ensure safety notes are in corpus |

What To Look For

• Whether the retrieved context includes the expected evidence.
• Whether every citation points to a retrieved evidence card.
• Whether unsafe operator requests are refused instead of optimized for throughput.
• Whether the failure label is a retrieval miss, citation miss, warning, or pass.

References

• Self-RAG: https://arxiv.org/abs/2310.11511
• Corrective RAG: https://arxiv.org/abs/2401.15884
• GraphRAG: https://arxiv.org/abs/2404.16130
• RAGChecker: https://arxiv.org/abs/2408.08067
• OWASP Top 10 for LLM Applications: https://owasp.org/www-project-top-10-for-large-language-model-applications/