Self-Improving Agentic RAG

The problem

Most RAG systems treat retrieval quality as fixed — tuned once, shipped, and trusted from then on. I wanted a system that could tell why an answer went wrong (bad retrieval vs. bad reasoning over good retrieval), catch unsupported answers before returning them, and actually get better over time instead of staying static.

What I built

A full-stack RAG platform with a clean separation between ingestion, retrieval, generation, verification, and improvement — orchestrated as an explicit LangGraph pipeline rather than a single opaque prompt:

• Hybrid retrieval — dense vector search (ChromaDB) combined with BM25 sparse keyword search, merged via Reciprocal Rank Fusion to balance semantic and lexical recall
• Cross-encoder reranking — retrieved chunks are rescored before generation to surface the strongest evidence, not just the first-pass matches
• Query understanding & rewriting — intent and keywords are extracted before search runs, with learned keyword boosts applied on rewrite
• Answer verification — a dedicated stage checks whether the generated answer is actually supported by the retrieved evidence, and can trigger regeneration when it isn't
• Self-improvement loop — when a response is marked incorrect, the system classifies why (a retrieval failure vs. a generation failure) and updates retrieval or prompting strategy accordingly
• Full traceability — every answer ships with source references, retrieval metadata, and pipeline diagnostics, not just a final string
• RAGAS evaluation harness — tracks faithfulness, answer relevancy, context recall, context precision, and user-feedback accuracy over accumulated real feedback, not a one-time benchmark

Backend on FastAPI, frontend on React with a chat interface, document upload, feedback controls, and a trace viewer so the pipeline's reasoning isn't a black box even to the person using it.

Why it matters

The self-improvement loop is the part I'd defend hardest: most "RAG projects" stop at generation. This one treats incorrect answers as training signal — classifying the failure mode first, then feeding that back into retrieval and prompting — which is the difference between a system that's tuned once and one that's designed to get better under real usage.