State of RAG — Q2 2026

Retrieval-augmented generation entered Q2 2026 as the default architecture for nearly every production LLM application that touches proprietary data. It also entered Q2 2026 with the most credible challenge it has ever faced: long-context models that genuinely make “just stuff the documents into context” a viable alternative for many workloads.

The takeaway from a quarter of Notifire coverage: RAG is not going away, but the design space has split. Long-context for slow, infrequent, high-value queries. Retrieval for cost-sensitive, latency-sensitive, freshness-sensitive workloads. The teams shipping the best RAG systems in 2026 know which side of that split each of their queries sits on, and design accordingly.

Three substantive shifts. First, graph-enhanced RAG crossed the production threshold. Neo4j shipped a first-class LLM tool-calling integration; TigerGraph followed with hybrid vector + graph indexes; LangChain and LlamaIndex both added graph retrievers to their default builders. We covered four production case studies of graph RAG replacing pure vector retrieval, each citing 20–40% accuracy gains on multi-hop questions.

Second, the long-context regime matured. Claude Sonnet 4.5 at 1M context with caching, Gemini 2.5 Pro at 2M, and GPT-5 holding at 400k all became reliable enough for production use. The cost economics shifted with them — caching means a 500-page document loaded once is essentially free on subsequent queries within the cache TTL. For workloads where one document is reused many times, long-context now beats retrieval on total cost.

Third, evaluation consolidated. Ragas hit 1.0 in April. OpenAI Evals open-sourced its full harness. Anthropic shipped Inspect with first-class RAG metrics. The “every team rolls their own scorer” era is over; CI gates with standardised metrics are now the default for production RAG.

The classic vector-RAG failure mode is the multi-hop question: “Which of our suppliers source materials from companies sanctioned in 2024?” Vector search finds documents that mention suppliers and documents that mention sanctions, but cannot reason across the relationships between them. Graph RAG fixes this by augmenting the vector store with explicit relationship metadata — entities, edges, and traversal queries — that the LLM can consult during retrieval.

The architecture is two-step. First, an ingestion pipeline extracts entities and relationships from source documents into a graph database (Neo4j, TigerGraph, KuzuDB, FalkorDB). Second, at query time, the LLM is given access to both vector search (for semantic similarity) and graph traversal (for structural reasoning) as separate tools, and decides which to invoke based on the question shape.

The trade-off is operational complexity. A graph database adds a second system to operate, and the entity-extraction quality bounds the whole approach — bad NER produces a bad graph and a worse-than-vector-only RAG. Teams shipping graph RAG in 2026 typically use an LLM-driven extraction pipeline with a human review gate on a sample of every batch.

The popular framing — “long context will replace RAG” — was overstated. The accurate framing: long context replaces RAG for a specific subset of workloads. Specifically: workloads where the same document or document set is queried repeatedly within a short window, where freshness doesn’t matter, where latency tolerance is several seconds, and where the per-query cost is acceptable.

Concrete example: a legal team analysing a single contract for 30 questions over an hour. Loading the contract once into a cached context costs the input-token price once; each of the 30 questions reuses the cache at the cache-read rate (typically 10× cheaper than input). That’s strictly cheaper than embedding the contract, querying a vector store 30 times, and paying retrieval-then-generate per query.

Counter-example: a customer-support assistant answering questions against a 50,000-document knowledge base. The base is too large to fit in context (even at 2M tokens), changes daily, and queries arrive from many users with different question patterns. RAG remains the only viable architecture.

RAG eval at the start of 2026 was a mess of ad-hoc scoring scripts, hand-curated test sets, and unstandardised metrics. By end of Q2 it had consolidated around three pieces: Ragas 1.0 for retrieval and faithfulness metrics, OpenAI Evals or Anthropic Inspect for full-pipeline scoring, and a CI gate that fails the build if scores regress.

The standard three-layer eval is: (1) retrieval precision/recall — did we find the right passages?; (2) faithfulness — does the answer match the retrieved text, or did the model hallucinate?; (3) answer quality — is the final response useful? Each layer has both deterministic (overlap, exact-match) and LLM-as-judge metrics. The CI gate runs daily against a frozen test set and pages an on-call engineer if any layer regresses below threshold.

What didn’t consolidate: test-set curation. Every team’s eval set looks different and the consensus is this is correct — your eval should reflect your users’ actual questions, not a generic benchmark. The standard advice is to mine real production queries (with user consent) into a test set, label answers in pairs, and treat the test set as a living document.

The vector-database market settled in Q2. Four real choices: pgvector (Postgres extension, fine up to ~100M vectors, killer feature is joined queries with relational data), Pinecone (managed serverless, dominant at the high end), Qdrant (self-hosted Rust, popular with cost-conscious teams), and Weaviate (self-hosted Go, popular with on-prem requirements). Milvus is still around but losing share to Qdrant.

The choice rarely matters as much as people fear. Embedding quality, chunk size, and retrieval-quality engineering matter far more than which vector store sits underneath. Most teams who switched between vector databases reported single-digit-percent change in eval scores.

If you’re already on Postgres and your scale is moderate, pgvector with HNSW indexing is the right answer until you have a specific performance reason to change. Notifire’s comparison page on pgvector vs Pinecone breaks down the crossover point in detail.

The boring secret of RAG quality is chunking. The standard recipe — split text every 1000 characters with 200 characters of overlap — works tolerably on prose but breaks on structured documents (code, tables, contracts) where the meaningful unit is a section, not a character window.

Three approaches gained traction in Q2. Semantic chunking (split where embedding similarity between adjacent sentences drops) is conceptually clean but slow and not always better. Hierarchical retrieval (retrieve coarse chunks, then re-retrieve within them) handles long documents well but doubles the retrieval cost. Late chunking (Jina’s approach: embed the entire document with a long-context embedding model, then chunk the resulting vector) is the most promising recent direction — it preserves document-level context inside each chunk’s vector.

We expect the next year to be a chunking-strategy bake-off. Notifire will cover whichever approach wins.

Pick the regime per workload, not per company. Some queries are long-context wins; others are RAG wins. The teams shipping the best systems route per query.

Adopt graph RAG only when you have multi-hop questions. If your queries are flat lookups (“find me the document about X”), graph RAG adds operational burden without helping. If your queries chain entities (“which X are affected by Y because of Z”), graph RAG is worth the complexity.

Industrialise your eval today. Ragas 1.0 + CI gate is a half-day setup that pays back the first time a model upgrade breaks production silently.

Use pgvector if you’re already on Postgres. The crossover point to a dedicated vector DB is higher than most teams assume.

Invest in chunking before reranking. Most teams reach for a reranker (Cohere Rerank, BGE-Reranker) to fix retrieval quality. Reranking helps, but better chunking helps more.