Verdict
Default to vector RAG and add a graph tier only when your query log proves you need it. The published evidence splits cleanly by query type: Microsoft's GraphRAG paper (arXiv 2404.16130) reports ~70-80% win rates over naive vector RAG on comprehensiveness and diversity for global, corpus-level questions - but the independent GraphRAG-vs-vanilla-RAG evaluation (arXiv 2506.05690) finds basic RAG comparable or better on simple fact retrieval, and cites graph-based retrieval averaging 2.3x higher latency on HotpotQA. The cost asymmetry is brutal: Microsoft's own LazyGraphRAG announcement states that LazyGraphRAG's indexing costs the same as vector RAG and just 0.1% of full GraphRAG's - which is Microsoft conceding that classic GraphRAG indexing runs roughly 1000x a vector index. If your workload genuinely needs global synthesis, LazyGraphRAG-style deferred summarization - not classic full indexing - is the third option that wins the cost row outright.
The two architectures do different work at different times. Vector RAG pays almost nothing at indexing - one embedding pass, no LLM calls in the pipeline - and answers local questions fast. GraphRAG pays enormously up front - every document chunk passes through LLM entity/relationship extraction, then hierarchical community summarization - to earn the ability to answer 'what are the main themes across this corpus,' which top-k similarity structurally cannot do because no single chunk contains the answer. Microsoft's LazyGraphRAG work (microsoft.com/en-us/research/blog, November 2024) reframed the tradeoff: comparable answer quality to GraphRAG Global Search at more than 700x lower query cost, and at 4% of the query cost it significantly outperforms competing methods on both local and global queries. Freshness matters here: the Azure GraphRAG solution accelerator (Azure-Samples/graphrag-accelerator) was archived on May 27, 2025, and Microsoft now surfaces GraphRAG and LazyGraphRAG technology through Microsoft Discovery, its agentic science platform - the microsoft/graphrag library remains the maintained open-source path. Treat the graph tier as a capability you buy for a measured class of queries, not a default upgrade.
Decision Table
| Criterion | Edge | Explanation |
|---|---|---|
| Indexing cost | Vector RAG (embed, ANN top-k retrieval, optional metadata filters) Best edge | Not close. Vector RAG's index is one embedding pass; GraphRAG's index is LLM entity extraction plus relationship extraction plus multi-level community summarization over the whole corpus. Microsoft's own LazyGraphRAG post states LazyGraphRAG's indexing cost is 'identical to vector RAG and 0.1% of full GraphRAG' - i.e., Microsoft's own framing puts classic GraphRAG indexing near 1000x a vector index, because every chunk pays LLM extraction and summarization tokens instead of one embedding call. Re-indexing on document updates repeats the premium, which is why incremental update (graphrag update) exists but remains the operational sore point. |
| Global, corpus-level synthesis ('what are the themes across these 10,000 documents') | GraphRAG (LLM-extracted entity graph + community summaries, global/local search) Best edge | This is the query class GraphRAG was built for and where its published wins are real: the original Microsoft paper (arXiv 2404.16130) reports ~70-80% win rates over naive RAG on comprehensiveness and diversity when using community summaries. Vector RAG structurally cannot answer these - no top-k of chunks contains a corpus-level answer, so the model synthesizes from an unrepresentative sample and presents it confidently. If your agent's job includes 'summarize what we know about X across everything,' plain vector retrieval is the wrong tool. |
| Single-hop factoid and detail lookup | Vector RAG (embed, ANN top-k retrieval, optional metadata filters) Best edge | The independent evaluation literature is consistent: arXiv 2506.05690 (GraphRAG vs vanilla RAG) finds basic RAG 'comparable to or outperforms GraphRAG in simple fact retrieval tasks,' and a follow-up systematic evaluation (arXiv 2502.11371) concludes RAG is stronger on single-hop, detail-oriented factual queries while graph methods win on multi-hop reasoning. For 'what plan is customer 4711 on' - the bread and butter of agent memory - the graph adds cost and latency without adding correctness. |
| Query latency in an agent's hot path | Vector RAG (embed, ANN top-k retrieval, optional metadata filters) Best edge | Graph retrieval traverses; vector retrieval does one ANN lookup. The evaluation cited in arXiv 2506.05690 reports graph-based retrieval averaging 2.3x higher latency than vanilla RAG on HotpotQA, and the systematic comparison in arXiv 2502.11371 notes KG-based GraphRAG variants have the highest retrieval latency of the methods tested due to LLM-driven entity expansion and multi-step traversal. An agent that calls retrieval several times per reasoning loop multiplies that penalty; for synchronous tool calls, vector retrieval is the latency-safe default. |
| Cost-efficient global search (the LazyGraphRAG row) | Tie | A third option beats both classic contenders here. Microsoft's LazyGraphRAG defers all LLM work to query time - no upfront summarization - and its published numbers are the strongest in this space: answer quality comparable to GraphRAG Global Search at more than 700x lower query cost, indexing at 0.1% of full GraphRAG, and at 4% of GraphRAG's query cost it 'significantly outperforms competing methods on both local and global queries, including GraphRAG Global Search' (microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost). If global synthesis is a minority query class - which query logs usually show - lazy/deferred graph construction dominates classic GraphRAG on economics. |
| Operational maturity and vendor posture | Vector RAG (embed, ANN top-k retrieval, optional metadata filters) Best edge | Vector RAG is boring infrastructure: pgvector in the Postgres you already run, mature ANN indexes, no LLM in the indexing path to version or re-run. GraphRAG's operational story has churned: the Azure solution accelerator (Azure-Samples/graphrag-accelerator) was archived May 27, 2025; Microsoft's project page now routes GraphRAG and LazyGraphRAG through Microsoft Discovery, and the maintained open-source path is the microsoft/graphrag library with graphrag update for incremental indexing. None of that makes the technique wrong - but a retrieval layer whose reference deployment was archived is a real operational-risk datapoint for a small team. |
Choose GraphRAG (LLM-extracted entity graph + community summaries, global/local search) if...
- Corpus-level synthesis workloads: research assistants, due-diligence and investigation tools, 'main themes' reporting over large document sets.
- Multi-hop reasoning where the answer requires traversing entity relationships no single chunk contains (arXiv 2502.11371's strongest GraphRAG category).
- Teams that can absorb an LLM-driven indexing pipeline as a batch cost on a mostly static corpus - and re-run it deliberately, not continuously.
- Products where answer comprehensiveness and diversity are the differentiator and per-query economics are secondary.
Choose Vector RAG (embed, ANN top-k retrieval, optional metadata filters) if...
- Agent tool-call retrieval in the hot path: single ANN lookup, predictable latency, no LLM in the retrieval step.
- Factoid and detail-oriented lookup over documents - the query class where independent evaluations show vanilla RAG comparable or better.
- Living corpora with frequent updates: re-embedding changed chunks is cheap; re-running graph extraction and community summarization is not.
- Small teams on boring infrastructure: pgvector in existing Postgres, no archived reference architectures, nothing new to operate.
Decision rules
- 01Pull two weeks of real agent queries and classify them: if fewer than ~10% are global/thematic ('across all documents...'), vector RAG alone is the right architecture - the graph tier would serve a rounding error at 1000x the indexing cost.
- 02If your agent must answer corpus-level synthesis questions and be defensibly comprehensive (research, diligence, intelligence work), budget for a graph tier: the ~70-80% comprehensiveness/diversity win rates from arXiv 2404.16130 are the published justification.
- 03If you need global synthesis but the corpus updates frequently, start with LazyGraphRAG-style deferred indexing rather than classic GraphRAG - indexing at vector-RAG cost with >700x lower query cost than Global Search removes the re-indexing tax that kills classic GraphRAG on living corpora.
- 04If retrieval sits in a synchronous agent loop with a latency budget, keep vector retrieval in the hot path and route only explicitly global questions to the graph tier - the measured 2.3x latency multiple compounds per reasoning iteration.
- 05For agent memory specifically (user facts, preferences, session recall), this decision is usually moot: that layer needs consolidation and recency logic, not community detection - see our mem0-vs-zep and pgvector-vs-qdrant comparisons for that stack.
- 06Never adopt the graph tier on benchmark win rates alone: the wins are query-class-specific, and the same literature shows vanilla RAG matching or beating GraphRAG on the factoid queries that dominate most production logs.
Migration Notes
- Adding a graph tier later is straightforward if you keep raw source documents and chunk provenance now - GraphRAG indexes from source text, so vector-first is not a one-way door.
- If you adopt microsoft/graphrag, plan the update path on day one: incremental indexing via graphrag update exists, but the accelerator's archival means the deployment glue (API, hosting, auth) is yours to own and maintain.
- Route, don't replace: production systems that succeed with both run a query classifier that sends global/thematic questions to graph search and everything else to vector retrieval - replacing vector retrieval wholesale re-pays the graph premium on queries it cannot improve.