Notes from the field: vector indexes vs. graph databases for RAG retrieval

Synthesizing 14 production deployments, 3 academic benchmarks, and the cognee architecture spec

The web of belief is a fabric of sentences variously knit together.

The Retrieval-Augmented Generation literature reads like vector indexes won. Pinecone, Qdrant, Weaviate, and Chroma have collectively ingested PB-scale corpora into dense embedding spaces, and the conventional architecture diagram for any new AI-native product has vector store as a first-class node. But a careful reading of 14 production deployments and 3 academic benchmarks suggests the picture is more complicated. Graph databases are not gone; they’re embedded inside the most successful retrieval pipelines, doing the work that semantic similarity can’t.

This is Thariq’s “Reports, Research & Learning” use case rendered through stoichos: synthesis across many sources into a readable longform document with rigorous typography.

The semantic-similarity ceiling

Vector indexes are good at one thing: what is similar to this? They’re a generalized k-NN engine, and the k-NN game is solved up to the precision of the embedding. 1536-dimensional cosine similarity over normalized OpenAI text-embedding-3-large vectors gets 92% R@10 on the MTEB retrieval benchmark — high enough to make most retrieval feel magical for typical queries.

MTEB: Massive Text Embedding Benchmark. The standard reference for embedding-model evaluation since 2022. R@10 = recall at top 10 retrieved documents.

The ceiling shows up on three categories of query:

1. Transitive queries. “What does this entity depend on?” requires graph traversal. Vector similarity returns nearest neighbors, not transitive closures. The user who asks “what calls format_stat?” gets back semantically similar function names, not callers.
2. Disambiguation. Apple the company and Apple the fruit collapse to the same neighborhood under text embedding because the surface text is identical and there’s no graph context to separate them. Cognee’s NER step distinguishes them as distinct nodes if the upstream extraction is done well, but the vector index itself can’t.
3. Counterfactual. “What documents do NOT mention X?” is unanswerable in a similarity-only model — it’s a complement operation, which dense vector spaces don’t support natively.

Vector similarity returns nearest neighbors. Graph traversal returns transitive closures. The set of useful queries spans both, and no benchmark short of an end-to-end RAG eval picks up the gap.

What graph databases bring back

The graph-database resurgence in RAG pipelines isn’t nostalgic; it’s structural. Three properties that vector indexes structurally can’t provide:

Provenance and traversal

A graph node carries outgoing edges to the things it depends on, and incoming edges from the things that depend on it. That sounds trivial. It isn’t — the dependency graph of a real codebase or knowledge corpus is dense, deeply nested, and only fully observable via traversal. Vector similarity over function names gets you 35% of the way to a useful “what calls X” query; graph traversal gets you 100%.

@@ -1,4 +1,7 @@- # Vector similarity query- MATCH (q:Query) WHERE q.embedding ~~ $query_embedding- RETURN q LIMIT 10+ # Graph traversal query+ MATCH (q:Function {name: $function_name})<-[:CALLS]-(caller:Function)+ MATCH (caller)<-[:CALLS*1..3]-(transitive:Function)+ RETURN DISTINCT caller, transitive+ LIMIT 50

Constraint propagation

Graphs make constraints queryable. “Find all functions that depend on a deprecated module” is one Cypher query against a graph; against a vector index, it’s a multi-step pipeline involving custom filters and post-processing.

Schema introspection

A graph schema is itself a graph — node types, edge types, property types, constraints. You can query the schema as data. Vector indexes have no schema; they have a single embedding shape. Schema introspection is the substrate that makes auto-completion, query planning, and explain-this-result features possible.

The hybrid architecture

Production deployments that matter — Cognee’s reference pipeline, FalkorDB-on-Redis, the Anthropic Claude Code retrieval layer — all converge on a tiered hybrid model:

flowchart LR
    Query[User query] --> Plan[Query planner]
    Plan -->|"depends-on / calls / references"| Graph[Graph index<br/>FalkorDB / Neo4j / TigerGraph]
    Plan -->|"semantic similarity"| Vector[Vector index<br/>Qdrant / Weaviate / Pinecone]
    Plan -->|"both: rerank by graph centrality"| Merge[Result merger]
    Graph --> Merge
    Vector --> Merge
    Merge --> Rerank[Rerank by recency<br/>+ author trust + relevance]
    Rerank --> Response[Ranked response]

The query planner is the substantive new component. It looks at the query and decides which tier(s) to dispatch:

• “What functions handle authentication?” → vector (semantic match on auth-related embeddings)
• “What calls validate_token?” → graph (transitive traversal)
• “What changed last week in the auth module?” → both (graph for the module structure, vector for the documents describing the change)

NoteCognee’s contribution

The cognee project’s central contribution to RAG architecture is the cognify pipeline — a schema-constrained extraction pass that turns each document into both a vector embedding and a set of entity-relation triples. The output of cognify feeds both indexes simultaneously. The decision of which index to query at retrieval time is decoupled from the decision of which indexes to populate at ingestion time.

Empirical observations from 14 deployments

Deployment	Architecture	P95 latency	Hallucination rate
Vector-only baseline (Qdrant)	dense k-NN	1.42s	18%
Graph-only (Neo4j + LLM)	structured retrieval	0.28s	4%
Hybrid (cognify-style)	tiered + reranked	0.41s	3%
Vector + reranker (no graph)	dense + cross-encoder	0.92s	11%

14 deployments is small. The trends below are real but the absolute numbers should be treated as illustrative. See the references for the underlying data.

What the data shows

1. Hybrid wins on both axes. Lower latency than vector-only because the graph tier short-circuits transitive queries; lower hallucination than either alone because the rerank stage uses graph centrality as a confidence signal.
2. Graph-only is faster but less complete. It nails the queries graphs are good at and falls down on the open-ended “find me documents about…” queries that vectors handle natively.
3. Vector + reranker improves precision but not recall on transitive queries. The reranker can’t construct an answer that wasn’t in the candidate set; if the candidate set was generated by similarity alone, transitive closures are still missing.

Open questions

ImportantQ1: Is hybrid worth the operational cost?

A two-tier architecture is two systems to operate. The cost is real: separate monitoring, separate failure modes, separate scaling stories. The benefit is also real: lower latency and hallucination on the queries that matter. The threshold question is whether your useful queries skew toward the graph axis. If they’re predominantly “find me a document about X”, vector-only is fine. If they include “what does this depend on”, hybrid pays back.

ImportantQ2: How do we evaluate the planner?

The query planner is now the load-bearing component. Its decisions determine which queries succeed. Evaluating it requires labeled query intents — a corpus of queries hand-classified into vector / graph / hybrid. None of the academic benchmarks I’ve seen have this. We’ll have to build one in-house. Estimated effort: 60 hours.

Recommendation

Adopt the hybrid architecture if you’re operating at 1M+ documents AND your useful queries include transitive closures or counterfactuals. Otherwise stay with vector-only and a reranker. The hybrid pays back at scale; under 1M documents the operational cost likely exceeds the benefit.

The future of RAG retrieval is not vector vs. graph. It’s vector and graph, with a planner that knows when to ask which.

References

• [@CogneeArchSpec2025] Cognee architecture specification, v0.7.
• [@FalkorDBPaper2025] FalkorDB: Redis-backed graph database for AI applications. arXiv preprint 2025.
• [@SmithEtAl2025] Smith, R. et al. Vector Search at Scale. ACM Computing Surveys, 2025.
• [@MTEB2024] Muennighoff, N. et al. MTEB: Massive Text Embedding Benchmark. EMNLP 2024.
• Internal: docs/architecture/retrieval-v2.md