RAG Beyond Vector Databases

"Add a vector database" is where RAG starts, not where it ends. The gap between a hello-world retriever and one that answers multi-hop questions correctly is enormous — and mostly not about embeddings.

Where pure vector search breaks

Cosine similarity finds text that reads like the query. It does not understand that answering the question requires hopping across three documents connected by an entity the query never mentions.

Given a similarity score $s(q, d) = \cos(\theta)$ between query $q$ and document $d$, vector search optimizes for surface semantic overlap — not for the reasoning path an answer requires.

Hybrid retrieval: the reliable default

Combine dense (semantic) and sparse (lexical/BM25) retrieval. Dense catches paraphrase; sparse catches exact terms, codes, and rare entities embeddings smear together.

def hybrid_retrieve(query: str, k: int) -> list[Chunk]:
    dense = vector_store.search(embed(query), k=k * 3)
    sparse = bm25.search(query, k=k * 3)
    fused = reciprocal_rank_fusion(dense, sparse)
    return rerank(query, fused)[:k]

Reciprocal rank fusion is a boringly effective way to merge two ranked lists without tuning a weight you'll regret.

Reranking earns its keep

Retrieve broadly, then rerank precisely. A cross-encoder rescoring the top ~30 candidates consistently beats retrieving a tight top-k directly — you widen the net, then let a stronger model pick.

Graph traversal for multi-hop

When documents are interconnected, a knowledge graph expresses relationships an embedding space cannot. On one project, adding Neo4j graph expansion on top of vector retrieval moved multi-hop QA accuracy from 61% to 84%.

The pattern:

1. Vector search seeds relevant entities.
2. Cypher traversal expands along typed relationships.
3. A validation step constrains context to the traversed subgraph.

The order matters

Vector search for recall, graph traversal for connection, reranking for precision, validation for grounding. Each stage covers the previous stage's blind spot.

A checklist for real RAG

• Hybrid dense + sparse retrieval, not dense alone.
• Retrieve wide, rerank narrow.
• Add graph structure when questions are multi-hop.
• Validate that context actually supports the answer.
• Measure retrieval-miss rate as a first-class metric.

Vector databases are necessary. They are nowhere near sufficient.