A Graph-RAG Assistant for Multi-Hop Questions

Vector RAG is superb at "find me the passage about X" and quietly terrible at "how does X relate to Y through Z." Our users kept asking the second kind of question — tracing ownership chains, dependency paths, regulatory lineage — and a flat embedding index simply cannot represent a relationship it never stored. So I built a Graph-RAG assistant that keeps both a knowledge graph and a vector index, and routes each question to the retrieval it actually needs.

Problem

The domain was a web of entities: companies, subsidiaries, standards, and the obligations linking them. Analysts asked multi-hop questions:

• "Which suppliers of Acme are subject to the same disclosure rule as Acme?"
• "Trace the certification chain from this component to the finished product."

A vector store answers these by hoping the right multi-hop path happens to be described verbatim in one chunk. It rarely is. Recall on our multi-hop eval set sat at 46%, and even correct answers came without a traceable path a human could audit.

Approach

The insight is that these questions have two halves. There is a semantic half ("what is a disclosure rule") that embeddings handle well, and a structural half ("which nodes are two hops from Acme along SUPPLIES and SUBJECT_TO") that is a graph traversal, not a similarity search.

So I built both retrievers and a router in front of them:

• A knowledge graph in Neo4j, populated by an extraction pipeline that pulls entities and typed relations from documents.
• A vector index in OpenSearch over the same source chunks.
• A router — a small classifier that inspects the query and decides whether to lead with graph traversal, vector search, or a blend.

I orchestrated the whole flow as a LangGraph state machine so each stage was a node with a typed contract and a replayable trace.

Architecture

from langgraph.graph import StateGraph, END
 
def build_assistant() -> StateGraph:
    g = StateGraph(RagState)
    g.add_node("route", classify_query)          # graph | vector | hybrid
    g.add_node("graph", traverse_knowledge_graph) # Cypher multi-hop
    g.add_node("vector", semantic_search)          # OpenSearch kNN
    g.add_node("fuse", merge_evidence)             # dedup + rank
    g.add_node("answer", generate_grounded)        # cite nodes + chunks
    g.set_entry_point("route")
    g.add_conditional_edges("route", pick_retriever, {
        "graph": "graph",
        "vector": "vector",
        "hybrid": "graph",
    })
    g.add_edge("graph", "fuse")
    g.add_edge("vector", "fuse")
    g.add_edge("fuse", "answer")
    g.add_edge("answer", END)
    return g

For structural questions the graph node compiles the intent into a bounded Cypher traversal:

MATCH path = (a:Company {name: $entity})
             -[:SUPPLIES|SUBJECT_TO*1..3]-(related)
RETURN related, relationships(path) AS edges
LIMIT 50

The traversal returns not just the answer nodes but the edges — the actual path — which becomes the citation. Every claim in the final answer points to a concrete subgraph a human can inspect.

To rank fused evidence when graph and vector both contribute, I score each candidate by a convex combination of its graph proximity and its semantic similarity:

where $d_g(c)$ is the hop distance from the seed entity and $\lambda$ is tuned per query type by the router — high for structural questions, low for definitional ones.

Outcomes

Evaluated on a 300-question benchmark split into single-hop and multi-hop subsets:

Single-hop performance held steady — the router correctly sends those to vector search — while multi-hop accuracy jumped 31 points. The latency cost of a graph traversal was real but bounded, and worth it for questions that were previously unanswerable.

The durable lesson: retrieval architecture should mirror the shape of the question. A knowledge graph is not a nicer vector store; it is a different representation that makes relationships first-class. The moment I stopped forcing every question through one index, both kinds of questions got the retrieval they deserved.