A Production RAG API on AWS

The mandate was deceptively simple: "let users ask questions about our document corpus and get grounded answers." The hard part was doing it at a p95 under one second, on a budget that survived contact with real traffic, without hallucinated citations. This is how the RAG API came together on AWS.

Problem

The corpus was ~180k documents — contracts, policy PDFs, and support articles — updated daily. The first prototype stuffed the top-20 chunks into a single Bedrock call and returned whatever came back. It demoed well and failed three ways in production:

• Latency. Naive top-k over a cold index plus a 20-chunk prompt pushed p95 past three seconds.
• Cost. Every query paid for thousands of input tokens, most of them irrelevant.
• Trust. Without citation grounding, the model confidently cited documents that did not contain the claim.

The KPIs I committed to: p95 under 1s, faithfulness above 0.90 on a held-out eval set, and a marginal cost under $3 per thousand queries.

Approach

I treated retrieval quality as the lever that moves everything else. Better retrieval means fewer chunks in the prompt, which cuts both latency and cost, and tighter grounding raises faithfulness for free.

Chunking. I moved from fixed 1000-character splits to structure-aware chunking that respects headings and table boundaries, targeting ~512 tokens with a 64-token overlap. Overlap matters: a claim split across a boundary is a claim no retriever can find.

Hybrid retrieval. Pure vector search missed exact identifiers (clause numbers, SKUs). I combined dense vectors with BM25 lexical search in OpenSearch and fused the rankings with Reciprocal Rank Fusion:

with $k = 60$. RRF is unglamorous and hard to beat — it needs no score normalization across retrievers, which is exactly the failure mode that sinks naive weighted sums.

Architecture

The request path is a small FastAPI service in front of Bedrock and OpenSearch, with a semantic cache in the middle.

async def answer(query: str) -> Answer:
    if cached := await semantic_cache.get(query):
        return cached
 
    dense = await embed(query)                       # Bedrock Titan embeddings
    hits = await opensearch.hybrid_search(
        vector=dense, text=query, size=40,
    )
    ranked = reciprocal_rank_fusion(hits)            # fuse dense + BM25
    top = await rerank(query, ranked)[:6]            # cross-encoder rerank
 
    prompt = build_grounded_prompt(query, top)
    resp = await bedrock.invoke(MODEL, prompt, max_tokens=700)
 
    answer = attach_citations(resp, top)
    await semantic_cache.set(query, answer)
    return answer

Three decisions carried the design:

1. Rerank, then truncate. Retrieve 40 candidates, rerank with a cross-encoder, keep the top 6. The reranker is cheap relative to the LLM and lets me send far fewer tokens without losing recall.
2. Semantic caching. A large share of production queries are near-duplicates. I cache on the embedding of the normalized query and serve a hit when cosine similarity exceeds 0.97. This alone removed roughly a third of Bedrock calls.
3. Citation enforcement. The prompt requires the model to cite chunk IDs, and a post-step drops any answer whose citations do not textually support it, falling back to "I don't have enough information."

Indexing runs as a separate nightly pipeline: documents land in S3, a Lambda fan-out embeds new and changed chunks, and OpenSearch is updated with a versioned alias so reads never see a half-built index.

Outcomes

Measured on a 500-question held-out set graded by an LLM judge and spot-checked by hand:

The faithfulness gain came almost entirely from reranking and citation enforcement; the latency and cost gains came from sending fewer, better chunks and from the semantic cache. The lesson I keep relearning: in RAG, retrieval quality is not a subsystem you tune later — it is the product.

What I would do differently next time is invest in the evaluation harness first. For two weeks I optimized on vibes; the moment I had a scored eval set, every change became a decision with evidence behind it instead of an argument.