Skip to content
All writing
Engineering#Production AI#MLOps#Infrastructure#Backend

Building AI Products That Scale

Scaling AI products is about cost curves, latency budgets, caching, and evaluation velocity — not just throwing a bigger model at the problem.

November 8, 20252 min read

Scaling an AI product isn't scaling a web app with a model bolted on. The cost and latency characteristics are different enough that naive scaling quietly bankrupts you or times out under load. Here's how I think about it.

Three curves that decide everything

Every AI product balances three curves: cost, latency, and accuracy. You don't get to maximize all three. Scaling well means choosing where each request should sit on those curves — per request, not globally.

The most important scaling decision is often model selection per request, not infrastructure. A cheap model on the 80% easy path and a strong model on the 20% hard path beats one expensive model everywhere.

Cache the deterministic parts

Large fractions of an AI workload are deterministic and cacheable: embeddings, common retrievals, and repeated queries.

async def embed_cached(text: str) -> Vector:
    key = f"emb:{sha256(text)}"
    if hit := await cache.get(key):
        return hit
    vec = await model.embed(text)
    await cache.set(key, vec, ttl=DAYS_30)
    return vec

Caching embeddings across 120K+ document chunks is what took one service's p50 latency from ~4.8s to ~1.9s. No bigger model, no more GPUs — just not doing the same work twice.

Budget latency like money

Give each stage a latency budget and enforce it with timeouts. Retrieval gets Xms, generation gets Yms, reranking gets Zms. When a stage blows its budget, degrade rather than block.

Async everywhere

Model and retrieval calls are I/O-bound. Async concurrency lets a single service handle far more in-flight requests without more hardware. Fan out independent retrievals; don't await them in series.

Scale evaluation, not just serving

Here's the counterintuitive one: your bottleneck to shipping faster is often evaluation velocity. If you can't quickly tell whether a change is better, you ship slowly and cautiously.

Invest in a fast, automated eval harness. It's the flywheel that lets you make aggressive cost/accuracy trade-offs with confidence instead of fear.

The scaling checklist

  • Route models per request against cost/latency/accuracy.
  • Cache every deterministic computation.
  • Give each stage a latency budget and a degradation path.
  • Go async for all I/O-bound work.
  • Make evaluation fast enough to run on every change.

Scaling AI is an optimization problem across three curves — solved with engineering discipline, not a bigger model.

Share
← Back to writing

Related reading