Lessons from Shipping LLM Systems

Everything that makes an LLM demo impressive is different from what makes an LLM system reliable. Here's what I learned the expensive way.

Lesson 1: The model is not your bottleneck — variance is

A single model call has a wide latency distribution and a nonzero failure rate. Chain a few of them and the tail explodes. Your p50 is a comforting lie; your users live in the p99.

async def call_with_guardrails(prompt: str) -> str:
    for attempt in range(MAX_RETRIES):
        try:
            return await asyncio.wait_for(
                client.complete(prompt), timeout=TIMEOUT_S
            )
        except (asyncio.TimeoutError, ProviderError):
            if attempt == MAX_RETRIES - 1:
                return await fallback_model.complete(prompt)
            await asyncio.sleep(backoff(attempt))

Timeouts, bounded retries, and a fallback model aren't optional extras — they are the system.

Lesson 2: Route across providers

Depending on a single provider couples your uptime and your cost to theirs. A fallback router across OpenAI, Anthropic, and an open-weight model turned provider incidents from outages into latency blips.

Lesson 3: Validate the boundary, always

Treat model output like untrusted user input. Parse it against a schema at the boundary; never let raw text flow into downstream logic.

The cost of a malformed response should be a caught exception and a repair attempt — not a corrupted database row three services deep.

Lesson 4: You can't improve what you don't measure

Instrument everything: latency per stage, token usage, retrieval-miss rate, schema-failure rate, fallback-trigger rate. When quality drops, these are the difference between a five-minute fix and a five-hour guess.

Lesson 5: Cache the expensive determinism

Embeddings and many retrievals are deterministic. An embedding cache alone cut one service's p50 from 4.8s to 1.9s — no model change required.

The uncomfortable summary

Most of the work in a production LLM system is not prompting. It's the same distributed-systems discipline you'd apply to any unreliable dependency — because that's exactly what a model is.