Why Production AI is Mostly Software Engineering

There's a myth that building AI products is mostly about models and prompts. In production, the model is maybe 10% of the work. The other 90% is software engineering — and that's not a complaint, it's the job.

The iceberg

The prompt is the tip everyone sees. Below the waterline:

• Service architecture, autoscaling, and health checks
• Authentication, throttling, retries, and timeouts
• Schema validation and error handling at every boundary
• Retrieval pipelines, caching, and data ingestion
• Evaluation harnesses and release gates
• Observability, dashboards, and alerting

None of that is "AI." All of it decides whether the AI is usable.

An honest request lifecycle

@router.post("/answer")
async def answer(req: Request, user=Depends(authenticate)):
    rate_limit(user)
    payload = validate_request(req)          # boundary check
    ctx = await retrieve(payload.query)       # data engineering
    result = await call_with_fallback(payload, ctx)  # reliability
    checked = validate_output(result)         # guardrail
    log_metrics(user, result)                 # observability
    return checked

Exactly one line of that touches the model. The rest is the engineering that makes that one line safe to expose to real users.

Why this is good news

Software engineering is a mature discipline. We know how to build reliable systems on top of unreliable dependencies — that's most of distributed systems. An LLM is just another unreliable dependency with an unusually good API.

That means the skills that make AI products succeed are largely transferable and learnable: interface design, error handling, observability, testing. The teams that win at AI aren't the ones with the fanciest prompts. They're the ones who treat AI as an engineering problem.

The reframe

Stop asking "which model?" first. Ask: how does this fail, how do I observe it, how do I recover, and how do I know it's still working tomorrow? Answer those, and the model becomes the easy part.