LLM observability is the practice of making a non-deterministic system legible. Traditional monitoring watches latency and error rates; that is necessary but nowhere near sufficient when the "correctness" of an output is itself fuzzy and context-dependent.
Trace the whole request
The unit of observability is the full trace: the user input, any retrieved context, the resolved prompt, the model output, tool calls, tokens, cost, and latency at each hop. For a [[Retrieval-Augmented Generation]] system that means capturing which chunks were retrieved and their scores — because "the answer was wrong" and "the right document was never retrieved" demand completely different fixes, and only the trace tells them apart.
Quality is a signal, not a status code
You cannot alert on 200 OK when the response is confidently wrong. Practical
signals include groundedness (did the answer use the context), refusal rate,
user feedback, and cheap LLM-as-judge scores sampled over live traffic. These
feed the same rubric used offline in [[Agent Evaluation]], closing the loop
between what you tested and what users actually experience.
The feedback flywheel
Good traces are not just for debugging — they are training data. Surfaced failures become new eval cases; recurring patterns motivate the next round of [[Prompt Engineering]]. Storage-layer metrics like recall drift in your [[Vector Databases]] belong here too, because a silently degrading index is invisible until observability makes it visible.
The honest summary
If you ship an LLM feature without observability, you are flying blind and calling it confidence. Instrument first, and every later decision — model choice, prompt change, index tuning — becomes evidence-driven instead of a guess.