1:1 mentoring with Big Tech AI engineers
LLM & Agentic

The Eval Stack

Complete guide to LLM evaluation and observability: automated evals, human feedback loops, A/B testing, and monitoring.

Last updated

Staff12 min readFirst readThe Agent Loop

After this section you can

  • Design a four-layer eval stack from a CI gate through golden sets and A/B tests to production monitoring
  • Build a golden set and an LLM judge that survive scrutiny: code graders first, calibrated and bias-checked judging
  • Instrument an agent with OpenTelemetry GenAI spans so every layer reads the same traces
41

The Eval Stack: Offline to Online

A score on a test set is one layer of evidence. This is the whole stack that lets you change a prompt, a tool or a model on a Tuesday and know by Friday whether users are better off.

Key idea

Evaluation is a pipeline of gates. Cheap, fast checks run on every change; slower, truer ones run on live traffic; production failures flow back as new test cases. Traces are the shared record every gate reads.

Four gates between a change and your users, all reading the same traces
LAYER · WHEN IT RUNS WHAT RUNS THE QUESTION IT ANSWERS 1 · CI gate every prompt, tool or model change assertions + a 40-case smoke set code graders, one pinned judge Did this break a known case? 2 · Offline eval before each release the full golden set, k runs per case code graders + calibrated judge Is it better than what ships? 3 · Canary and A/B a live slice of traffic 1–5% canary, then a randomised split one primary metric + guardrails Do real users do better? 4 · Production monitoring always on implicit signals, sampled judge drift checks and alerts Is it still working this week? failures become new cases Traces · every layer reads the same OpenTelemetry GenAI spans Blue layers run before users see the change, green layers run on real traffic. Each one catches what the layer above cannot see.

Related

More in LLM & Agentic

Tokens, Context Windows & Sampling

Preview

What a language model actually does on every call: split text into tokens, predict one next token at a time, fit everything into a fixed context window, and pick each token by sampling — with interactive temperature and context-budget explorers.

How LLMs Are Built

Preview

Complete lifecycle of large language models from pre-training through fine-tuning, RLHF, and deployment — with architecture diagrams and production considerations.

How LLMs Call Tools

Preview

How LLMs use function calling and tool use — the mechanics behind tool-calling agents, from prompt engineering to structured output.

Prompt Engineering Foundations

Preview

Where an instruction belongs — tool description, system prompt, or user turn — why emphasis that worked on older models now causes over-triggering, and the prompt patterns that have expired.

Get full access to all 74+ sections with code examples, diagrams, and interactive animations.

Unlock Premium