Safety · AI System Designstaff

Eval & Guardrail Platform

Components

Drag onto the canvas, or click to add.Tap to add to the canvas.

Utilities

Text Box

Group / Container

Circle

Diamond

Divider Line

Heading

Recommended

Client / User

LLM / Model

Database / State

Guardrail / Classifier

Eval / Judge

Observability / Tracing

Human-in-the-Loop

Entry & gateway

Client / User

LLM Gateway

API Gateway

Orchestration

Orchestrator / Planner

Agent / Worker

Agent Loop

Models

LLM / Model

Model Router

Embedding Model

STT / TTS

Retrieval / RAG

Retriever

Reranker

Vector Store

Knowledge Source

Tools & execution

Tool / Function

MCP Server

Code Sandbox

Browser / Computer Env

External API

Memory & state

Working Memory

Long-Term Memory

Cache (semantic/prompt)

Safety & eval

Guardrail / Classifier

PII Filter / Redactor

Eval / Judge

Observability / Tracing

Infra & data

Database / State

Queue / Event Bus

Human

Human-in-the-Loop

Drag or click a component on the leftTap Components to add your first piece

connect · iterate · submit for review

// brief

Eval & Guardrail Platform

Design a platform to evaluate LLM/agent quality and enforce safety guardrails in production.

Key Requirements

01Offline eval: versioned datasets, graders, regression gating in CI
02Online eval: production sampling, human feedback, A/B
03Defense-in-depth guardrails (input + output) with a latency budget
04LLM-judge calibration against humans (judges can be gamed)
05A data flywheel: prod failures → new eval cases → fixes

Rubric must-haves

0/3

Has an eval/judge component
Has input/output guardrails
Has observability/tracing

These checks are the structure score, 30% of your total. They update as you draw.

Practise it like the real round

60-minute timer paced through requirements, design and deep dive. The follow-up questions stay hidden until the last phase, and solution hints are off.

AI Review

0/5

Review me as:

Build your design, then submit for an AI-powered review with dimension scores, strengths, gaps, and actionable suggestions.

Try designing your solution first before viewing hints.

Comments (0)

Loading comments...