Online Evaluation
⚠️
This page is a work in progress and will be released in the coming days.
Online evaluation is a way to evaluate the quality of your LLM application in real-time. The core motivation is to:
- Monitor quality, cost, latency, and security issues in real-time
- Learn from production data to improve prompts, tools, and other application components
- Debug issues upon negative evaluation scores or user feedback
How to get started
Capture Traces of real user interactions
Production traces are the source of truth for online evaluation. See Observability section on how to get started with tracing.
Set up online evals
Depending on your use case, you can set up online evals in the following ways:
- User Feedback
- Implicit User Feedback
- Run-time guardrails
- Run-time executionable checks
- Non-grounded LLM-as-a-Judge
Was this page helpful?