Capturing signals

Once you have traces, you will want to know whether what your agent did was good. You can either set up online evaluators that measure specific aspects of a trace, or you can capture signals of judgment from your users.

Unlike traditional online evaluation, capturing signals does not directly evaluate the quality of the agent's work. Instead, it captures things a user did or said that indicate the user perceives the agent's work as very good or very bad: a thumbs down, a retry, a request for a human, a draft sent unedited.

Why this is important

This is a cheap and scalable way to get quality judgments on production traffic, compared to other methods such as experts annotating traces (human time) or online LLM-as-a-judge evaluators (LLM cost). Your users judge every output anyway, through what they do next, so you just need to instrument your application to catch these signals.

While through traditional evaluation you encode what you believe is important, user signals will also surface behaviors you didn't anticipate but might be worth taking a look at. This makes them especially valuable in staying on top of how users experience your agent's work.

Four kinds of signals

Every signal falls into one of four kinds: explicit ratings, behavioral signals, conversation signals, and outcome signals.

Explicit ratings

Explicit ratings are the only signals where the user knowingly grades the output, which makes them unambiguous, but also rare.

In Meta's production assistant, about 0.1% of model messages receive a positive emoji reaction,¹ and the ratings that do arrive are skewed toward users with strong reactions.

Behavioral signals

Behavioral signals are actions users take on the output: accepting, copying, editing, regenerating, skipping, abandoning. They occur inside the normal workflow, so coverage is high; on a copilot-style product, every suggestion produces one.

Conversation signals

In a conversational product, the user's next message often tells you what they thought of the agent's output: a rephrased question, a correction, a request for a human, or a thank you all say something about the quality of the output they received.

Amazon used rephrase and follow-up patterns to curate training data for Alexa's language understanding from live traffic.² Microsoft's SPUR goes a step further: an LLM learns satisfaction and dissatisfaction patterns from the small slice of thumbs-labeled conversations, condenses them into a rubric, and scores the unlabeled majority with it.³

Outcome signals

Outcome signals tell you what the environment did with the output: a draft was sent unedited, a ticket was closed, etc. They are the strongest kind because they are grounded in the environment rather than in someone's opinion.

How to interpret signals

Once relevant signals are implemented, every trace sits in one of three groups: labeled negative, labeled positive, or unlabeled, with unlabeled being the large majority.

A negative is close to a confirmed bad experience. Users rarely retry or correct an answer they liked. Each trace with a negative user signal is worth opening, and negatives are the natural entry points for error analysis.

A positive signal shows acceptance, but not necessarily quality.⁵ Read positives as a trend, and never make one the optimization target on its own.

Who labels is not random

Often, you won't get a signal from the entire population of users, and there will be biases in labeled/unlabeled data groups. You can split these kinds of biases into:

• Self-selection. People rate what they feel strongly about.
  • When Yahoo! Music surveyed its listeners, 65% said their opinion of a song decides whether they rate it at all, and songs they loved or hated were rated far more often than songs they felt neutral about.⁶
• Survivorship. All your signals are from users who stayed long enough to take the action you're tracking. Keep in mind that it's likely not the full set of users that started an interaction.
• Exposure and interface. Users react only to what the system shows, in the form it is shown.
  • For example, search users will keep clicking top-ranked results disproportionately regardless of whether they are the most relevant for their use case.⁷
• Cohorts. Signal rates vary across users, languages, and time of day without the quality of the output changing.

Which signal should you optimize for?

Prefer signals closest to what the product is for. Otherwise you might start optimizing for behavior you don't want.

Two examples of this:

• YouTube ranked recommendations by clicks and got clickbait; ranking by expected watch time fixed it, because watch time is closer to what the product is for.⁸
• When OpenAI added a reward signal built from ChatGPT thumbs data, the model turned sycophantic: user feedback favored agreeable answers and weakened the signal that had been holding sycophancy in check, while offline evaluations and A/B tests looked fine and only expert testers noticed the model felt off.⁹

What comes next

Once signals land as scores on your traces, put them to work in four places:

• Set up alerts on the signals that mark a bad experience, so that a spike in them reaches you right away.
• Create filtered views of negatively labeled traces, so that you can go through them frequently.
• Add graphs of signal rates to your dashboards to track how they develop over time.
• When reviewing flagged traces, use them: run error analysis on them and add them to datasets.

References

---

1Reinforcement learning from user feedback (Han et al., Meta)2A scalable framework for learning from implicit user feedback (Park et al., Amazon Alexa)3Interpretable user satisfaction estimation for conversational systems with LLMs (Lin et al., Microsoft)4WildFeedback: aligning LLMs with in-situ user interactions and feedback (Shi et al., Microsoft)5Patterns for building LLM-based systems and products (Eugene Yan)6Collaborative filtering and the missing at random assumption (Marlin et al.)7Accurately interpreting clickthrough data as implicit feedback (Joachims et al.)8Deep neural networks for YouTube recommendations (Covington et al.)9Expanding on what we missed with sycophancy (OpenAI)

---

---

---