Langfuse launches AI Gateway Learn more
November 19, 2026

Evaluating proactive agents

Proactive agents decide on their own whether, when, and how far to act. Learn how to trace that decision, measure it like a classifier, and build a dataset of moments to evaluate it.

Picture Annabell SchäferAnnabell Schäfer

My personal Muse agent, Dr. Doodle, lives in my WhatsApp and slides into my DMs multiple times a day. Dr. Doodle is part of the newest wave of agents that doesn't wait to be asked. Muse from Meta, Grok Bot from SpaceXAI and Instinct watch your inbox, goals, calendar and projects, and step in when they think something needs doing.

WhatsApp message from the Dr Doodle agent reminding the user that a package has arrived for pickup at a Berlin post office, with opening hours and pickup instructions

"Proactive" is the label everyone uses. But what actually makes an agent proactive? And how do you evaluate whether it acts on the right things?

What makes an agent proactive

Proactive agent: an agent with a standing goal that monitors its context for triggers and decides on its own whether, when and how far to act, without being asked each time.

Reactive agent: an agent that acts when triggered, on a task it was given.

Diagram comparing a reactive agent, where a trigger always leads to the agent acting, with a proactive agent, where a trigger and a standing goal feed into worth_acting_on(), which either acts (notify, suggest, ask or act), stays silent, or waits and schedules its own next trigger

Under the hood, both are built from the same parts: instructions, tools, an agent loop, memory and a trigger. The difference is the goal and the decision. Some examples:

CaseProactive?Why
Chatbot with memoryNoNo standing goal. The user decides when it acts.
Nightly job that always runsNoNo decision. The schedule decides.
Webhook that triages every new ticketNoIt acts on every trigger.
Agent that checks your inbox and only pings you about what mattersYesIt has a goal, watches context, and can decide to do nothing.
Agent that rebooks your flight after a delayYesIt assesses the delay, decides it needs solving, then asks or just rebooks depending on your instructions.

The most important difference between both is how they handle triggers. A reactive agent treats the trigger as an instruction, a proactive agent treats it as a situation to assess in light of the larger goal. A nightly job that always runs the same task is automation, not proactivity.

What it looks like in code

The whole distinction fits in two lines:

  • Reactive: trigger → act
  • Proactive: trigger → decide whether, when and how far → act (or not)

In code, it's one extra step before the agent runs:

# trigger: a message, an event, or a scheduled run

# reactive
def on_trigger(trigger):
    run_agent(trigger)

# proactive
def on_trigger(trigger):  # e.g. a new email, or the hourly inbox check
    decision = worth_acting_on(trigger, goal, memory)
    # whether: act or stay silent
    # when:    now or later
    # how far: notify, suggest, ask, or act
    if decision.act:
        run_agent(trigger, decision)

worth_acting_on answers three questions. Say an email arrives: your flight is delayed by two hours.

  1. Whether to act: does the delay break your connection, or is it harmless?
  2. When to act: tell you now, or wait an hour to see if the airline rebooks you? Waiting means the agent schedules its own next trigger.
  3. How far to go: tell you, suggest a new flight, ask to book it, or book it on its own.

All three depend on memory. The agent has to know your plans and preferences to judge what deserves your attention.

Everything interesting about a proactive agent lives inside worth_acting_on. That's the part you need to see and evaluate, and it's what the rest of this post is about.

Trace the decision, not just the action

With a chatbot, every output has a reader: the person who asked. That's free quality control. With a proactive agent, most runs end in "do nothing" or finish in the background, and nobody looks at them.

So the decision itself needs to be in the trace. For every event that reaches the decision step, log:

  • The event: what woke the agent up.
  • The context: what it retrieved from memory and connected apps.
  • The outcome: act now, wait, or stay silent.
  • How far: notify, suggest, ask for approval, or act.
  • The rationale: why it decided this.

If you split worth_acting_on into a cheap filter (rules, embeddings, or a decision model like Jev) and an expensive judgment call, as Navan recommends, trace both steps. Otherwise you can't tell whether a missed alert was dropped by the filter or judged harmless.

Keep the runs that end in "do nothing". A decision not to act produces no output, so it never shows up in a review queue unless you put it there. Navan makes the same point: staying silent is a valid outcome, and often the right one. That only holds if you can check it.

Measure it like a classifier

worth_acting_on takes an event and returns a decision given the overall context. That makes it a classifier, so evaluate it like one.

FailureWhat it looks likeSignals in production
Low precision: too eagerThe agent pings you about things you don't care about. Users mute it.Dismissed suggestions, ignored notifications, denied approvals
Low recall: too cautiousThe agent misses what you actually needed. Nobody notices, because nothing happened.Users doing the task themselves, "why didn't you tell me?"
Wrong depth: went too farThe agent acts where it should have asked.Undone actions, complaints, support tickets

Being too eager or going too far is a core risk, it is Dr. Doodle reminding me 5x about a package I already picked up. But it can also have security consequences. One user reported that Instinct hit a login wall while buying something and reset the user's password instead of asking.

The costs are not symmetric. A missed nudge costs little. A wrong purchase costs money and trust. So don't tune one global threshold. Set a stricter bar for actions that are expensive or hard to undo, and a looser one for a suggestion the user can ignore.

To turn dismissals, ignored notifications and undone actions into scores on your traces, see Capturing user signals.

Build a dataset of moments and run evals

For a chatbot, a dataset item is a question with an expected answer. For a proactive agent, it's a moment with an expected decision.

  • Input: the trigger plus a snapshot of the user's memory and instructions. Without it, you can't judge the decision: the same flight delay means "stay silent" for one user and "rebook" for another.
  • Expected output: act, wait or stay silent, and how far, plus a short note on why.
  • Sources (if based on real failures): flagged traces (user feedback like dismissals, undos, "why didn't you tell me?"), a random sample including runs where nothing happened, and pairs with the same trigger but different users.
  • Experiments: rerun on every change to prompt, model, memory or thresholds, or in CI. Track precision and recall separately, since more helpful often means more annoying. When a run gets it wrong, check whether it's a judgment bug or a memory bug.

It's the same AI engineering loop as for any agent, with decisions as the unit instead of answers.

The decision is the product

The skill, the tools, the loop, the memory and the trigger are parts every agent has. What makes a proactive agent good or bad is one function: whether this moment is worth acting on, when, and how far.

That function runs mostly where nobody is watching. Make it visible, and it becomes something you can measure and improve.

Sources

Further reading

Ready to get started with Langfuse?

Join thousands of teams building better LLM applications with Langfuse's open-source observability platform.

or Talk to an expert

No credit card required, free tier available, self-hosting option


Was this page helpful?

Get started today

Interested in tracing and improving your AI agents? Get started with Langfuse in minutes.