Skip to main content
Your AI agent answers real users, and some of those conversations go wrong. To fix the right problems, you need to know which conversations failed, why, whether the failures cost you outcomes, and whether a change helped. This guide takes you from no data to a review you can repeat, using Agent Intelligence and the Mixpanel reports you already know. The examples follow a support agent that answers questions in a help widget. The agent is named support-agent, and the outcome you care about is an Issue Resolved event. By the end of this guide, you can:
  • Send agent traces to your Mixpanel project so conversations are readable
  • Find the conversations that errored, ran slow, or cost too much, and work out what went wrong
  • Connect agent activity to a product outcome in a funnel or retention report
  • Check a fix with an experiment or a before-and-after comparison

​
Who does what

​
Before you start

You need:
  • A Mixpanel project with access to Agent Intelligence. To request access, see Agent Intelligence.
  • An agent that sends traces with OpenTelemetry. See the requirements in Agent Intelligence.
  • An outcome event already tracked in your project, such as Issue Resolved. You use it in Stage 4.
A few terms appear throughout. A span is one piece of work the agent did, such as a call to a language model. A turn is one prompt and the agent’s response. A conversation is a user’s whole exchange, made of one or more turns. Mixpanel stores every span as an $mp_ai_span event, so agent data works in the same reports as the rest of your product data.
Conversations appear from the time you finish setup. To check whether you can add past conversations, see the Agent Intelligence FAQ.

​
Stage 1: Send readable traces to Mixpanel

Role: Engineer To review conversations later, send the right details with your spans.
1

Connect your exporter

Follow Implementation to point your exporter at Mixpanel and set user.id on every span. Set user.id to the same value you use as distinct_id, so agent activity joins the rest of the user’s events.
2

Add the attributes this guide relies on

Follow Add the attributes that make the data useful. This guide uses the conversation ID, agent name, operation name, model and token counts, tool name, message content, and an error status on failed spans.
3

Add an attribute for each change you plan to make

Add a custom attribute for each change you make on purpose, such as prompt.version. In Stage 5, you use it to compare results before and after a change. For naming rules and an example, see Send your own attributes.
4

Decide whether to send message content

If you work in a regulated industry, you can leave out gen_ai.input.messages and gen_ai.output.messages. You still get cost, tokens, latency, errors, models, and tool use. For redaction options, see Data privacy.
You’re ready for Stage 2 when a real conversation with at least two turns appears under Agent Intelligence → Conversations, both turns are grouped into one conversation, and its messages are readable. Let traffic build up so there are enough conversations to review.

​
Stage 2: Find conversations that went wrong

Role: Product manager or analyst To see where the agent struggles, start with the numbers, then read the conversations behind them.
1

Open Usage

Open Agent Intelligence → Usage. Use the filters and date range at the top to narrow the view to your agent, for example gen_ai.agent.name equals support-agent. Every attribute you sent is available as a filter or breakdown.
2

Look for errors, slow turns, and costly turns

These reports show where to look:
  • Agent Turns with Errors (%): the share of turns with at least one failed span
  • Errors by Tool: failed tool calls, split by gen_ai.tool.name
  • Agent Run Duration Over Time: median and 90th percentile turn duration
  • LLM Spend Over Time and LLM Spend per Model: where the cost goes
For how Mixpanel adds up spans into turns and conversations, see Conversation level data.
3

Open the conversations behind the numbers

Open Conversations. The list doesn’t have an error column, so filter to the spans you’re looking for:
  • To find errors, filter AI Status Code ($mp_ai_status_code) equals STATUS_CODE_ERROR. To find one tool’s errors, also filter gen_ai.tool.name.
  • To find slow or costly conversations, sort by Duration, Cost, Tokens, or Turns.
A conversation is listed when one of its spans matches every filter. You can also start from an Insights chart of $mp_ai_span: click a data point and select View Conversations.
4

Save the users who had a bad time

Create a cohort of users who did $mp_ai_span where AI Status Code equals STATUS_CODE_ERROR at least a few times in the last 30 days. Pick the count that fits your traffic. You use this cohort in Stage 4.

​
Stage 3: Decide what went wrong

Role: Product manager and engineer Open a conversation. Mixpanel shows every turn, and every span inside each turn, in the order they ran. If a span failed, it’s marked as an error, so you can read the error and the tool call behind it. If your project uses Session Replay and user.id matches the user’s distinct_id, the conversation also has a Replay tab that plays the user’s session from when the conversation started. A failure can be an errored turn, a turn much slower or costlier than the rest, or a reply that missed the request. You can judge the last one only if you sent message content. Use this table to sort what you see. Write down each failure pattern with one example conversation. For each conversation, record the conversation ID, what happened, your decision, and who owns the next step. A shared spreadsheet is enough.

​
Stage 4: Check whether failures cost you outcomes

Role: Product manager or analyst A failing conversation matters most when it changes what users do next. These reports show that link. To see whether agent turns lead to your outcome, create a Funnels report:
1

Add an agent turn as step one

Select $mp_ai_span, and filter gen_ai.operation.name equals invoke_agent so each turn counts once. Add a filter for gen_ai.agent.name equals support-agent.
2

Add your outcome as step two

Select Issue Resolved.
3

Break the funnel down

To compare versions of one agent, keep the gen_ai.agent.name filter and break down by your custom attribute, such as prompt.version. To compare agents, remove the gen_ai.agent.name filter from step one and break down by gen_ai.agent.name. To compare models, break down by gen_ai.request.model only if your root spans carry it. Many tracers set the model only on model-call spans.
To see whether users who try the agent come back, check Weekly Retention on the Usage page. To compare agent users with users who never use the agent, create a cohort of users who did $mp_ai_span. Then run a Retention report that starts from an event every user does, such as sign-up, and break it down by that cohort. Both groups then share the same starting event, return event, and time window. To see whether bad experiences cost you users, use the cohort from Stage 2. Filter or break down your funnel or retention report by that cohort, and compare it to everyone else.

​
Stage 5: Fix the problem and check the result

Role: Engineer and product manager To know a fix worked, change one thing at a time and record which change is live.
1

Make one change

Change one thing, such as the prompt or the model. Set your custom attribute to a new value, such as a new prompt.version, so spans from the change are easy to separate.
2

Pick how to check the change

  • Run an experiment. Use feature flags and Experiments to test models, prompts, tools, and interface changes. Because agent data is event data, cost and error rate can be experiment metrics like any other.
  • Compare before and after. On the Usage page, break down Agent Turns with Errors (%), Agent Run Duration Over Time, or LLM Spend Over Time by your custom attribute, such as prompt.version. Use the same breakdown on your funnel report. A before-and-after comparison is quicker to set up than an experiment, but it can’t rule out other things that changed over the same period.
3

Watch the experience

Use Session Replay to watch what a slow turn felt like, or to find users who opened the agent and left without sending a prompt.
4

Recheck your failure patterns

Return to Conversations and check that the failure patterns you wrote down are gone or smaller.
Turning a failure into a repeatable test belongs in your own evaluation tooling and is outside this guide. Keep the conversation ID of each failure so you can find it again.

​
Check that it worked

​
Troubleshoot

If your data doesn’t look right, check Troubleshooting implementation in the Agent Intelligence FAQ. It covers four common problems:
  • Nothing is arriving
  • Every turn shows as a separate conversation
  • Prompts and responses are missing
  • Spans arrive but properties are empty
If a span is missing or a message is cut short, check the size limits in Send your own attributes.

​
Next steps

Agent Intelligence

Setup, concepts, and reference details.

Cohorts

Group users by behavior and compare them.

Experiments

Test a change and measure its effect.

Session Replay

Watch how users move through a flow.

Mixpanel MCP server

Connect an AI client to your project.