Custom success metrics in Agent Analytics let you define what a successful conversation looks like for an AI agent and track how often your agent achieves it. They go beyond usage volume to measure whether your agent is meeting the outcome it was built for.
Use cases
Use custom success metrics to:
- Measure whether your agent completes the job it was designed for, such as resolving a billing question or finishing onboarding.
- Compare success rates over time to see whether changes to your agent improve outcomes.
- Track progress toward a target by setting a goal on a metric.
How custom success metrics work
When you create a custom success metric, you describe what a successful conversation looks like in plain language. Pendo generates a set of yes-or-no rules from your description, which you review and edit before saving. Pendo then evaluates each conversation against those rules and counts it as successful only when it meets every rule.
Because a conversation must meet every rule to pass, adding more rules makes a metric stricter. If you add a rule to an existing metric, expect the pass rate to stay the same or go down, not up.
After a conversation ends, Pendo records whether each rule passed or failed, along with an overall confidence score for the evaluation. Use the confidence score to find borderline cases and refine your rules over time.
Each custom success metric belongs to a single agent and appears alongside the built-in metrics in the agent's Adoption section.
Note: Evaluation runs after a conversation ends rather than in real time, so results can take some time to appear.
Before you begin
Any user who can access Agent Analytics can create, edit, and delete custom success metrics.
Custom success metrics are available for both prompt agents and conversation agents.
Create a custom success metric
Create a custom success metric from your agent's Adoption section. Creation has two steps: define the metric, then review the rules Pendo generates.
- Go to Product > Agent Analytics, then select an AI agent or open a saved report.
- In the Adoption section of the Overview tab, select Create metric.
- Enter a Metric name. The name must be different from any default or existing custom success metric.
-
Under What to track, answer How can you tell a conversation succeeded? Describe an outcome you could confirm by reading the conversation. A detailed description generates more accurate detection rules. After you save, this description appears in a tooltip next to the metric name.
Note: In the coming weeks, you'll also be able to link a Page, Feature, or Track Event to a metric as the success criteria. - Select Next to generate the rules.
- On the Review rules step, review the yes-or-no rules Pendo generated from your description. Edit, remove, or add rules so they match what you want to measure. Write each rule so it can be answered with yes or no.
- Select Create metric to save.
The metric is added to your agent and appears in the Adoption section.
Tip: Be specific about what counts as success. A description like "The visitor confirmed their issue was resolved" generates better rules than "Improves the experience." Because every rule must pass, keep your set of rules focused on what truly defines success rather than adding every possible condition.
Read your results
Custom success metric results appear at two levels in the Adoption section:
-
In the metric tile, you see the number of conversations that passed, the pass rate for your selected filters and date range, and the change compared to the previous period of the same length. Because a conversation passes only when it meets every rule, read the pass rate as "conversations that met all of my success criteria," not "conversations that were partly successful."
- In the trending chart, select View details at the top of the chart or select a data point to see the conversations counted as successful, as well as the related visitors and accounts. Each conversation contains a summary of why it was considered successful. If you review conversations that pass or fail in a way you didn't expect, consider refining your rules.
Track a metric against a goal
Set a goal to track a metric's performance against a target over time.
- After the metric finishes processing, select it to open the trending chart.
- In the top-right corner of the chart, select Create goal.
- Choose a cadence, such as Weekly.
- Enter a Goal value, such as the number of conversations to reach in that period.
- To check your goal matches the metric's intent, review the name and success description at the bottom of the window.
- Select Save.
After you set a goal, the metric tile indicates that there's a Goal set at the bottom of the tile. The metric's trend chart shows the metric value over time, the goal, a baseline reference, and a summary of how many recent periods met or exceeded the goal. You can edit or remove a goal from the metric's trend view at any time.
Manage your metrics
Manage which metrics appear, keep them up to date, and remove the ones you no longer need. You can't edit or delete the default metrics, but you can hide them from the view.
You manage all of these from your AI agent's details in the Adoption section of the Overview tab.
Change which metrics appear
Use the column manager icon in the top-right corner of the Adoption section to choose which metrics to show and in what order.
Your visibility and order settings are saved for your account and persist across sessions.
- Reorder metrics by dragging them into the order you want.
- Remove a metric from the view without deleting it, and add it back with + Add metric at the bottom of the window. You can also remove it from view by selecting the options menu in the metric tile, then choosing Hide.
Edit a metric
Update a custom success metric's name, success description, or rules as your agent evolves.
Important: Editing a metric's rules changes how past and future prompts and conversations are scored against it.
- Select the options menu in the top-right corner of the metric tile, then choose Edit.
- Update the name, success description, or rules.
- Select Save.
Delete a metric
- Select the options menu in the top-right corner of the metric tile, then choose Delete.
- Confirm the deletion when prompted.
Deleting a custom success metric is permanent and also removes any goal data for that metric. You can add it as a new metric at any time.