Manage NPS themes and insights

Note: NPS insights is currently in open beta.

Pendo helps you to make sense of textual responses to NPS surveys with themes and insights. Use NPS themes and insights to organize and analyze NPS responses and to generate an actionable understanding of how users are experiencing your application, including pain points and what drives satisfaction. With NPS themes, you can:

• Manually manage NPS themes for textual responses.
• Leverage NPS insights to automatically surface key themes based on Natural Language Processing (NLP) models and generate Language Learning Models-based summaries of information that you can present to colleagues and act on.

Prerequisites

Permissions to manage NPS themes are granted to the following roles:

• Default user roles: Admin, Guide Creator, and Guide Content Editor.
• Custom Role: A new custom NPS permission, Triage NPS Themes, which Admins can enable or disable when creating a custom role.

NPS insights uses generative AI models to process your visitors’ content. You must enable the following settings to allow this to happen.

1. Navigate to Settings > Subscription Settings.
2. To enable auto-suggested themes, select Enable NPS insights auto-suggested themes. 
3. To enable theme summaries and theme name enhancement, select Enable NPS insights summaries and theme name enhancement.

For automatic theme generation and response assignment using NLP models, you must have at least one NPS survey generating a minimum of 600 English textual responses. Only English textual responses are supported. Non-English responses are grouped under the Other theme. Summaries using Language Learning Models do not require a NPS survey with a minimum of 600 English textual responses.

Theme assignment

Themes can be assigned manually and, if you’re an NPS insights customer, automatically by Pendo AI, the NLP model used for theme generation and response assignment. For instructions on how to manually assign themes, see Assign themes to NPS responses.

If you’re an NPS insights customer, we highly recommend waiting for Pendo AI to assign the initial theme before manually assigning, reassigning, or removing themes.

Pendo AI runs periodically on new responses. When you manually assign, reassign, or remove themes for a textual response, Pendo AI assigns a theme to a new response within seven days. Once a response has a suggested theme, you can manually add or remove themes to remove inaccurate data or add themes if none were detected.

If you manually assign a theme to new responses before Pendo AI suggests a theme, Pendo AI learns from this and can suggest the theme in the future but doesn’t automatically select the theme. This means you must then manually select the suggested theme. Pendo NPS users can assign as many themes as they want, on top of any themes assigned by Pendo AI. Pendo AI can’t unassign manually assigned themes.

Manually manage NPS themes

NPS themes are manually managed and triaged in the Responses tab of your NPS survey.

1. In the left-side menu, select NPS.
2. In the Overview tab, search for and select a survey, and then open the Responses tab. From here, you can manually manage and triage NPS textual responses themes in the following ways:

• Create NPS themes
• Assign themes to NPS responses
• Remove themes from NPS responses
• Rename NPS themes
• Delete NPS themes
• Bulk-assign and bulk-remove themes

Create a theme

To create a theme:

1. In the Responses tab of your NPS survey, choose + Create NPS Theme either from:
   
   • The NPS Themes picker in the filters section of the page. The default is All NPS Themes
   • The plus (+) icon next to a specific response in the Responses table. This option immediately assigns the theme to that response
2. Enter a name for your new theme. Theme names can include English characters, dashes, and underscores only. Other characters are unsupported and cause error messages to display.
3. Select Create NPS Theme.

Assign a theme

To assign or reassign themes to a textual response:

1. In the Responses table of your NPS survey, select the plus (+) icon next to the response.
2. Choose one or more themes from the dropdown menu that opens.

If you're an NPS insights customer, Pendo AI suggests themes for the selected response after learning the responses. Suggested themes are indicated with the Wand icon. If there is no suggested theme, Finding the best theme displays. Pendo AI takes up to seven days to learn the responses and suggest a theme. When a new suggested theme is detected, it changes automatically to the relevant theme.

You can assign or remove themes for a response from the existing themes. These themes already exist in the system from the machine learning model. Select x to remove a theme from a response, or open the drop-down list and clear the selected theme.

In the NPS Themes filter, filter by No Theme to see responses that still don’t have a theme assigned. This may be because the response is new and Pendo AI is still learning the new responses or because the assigned themes have been removed.

 

Remove themes from NPS responses

To remove themes from textual responses, you can either:

• Select the cross (x) icon next to the theme you want to remove for a response in the Responses table for your NPS survey.
• Select the plus (+) icon next to the response in the Responses table for your NPS survey and then deselect the theme or themes in the dropdown menu that opens.

As you remove themes to responses, the Promoter and Detractor widgets in the Overview tab automatically populate with the relevant information and themes.

Rename a theme

To rename a theme:

1. In the Responses tab of your NPS survey, find the theme from either:
   • The NPS Themes picker in the filters section of the page.
   • The dropdown menu that appears when you select the plus (+) icon next to a specific response in the Responses table.
2. Hover over the theme name and select the Edit icon that appears.
3. Enter a new name and then select Save NPS theme.

Delete a theme

To delete a theme:

1. In the Responses table of your NPS survey, find the theme from either:
   • The NPS Themes picker in the filters section of the page.
   • The dropdown menu that appears when you select the plus (+) icon next to a specific response.
2. Hover over the theme name and select the trash icon that appears.
3. Confirm that you want to delete the theme and select Delete NPS Theme.

When a theme is deleted, the responses assigned to it are now assigned to No Theme.

Assign or remove themes in bulk

You can assign or remove themes from selected responses as a bulk action. These actions are limited to 20 responses at a time.

1. In the Responses table, select the checkboxes to the left of the chosen responses.
2. Select Assign NPS Themes or Remove NPS Themes as appropriate.
   
   
3. In the window that opens, select themes from the dropdown list
4. Select Save changes.

Handle duplicate responses

Duplicate NPS responses can be sent to Pendo for multiple reasons:

• Multiple tabs open to multiple pages on your app (the NPS pops up on each tab until it is dismissed. It'll persist on the pages it's open on, but won't pop up again on the next page load).
• The App is stuck in a loop, meaning the page keeps refreshing
• Loss of internet connectivity on the Visitor's side.

To exclude duplicate responses from your analytics:

1. In the Responses table, select the More icon in the upper right-hand corner.
   
   The Excluded column displays. This allows you to mark duplicate or incorrect NPS responses to be excluded from the NPS calculation.
2. After making your selections, select Save in the upper right-hand corner.
3. In the Responses Updated window, select Refresh. 
   
   
   The NPS score is updated in all charts, widgets, and dashboards.

View NPS themes

Regardless of whether your themes are manually created or automatically generated by Pendo AI, they appear in the Overview tab of an NPS survey’s details page. In Pendo, go to Behavior > NPS from the left-side menu and then search for and select an NPS survey.

The Overview tab displays the survey dashboard with the Promoter NPS Themes and Detractor NPS Themes widgets. These show:

• All themes created from textual responses of promoters and detractors.
• The number of textual responses per theme.
• The percentage of responses out of total responses, not including the Unassigned theme.

 

There might be themes where there are only detractors and no promoters or only promoters and no detractors.

Use the dashboard filters to change the information displayed in the widgets.

Within these widgets you can:

• Select a theme to see its responses and distribution chart for each response group in a slide-out panel on the right side of the page.
• Select the download icon in a widget to download an image of it, or select the dashboard icon to add the widget to your main dashboard.
• Select Create Segment to create a segment from the theme.

View responses

The table in the Responses tab shows information about the textual responses to your NPS survey. It shows a maximum of 10,000 responses. From this table, you can:

• See the Visitor ID, their response, and the assigned theme. Responses older than two years appear in the table without a theme.
• Resize columns by dragging them left or right.
• Assign, reassign, or remove themes for a response. For instructions, see Manually manage NPS themes.
• Filter the table by:
  • NPS Themes. For instructions, see View responses by theme.
  • Ratings per response group.
  • Textual or blank (non-textual) responses.
• Perform other actions:
  • Download the table as a comma-separated values (CSV) file for further analysis.
  • Manage responses.

View responses by theme

There are two ways to view responses by theme:

• From the Overview tab
• From the Responses tab

Overview tab

To access and review responses assigned to a theme in the Overview tab, select a theme from one of the widgets. This opens a panel on the right side of the page, which shows all the responses assigned to that theme. From here, you can:

• Rename the theme by hovering over the name and selecting the Edit icon.
• Use the Sort by and Filter by options to sort and filter the responses in the theme.
• View the NPS score given for each response.
• Select the theme name in a response to reassign or unassign the theme.
• Select the NPS Theme Distribution tab to see responses for each response group, Promoter, Passive, Detractors.
• Go to the Responses table to view responses from there, which are automatically filtered by theme. Select Open Responses Tab in the top-left of the panel.

Each time you make a change, the number count for the themes assigned to that response updates according to the change you make. The change takes place when you close the side panel and reselect the theme.

Responses tab

The table in the Responses tab of your NPS survey shows all responses by default. To access and review responses assigned to a specific theme, use the NPS themes picker in the filters at the top of the page. This updates the Responses table to include only the responses that relate to the themes that you selected.

If you have AI enabled for NPS insights, any change you make to a response associated with a theme helps Pendo AI learn and improve future theme assignments.

Leverage NPS Insights

NPS insights uses our proprietary machine learning algorithm, Pendo AI, to automatically generate key themes based on NPS responses. NPS insights also creates summaries for themes and gives you a visualization of responses from promoters, detractors, and passive contributors.

Insights for NPS themes are for each survey and themes are generated from responses from the previous two years. Pendo AI runs periodically on the responses, so it can take up to one week for a new response to be associated with a theme after it first shows in the NPS dashboard.

To be eligible for theme generation, a survey must have at least 600 textual responses. If multiple surveys exist, you must select the required survey on the NPS page. Each theme has a name created for it automatically. NPS Insights uses Google AI to automatically enhance theme names for newly-generated themes. 

How it works

Pendo AI uses a natural language processing model. The model finds the top themes that best represent the main subject of a textual response and tries to associate each response to a theme. The algorithm groups responses into the most significant themes. The number of extracted themes is proportional to the number of textual responses in the survey.

Each response goes through a data-cleaning process, which takes out stop words, special characters, and words that don’t exist in English. Themes go through a merging process to ensure that there are no duplicate themes. For instance, the themes discoverability, discovers, and discovering might be merged into a single theme called discover.

Themes are created based on their business relevance. For example, if your product is based around brewing the best coffee in the world, the theme 'coffee' wouldn’t be relevant because most responses would be centered around coffee. Thus, the theme 'coffee' in this example is considered noise.

The algorithm isn’t always able to remove meaningless themes from your data. We try our best by looking at the frequency of the word in the English language. If the word is common, it’s likely not a meaningful theme for your data, and we filter it out. We also group responses like “I love your product” into the Likes/Dislikes theme.

If the machine learning model can’t associate a response to a theme, these responses are grouped under the Other theme. This might be because the response is meaningful, but there aren’t enough related responses to group together and surface as a significant theme emerging from the NPS responses.

In addition, NPS Insights uses Google Generative AI to:

• Automatically generate theme and survey summaries. Theme summaries are generated when there's at least one active NPS survey.  

• Automatically enhance theme names for newly-generated themes.

Functionality

With NPS insights, you can:

• Look at themes emerging from NPS textual responses. Each theme has a name created for it automatically.
• Save the time of manually reviewing each response.
• Gain in-depth insights from NPS responses as new themes surface.
• Track how themes are trending over time.
• View automatically generated theme summaries.
• See how themes are distributed between promoters, passives, and detractors.
• Create segments from automatically generated themes.

Best practices for triaging themes when leveraging NPS insights

Pendo AI learns most effectively when given themes that are:

• Consistent. The NPS responses in each theme revolve around the same topic.
• Well-separated. Each topic has a meaning far different from the other themes.
• Rich in data. The more responses in a theme, the better.

In practice, however, it is often hard to obtain themes that maintain all these properties perfectly:

• A single NPS response can discuss several different topics, for example, both system lags and customer support).
• Two different themes can be related in meaning, for example, costs and service renewals.
• Some themes may contain only a few responses but still be important, for example, responses about a recently added feature.

These conditions vary by the survey size, response contents, and business-specific topics. So, while triaging your NPS responses, it's recommended to maintain consistency, keep topics well separated and aim for themes rich in data, while also using the following rules of thumb:

1. Create a pool of themes that are well-suited to your business needs and, preferably, diverse in meaning.
2. Several dozens of themes usually strike a good balance between the ease of using Pendo’s NPS tool and Pendo AI's ability to learn your preferences.
3. In general, avoid themes with less than 25 responses unless their topic is important to follow.
4. If a response clearly has several main themes, assign it to each of them.

See Manually manage NPS themes for information on triaging themes. NPS insights uses Pendo AI to make this both an automatic and manual process.

Theme summaries

If you’re an NPS insights customer, you can further explore and present the main themes emerging from NPS textual responses. 

For information on enabling this feature, see Prerequisites.

Theme summaries are generated automatically at the theme level and survey level. 

Theme level summary

In the Overview tab of your NPS survey, choose a theme from the Promotor NPS Themes widget or the Detractor NPS Themes widget. This opens a panel on the right side of the screen. From here, you can view an automatically generated theme summary, the textual responses associated with the theme, and each theme’s percentage by responder group (promoter, passive, and detractor).

You can also see the number of responses manually assigned by users and the number of responses automatically assigned by Pendo AI at the bottom of the panel.

A minimum of five responses for each theme is required to generate a theme summary. 

NPS survey summary

You can subscribe yourself and other Pendo users to receive a summary of an NPS survey on a monthly basis.

A minimum of five responses in one month is required to generate the monthly summary.

1. In your NPS survey dashboard, select Manage subscription.
2. In the Subscribe to survey modal, type in the name or email address of other users to subscribe them to the survey.

Subscribed users receive the previous month's report immediately, and then monthly reports are sent during the first week of each month. You must subscribe to each NPS survey you want to receive a summary for, and you're sent a separate email for each one.

View AI activity messaging

Open the AI activity log tab to see how Pendo AI has been learning from your actions to assign and reassign themes to responses.

Notifications of activity give you detailed updates about the changes made by Pendo AI.

You can manually reassign a response to a different theme by selecting a theme from the dropdown list in the relevant response. After you manually reassign the response to a different theme, it no longer displays in the AI Activity Log.

Create a segment from an NPS theme

You can create a segment from a theme to group users based on feedback topics to identify common product behavior of detractors or announce a new feature. Use the segments to determine product usage or target in-app guides based on common feedback. When you create a segment, the Create Segment page opens with all fields pre-populated with information from the theme you selected.

Important: Themes are updated each time the machine learning model runs. This means that because segments are live, the theme clustering may change. If a theme changes significantly, this may affect the segment. It's expected that approximately 10 percent of the responses may have their theme changed when the model is updated.

You can change all filters as required, specifically, the filters relating to NPS Themes:

• NPS Themes or NPS Polls:
  • If you select NPS Themes, the Themes filter is available to select a theme
  • If you select NPS Polls, the Themes filter disappears
• Theme. Select a different theme than the one you selected in the NPS dashboard
• Response groups. the default is All Response Groups. You can select Promoters, Detractors, or Passive

If required, you can combine an “And” rule with different attributes. See the example below:

Create the segment by selecting +Create Segment. This adds your segment to the Segments list. You can also create a segment for an NPS Theme directly from the Segments page.

Important: As you create a rule for your segment, if you select a survey that doesn't meet the requirements for NPS Themes, then the NPS Themes option is disabled as it is irrelevant.