Product Engagement Score (PES) Overview

Overview

Product Engagement Score (PES) is a single number that represents the combined impact of feature adoption, return user stickiness, and new user growth. PES is a quantitative value that summarizes your existing usage data without requiring any additional data collection or user participation. Similar to NPS, which is a subjective measurement of user sentiment collected with a survey, PES is an objective measurement of user behavior. Track PES over time and compare PES for different Segments to discover who has the most engagement with your product and why. PES provides a benchmark of your product engagement which can be used to measure the impact product decisions have on your users' behavior.

PES uses Core Events, Visitor, and Account data to calculate the percentile score for each component and combines them into a single score from 0 - 100. Your end users don't need to do anything other than use your product to contribute to the score. Product usage data in Pendo is used to calculate each component of the PES.

Requirements

• Pendo Admin users can configure PES
• Any Pendo user can view PES
• Core Events are required for the Adoption component of PES

Using PES

Product Engagement Score (PES) is a new metric that Pendo introduced to the SaaS business landscape. PES provides a singular, quantitative, business metric to measure the overall engagement of a product with the goal of quickly diagnosing how a product is performing. Historically, product leaders have had to rely on qualitative metrics like NPS to get a feel for customer sentiment. NPS uses a simple recommendation score to determine the virality of a product or service. It has always depended on polling customers and has never provided an accurate look at actual user behavior. While important, NPS cannot be the only business metric for a product. PES is intended to fill that quantitative gap and partner with NPS, by truly examining user behavior and quantifying how users are engaging with a product.

PES is a high-level way to assess the health and success of a product through quantitative engagement data, as a counterpart to the sentiment-driven and highly subjective NPS. It is applicable as a business-level KPI for a product and, when analyzed further with Segments and date filters, can be used to gain a more granular understanding of the engagement of user groups over the life of your product. PES partners with other Pendo tools, like Core Events, Goals, and NPS, to provide a comprehensive understanding of user behavior. These insights can then be used to drive product development, in-app messaging, and outreach campaigns with the ultimate goal of increasing product engagement and user satisfaction indicated by improved metrics, or as we like to say, delivering great products that users love.

PES is visible to all Pendo users on the PES page in the Behavior section. Filters for segment, date range, account, and app are available for all users to analyze PES.

Tip: Use a Custom Date Range filter to measure PES quarterly and report your app's PES with your NPS.

Understanding the Data

The final PES is the average of each component score. Possible scores for all components range from 0 - 100. Any percentages above 100 are adjusted to a score of 100 and percentages below 0 are adjusted to a score of 0.

The PES chart shows changes in the overall PES score and each component compared to the previous time period. Trending PES shows the score change with trend direction indicated by the arrow and the score for the previous time period in parenthesis.

Adoption

Adoption tells you if your users are using your product effectively as indicated by the usage of your Core Events over time. It is calculated as the average number of Core Events adopted across all active Visitors or Accounts. A Visitor or Account with any event recorded by Pendo is considered active during the time period.

Core Event selection may significantly impact the Adoption score. If all Core Events are weighted towards specific groups of users or focus on features with limited use, like first-time setup, it may lower adoption. If Core Events include site navigation or features that a user must engage with, like login, it may increase adoption. A distributed selection of Core Events that includes high-value features for all user groups will result in a balanced Adoption score. There is nothing wrong with Core Event selection deliberately focusing on certain user groups and feature sets for the core of your user base. Just understand how that will shift the Adoption score and account for it in your analysis. Consider calculating adoption at the Account level if Core Events aren't applicable to all users.

Adoption Drilldown

Click on the Adoption component to open the Adoption drilldown. Adoption drilldown shows the filters for your current PES score and the definition for the Adoption component as configured for the application. Drilldown analytics include trending Adoption for the date range and a breakdown of adoption for each core event.

The Trending Adoption chart shows the score as it would have been reported on the date at the bottom of the chart for the number of days in the selected date range. Hover over a point on the chart to see the score and date range used to plot that point. The score is plotted in increments of one day, one week, or one month depending on the size of the date range.

The Core Event Adoption chart shows how many active visitors or accounts have used each core event. The total number is displayed with the percentage of active visitors or accounts who have used the core event. This breakdown allows you to see the distribution of usage across your core events and find if a single core event is carrying your Adoption score. Click on a core event to open the Details page for that tagged feature or track event and see additional usage data.

Stickiness

Stickiness looks at the average percentage of users or accounts who are high-frequency return users in Daily/Weekly/Monthly active user metrics. The Stickiness component finds the ratio of average daily or weekly active visitors or accounts to average weekly or monthly active visitors or accounts in the selected time period. A Visitor or Account with any event recorded by Pendo is considered active during the time period. The DAU/WAU/MAU values use the same calculation and data as the Stickiness Dashboard widget.

Active user metrics measure unique users who used your product during a time period. A user who uses the product today is a daily active user and also a weekly active user. A user who used the product this week but not today is a weekly active user. In this example, if we look at DAU/WAU for today, it's 50%. There is one unique active user today and two unique active users this week. This calculation can be processed for every day and averaged to produce the average DAU/WAU for a larger time period. If users are engaging with a product and finding value, they will return frequently and the ratio of short term active users to long term active users shortens since weekly active users are frequently also daily active users producing a DAU/WAU closer to 100%. The time periods used are up to you based on how frequently you expect Visitors to use your product.

Note: The Stickiness Dashboard widget calculates DAU/WAU/MAU for all apps by default. PES calculates Stickiness for a single app. Make sure the app configured for the Stickiness widget matches PES when comparing stickiness metrics.

Stickiness Drilldown

Click on the Stickiness component to open the Stickiness drilldown. Stickiness drilldown shows the filters for your current PES score and the definition for the Stickiness component as configured for the application. Drilldown analytics include trending Stickiness and a breakdown of DAU, WAU, and MAU for the date range.

The Trending Stickiness chart shows stickiness over time with the average for the entire date range. Hover over a point on the chart to see the raw data used to calculate the percentage for every day or week on the chart depending on the PES configuration. This data matches the visualization in the Stickiness Dashboard widget.

The MAU, WAU, DAU breakdown chart shows the number of daily, weekly, monthly users or accounts at each interval during the date range. This shows you the proportion of each user group to the others and how those proportions influence the Stickiness score. If two groups have a similar trend line, that combination would result in a higher Stickiness score and provides insight into the frequency patterns of your users. Hover over a point on the chart to see the raw data for each user group on that day.

Growth

Growth uses a specially configured version of the quick ratio to measure positive or negative net growth of visitors or accounts. The calculation then scales the resulting ratio to a score between 0 and 100 to convey growth during a specified time period. The quick ratio is the ratio of the sum of new and recovered Accounts or Visitors divided by dropped Accounts or Visitors. 

To compute net growth, the components of the quick ratio are defined as follows: 

• New - Visitor or Account's first ever event is in the current period
• Recovered - Visitor or Account wasn’t present last period but is present this current period
• Dropped - Visitor or Account was present last period but is not present in the current period

When it's greater than one, the ratio indicates positive net growth. A ratio of less than one indicates negative net growth month-over-month. A Visitor or Account with any event recorded by Pendo is considered active during the time period.

Quick ratios range from 0 to infinitely high, but to effectively scale quick ratios to a score between 0 and 100, we set a reasonable range of quick ratios after analyzing quick ratios across Pendo's user base. We observed our customers' quick ratios rarely dropped below 0.5 (twice as much churn as new or resurrected users) and rarely exceeded 5 (new and resurrected users 5x more than churn). The median quick ratio is 1.5. Using minimum, maximum, and median quick ratios, we developed a scaling factor to convert the quick ratio to a growth score between 0 and 100. 

If the quick ratio is less than the median quick ratio, the scaling factor is:

If the quick ratio is more than the median quick ratio, the scaling factor is:

Below is a reference table of how the quick ratio would convert to a growth score using this scaling factor and our observed quick ratio range. Quick ratios below 0.5 convert to a growth score of 0 while quick ratios at 5 or above are capped at a growth score of 100. 

Growth Drilldown

Click on the Growth component to open the Growth drilldown. Growth drilldown shows the filters for your current PES score and the definition for the Growth component. Drilldown analytics include an overview of the quick ratio components during the selected time period and a breakdown of accounts by active visitor growth.

The Active Accounts chart shows the number and composition of new, recovered, and dropped accounts. The chart includes the account composition for the last four cycles based on selected time period.

The Accounts by Active Visitor Growth chart shows the number of active visitors and percent change for the top 10 or bottom 10 accounts. This data is a breakdown of the Active Accounts chart and shows which accounts contributed to positive and negative changes in your growth rate. The chart can be sorted by the Visitors or % Change columns. The % Change column indicates the percent change from the previous time period and the number of active visitors in the previous time period. Click on an account ID to go to the Account Details page and see additional data .

Trending PES Chart

The Trending PES chart shows your PES scores over the four preceding time periods. The time period is set in the filters at the top of the PES page. The PES score is shown at each interval and a line graph shows the trend of each score over time.

Hover over any of the time periods to see a summary of the values for PES and each component score.

Click on the score at the bottom of the chart to toggle visibility for that line. This can help focus the visualization to certain scores if you're sharing the chart in a presentation. The hover score summary only shows the visible scores.

Manage PES

PES is managed from the PES page in the Behavior section. PES can be configured individually for each app and filtered by Segment, time period, and app. The configured PES for an app is visible to everyone in the subscription. Filters are applied to a user's unique PES view. Only Pendo Admin users can configure PES. Anyone can apply filters on the PES page.

Core Events are required to calculate the Adoption score. Access to Core Events setup is available on the PES page for Pendo Admin users.

Configure PES

1. Navigate to PES in the Behavior section.

2. Click Manage PES in the top right to open the Configure Product Engagement Score overlay.

3. Select App on a multi-app subscription.

4. Change dropdown selections for Adoption, Stickiness, and Growth. A preview of the 30 day PES updates as changes are made.

5. Select a different app and make additional changes if necessary.

6. Save & Exit to apply all changes to PES for the subscription. Save & Exit is not available if no changes are made to PES configuration.

Adoption

Adoption measures the proportion of users engaging with Core Events during the time period. Core Events are required to calculate an Adoption score for an app.

Adoption can be configured to measure usage for

• Individual Visitors or anyone in an Account
• Any Core Event or every Core Event

Optimal selections depend on your product, usage patterns, and Core Events selections. A SaaS app may expect individual Visitors to access the app and interact with one or two Core Events while a CRM platform may expect every Core Event to be used across an Account as individual contributors perform their tasks.

Stickiness

Stickiness measures how frequently users return to the app as a ratio of short term active users to long term active users. Select the pairing that most closely represents your ideal usage. Stickiness can be calculated using active visitors or accounts.

Possible Combinations

Measuring by active users is used when each individual user's activity is important to the success of the business. Your app could be targeted to single users with personal subscriptions or to businesses using a per-seat pricing model where each user should be finding value to justify the cost. Maybe cost isn't a concern and you just want your app to be a valuable tool that all users return to frequently. In any of these scenarios, measuring stickiness by user could be the right choice.

• Daily Active Users/Weekly Active Users - Active user base. When users use the app they use it frequently but maybe not habitually. Typically produces a moderate stickiness percentage as unique daily visitors are compared with a smaller sample of unique weekly visitors.
• Daily Active Users/Monthly Active Users - Highly active users. Users are expected to use the app habitually every day. Typically produces a lower stickiness percentage as unique daily visitors are compared with all unique visitors for the month.
• Weekly Active Users/Monthly Active Users - Involved user base, when users use the app they do infrequent tasks or maintenance but aren't expected to use it daily. Typically produces a high stickiness percentage due to the relatively small number of weeks in a month and the longer time window available for a user to qualify as a WAU.

Measuring by active accounts is used when individual users aren't expected to return to the app frequently but someone in the account should. This may be expected for system monitoring tools, financial apps, or platforms that use automatic alerts to prompt user activity. Users don't have daily tasks in the app but return regularly for maintenance, completing workflows on a schedule, or responding to issues.

• Daily Active Accounts/Weekly Active Accounts - Active customers. The app is part of a frequent workflow, likely belonging to a designated team, that's performed more than once a week. Typically produces a moderate Stickiness percentage as unique daily accounts are compared with a smaller sample of unique weekly accounts.
• Daily Active Accounts/Monthly Active Accounts - Highly active customers. The app is used daily by someone at the account, maybe to complete an important daily workflow or access information. Typically produces a lower stickiness percentage as unique daily accounts are compared with all unique accounts for the month.
• Weekly Active Accounts/Monthly Active Accounts - Habitual customers. Someone at the account is expected to access your app on a weekly basis, possibly to complete a transaction or run a report. Typically produces a high stickiness score due to the relatively small number of weeks in a month and the longer time window available for an account to qualify as a weekly active account.

Exclude Weekends removes the data collected on weekends from the overall average calculation. Business apps with most usage between Monday and Friday will likely see improved metrics after removing days with no expected usage from calculations. This generally improves stickiness for DAU ratios as weekend days with no daily usage are removed and no longer pull down average usage on week days. Ratios with WAU may decrease slightly as users who only use the app on weekends are removed.

Note: Weekly/weekly is not an available combination. It's the same number and would produce a Stickiness score of 100 every time. That's cheating. Make your users want to come back for more! Earn that stickiness.

Growth

Growth compares the difference in the total number of active Visitors or Accounts between time periods normalized as a one year growth rate.

• Active Visitor growth is likely the preferred choice if individual Visitors using the app regularly make up the majority of the user base.
• Active Account growth may be preferred if some Visitors use the app infrequently or are only involved briefly, for example for demos or implementation, but Account health and retention is not impacted by the churn of these personas. Measuring Growth by Account will prevent expected user inactivity from negatively impacting Growth.

Frequently Asked Questions

Do Track Events count towards any of my PES metrics?

Track Events can count towards the Adoption score when Track Events are setup as Core Events. By default, Track Events are not counted as activity for the Stickiness and Growth scores. Some customers want Track Events to count as activity in the application. The Count Track Events Received for a Visitor as Time in the Application setting in Subscription settings can be enabled by a Pendo Admin user and Track Events will be counted as activity when calculating the Stickiness and Growth scores.