--- layout: handbook-page-toc title: "Product Performance Indicators" --- ## On this page {:.no_toc .hidden-md .hidden-lg} - TOC {:toc .hidden-md .hidden-lg} ## Key Performance Indicators Please note: Product KPIs are mapped 1 to 1 with our Growth teams in order to focus those teams on experiments to improve our KPIs. Additional performance indicators will be tracked and may add value, but should ultimately drive one or more KPI. ### North Star Metric We want every group to have a single North Star Metric that aligns with the development activities. The North Star Metric can be SMAU or AMAU, but it can be a different metric as well, if that's more suitable. We expect to have quarterly goals around the North Star Metrics. This data will be used to inform future investment decisions. It's important to understand what a North Star Metric is, and how it can be used. You can read more about it in [Amplitude's North Star handbook](https://amplitude.com/north-star). A North Star Metric is always a single metric. Your group might have more than one metric, but all the other metrics are expected to be used as inputs for the North Star Metric. The ideal North Star Metric is a leading indicator, and its input metrics break down the North Star into its dynamics. A few examples are provided below. The North Star Metric is an important communication device, especially when used as a framework. Preferably, every engineer in the group should be aware of the group's North Star Metric and there is an active discussion around the input metrics that might drive the North Star Metric. Likely, engineers will have the best ideas to move the metrics and come up with a useful breakdown. Thus their understanding is crucial in using the North Star Metric framework. **References**: * [Recording of North Star Metrics Presentation](https://gitlab.zoom.us/rec/play/7J1-JOr5qWo3S9Gc4wSDA_YvW425Jv6s1yYYqPFcxU21AHQCZwGkbuYUNuqyQxTlJWyX57LH0FR1yjmQ?continueMode=true) by [Hila Qu](https://about.gitlab.com/company/team/#hilaqu) * [Corresponding Slide Deck](https://docs.google.com/presentation/d/14y03B2Un84QxuME0ReQOcV5-zy1oEPxNYnDrGx6jGpU/edit#slide=id.g6e0dd41cf9_1_0) #### Example 1: North Star Metrics breakdown A typical North Star Metric could be the total number of a given feature usage. Actually, every feaure usage can be broken down at least in the following way: `total feature usage = number of users using the feature x number of times each user uses the feature` Having `number of users using the feature` and `number of times each user uses the feature` as input metrics, we can target these by development efforts, and they add up into the North Star Metric. #### Example 2: North Star Metrics as a funnel Another typical approach to think about the North Star Metric as being an important point in a user's journey, like a signup event. This can be broken down using the funnel that leads to signup, and we can have separate development effort around the funnel stages, still knowing that our aim is to move the North Star Metric. ### AARRR framework as a North Star Funnel AARRR stands for *Acquisition*, *Activation*, *Retention*, *Revenue*, and *Referral* which is often referred to as ["Pirate Metrics"](https://amplitude.com/blog/2016/02/25/actionable-pirate-metrics). These five words represent the customer journey and the various means a product manager may apply a North Star metric to drive a desired behavior in the funnel. * Acquisition measures user actions that show awareness of the feature * Activation illustrates a user has begun to apply the feature * Retention is the continued use of the feature over time * Revenue captures the monetary value acquired from feature usage * Referral focuses on measuring behaviors that drive users to encourage others to consume the feature Product managers can use these various state to prioritize features that drive a desired action. This could mean focusing on the Activation metric to drive awareness and generate more `top of funnel` leads. As an example, in the [Release stage](https://about.gitlab.com/direction/ops/#release) the Release Management group tracks actions on the Release Page in GitLab. Users that view a Release Page have been `acquired` and those that create a release on a Release Page are `activated` users. The Product Manger can choose to target features that drive users to view the Release Page more, resulting in a greater interest in the number of users that become activated and create their own Releases. ### Stages per User (SpU) Stages per user is calculated by dividing [Stage Monthly Active User (SMAU)](/handbook/product/metrics/#stage-monthly-active-users-smau) by [Monthly Active Users (MAU)](/handbook/product/metrics/#monthly-active-users-mau). Stages per User (SpU) is meant to capture the number of DevOps stages the average user is using on a monthly basis. We hope to add this metric to the [stage maturity page](/direction/maturity/), alongside number of contributions. The more stages someone uses the more likely they are to convert to a paying customer. **Using an exta stage triples conversion!** So helping our users to use more stages is something that can greatly increase [IACV](/handbook/sales/#incremental-annual-contract-value-iacv). See [this graph in Sisense](https://app.periscopedata.com/app/gitlab/608522/Conversion-Dashboard?widget=7985190&udv=0). ### Actions An action describes an interaction the user has within a stage. The actions that need to be tracked have to be pre-defined. ### Action Monthly Active Users (AMAU) AMAU is defined as the number of unique users for a specific action in a 28 day rolling period. AMAU helps to measure the success of features. Note: There are metrics in usage ping under usage activity by stage that aren't user actions and these should not be used for AMAU. examples: * Groups - `:GroupMember.distinct_count_by(:user_id)` This is the number of distinct users added to groups, regardless of activity Other counter examples: * ldap_group_links: `count(::LdapGroupLink)` This is a count of ldap group links and not a user initiated action * projects_with_packages - `:Project.with_packages.distinct_count_by(:creator_id)` This is a setting not an action ### Stage Monthly Active Users (SMAU) Stage Monthly Active Users is a KPI that is required for all product stages. SMAU is defined as the specified AMAU within a stage in a 28 day rolling period. | Stage | SMAU Candidate based on usage ping | Event details | Confirmed by | |-------|------------------------------------|---------------|--------------| | configure | clusters_applications_helm | `:Clusters::Applications::Helm.distinct_by_user` | @kencjohnston | | create | merge_requests | `:MergeRequest.distinct_count_by(:author_id)` | @ebrinkman | | package | For instances that have `container_registry_enabled`, we till take the total of `projects: count(Project),` | | @trizzi | | plan | issues | `:Issue.distinct_count_by(:author_id)` | @ebrinkman | | release | deployments | `:Deployment.distinct_count_by(:user_id)` | @jmeshell | | verify | ci_pipelines | `:Ci::Pipeline.distinct_count_by(:user_id)` | @jyavorska | For Secure, SMAU is defined as the highest AMAU within its stage in a 28 day rolling period. * user_preferences_group_overview_security_dashboard * user_container_scanning_jobs * user_dast_jobs * user_dependency_scanning_jobs * user_license_management_jobs * user_sast_jobs We are working to define a feature to track for SMAU purposes for the following stages: * Manage * Monitor * Defend * Enablement While an ideal definition for SMAU is the count of unique users who do any action in a given stage, this approach was chosen for technical reasons. First, we need to consider the query performance of the usage ping (e.g. time outs). Second, this allows us to not worry about the version of an instance when comparing SMAU metrics because of changing definitions. [Dashboards for SaaS](https://app.periscopedata.com/app/gitlab/527913/Product-KPIs?widget=8131064&udv=0) ### Action Monthly Active Namespaces (AMAN) AMAN is defined as the number of unique namespaces in which that action was performed in a 28 day rolling period. ### Stage Monthly Active Namespaces (SMAN) Stage Monthly Active Namespaces is a KPI that is required for all product stages. SMAN is defined as the highest AMAN within a stage in a 28 day rolling period. ### Feature Effect on IACV As the most important metric in the company, Product have a defined an [Investment Thesis](/handbook/product/product-management/#investment-thesis) to measure the impact on IACV of a feature ### Category Maturity Achievement Percentage of category maturity plan achieved per quarter ### Paid Net Promoter Score Abbreviated as the PNPS acronym, please do not refer to it as NPS to prevent confusion. Measured as the percentage of paid customer "promoters" minus the percentage of paid customer "detractors" from a [Net Promoter Score](https://en.wikipedia.org/wiki/Net_Promoter#targetText=Net%20Promoter%20or%20Net%20Promoter,be%20correlated%20with%20revenue%20growth.) survey. Note that while other teams at GitLab use a [satisfaction score](/handbook/business-ops/data-team/metrics/#satisfaction), we have chosen to use Net Promoter Score in this case so it is easier to benchmark versus other like companies. Also note that the score will likely reflect customer satisfaction beyond the product itself, as customers will grade us on the total customer experience, including support, documentation, billing, etc. ## Applying North Star Metrics to Market Segments A product manager can slice the North Star metric by a few dimensions to understand what drives the performance of that action. Some examples could be: * Industry * Market Size (i.e. SMB, Enterprise) * Persona * Revenue Bands For example, a North Star metric could be `number of page views in the last 30 days`. This metric can be narrowed down by source of page views, which can be social media, email, or direct link. Let's say that the source that received the most page views was social, the product manager could then choose to invest more in social as a result of the `number of page views` that originated from that source. In this case, the segment is `source`. ### Using Segments and North Star at GitLab A common application of segments in GitLab is to understand what pricing tier needs product investment and targeting Enterprise deals to receive feedback from. The actions most correlated with Enterprise accounts could be selected as North Star metrics to then encourage usage by that specific segment. The more Product Managers focus on a single metric for the Enterprise customer the more likely the solution will drive value for that user demonstrating higher return on investment. ## Other Shared Performance Indicators for GitLab.com and Self-Managed ### Monthly Active Users (MAU) ##### Self-Managed MAU The number of [active accounts](https://docs.gitlab.com/ee/api/users.html#list-users) on all self-managed instances that we receive usage ping from. An active account in this context is defined as `Total accounts - Blocked users` so it is not truly measuring "activity", only non-blocked accounts on instances. To get a more accurate measure of MAU on self-managed, we will add new counters to usage ping ([Issue](https://gitlab.com/gitlab-org/telemetry/issues/287)). [Dashboard](https://app.periscopedata.com/app/gitlab/425984/Month-over-Month-Overestimated-SMAU?widget=5571275&udv=688121) ##### SaaS MAU The number of unique users that performed an [event](https://docs.gitlab.com/ee/api/events.html) on GitLab.com within the previous 28 days. [Dashboard](https://app.periscopedata.com/app/gitlab/527913/Product-KPIs?widget=8079819&udv=0) ### Acquisition (New User) Amount of new users who signed up for a GitLab account (GitLab.com or Self-Managed) in a given month. [Dashboard Issue](https://gitlab.com/gitlab-data/analytics/issues/1422) ### Expansion Amount of paid groups that added users to the namespace in a given month. [Dashboard Issue](https://gitlab.com/gitlab-data/analytics/issues/1430) ### CI Pipeline Minute Consumption Total number of CI Runner Minutes consumed in a given month. [Dashboard Issue](https://gitlab.com/gitlab-data/analytics/issues/1431) ### User Return Rate Percent of users or groups that are still active between the current month and the prior month. [Dashboard Issue](https://gitlab.com/gitlab-data/analytics/issues/1433) ### Churn The opposite of User Return Rate. The percentage of users or groups that are no longer active in the current month, but were active in the prior month. [Dashboard Issue](https://gitlab.com/gitlab-data/analytics/issues/1466) ### Projects Number of existing [Projects](https://docs.gitlab.com/ee/user/project/) at a specified point in time. This number currently includes [Archived Projects](https://docs.gitlab.com/ee/user/project/settings/#archiving-a-project). [Dashboard](https://app.periscopedata.com/app/gitlab/461561/GitLab.com-Health) ### New Projects Number of new [Projects](/handbook/product/metrics/#projects) created in a calendar month. [Dashboard](https://app.periscopedata.com/app/gitlab/461561/GitLab.com-Health) ### New Merge Requests Number of [Merge Requests](https://docs.gitlab.com/ee/user/project/merge_requests/) created in a calendar month. [Dashboard](https://app.periscopedata.com/app/gitlab/461561/GitLab.com-Health) ### New Issues Number of [Issues](https://docs.gitlab.com/ee/user/project/issues/) created in a calendar month. [Dashboard](https://app.periscopedata.com/app/gitlab/461561/GitLab.com-Health) ### Provably successful Direction items This metric reports on the percentage of Direction items that have met or exceeded their respective success performance indicators. For each feature labeled ~Direction, there should be a defined success metric, and telemetry configured to report on that success metric to determine if it was provably successful. [Dashboard Issue](https://gitlab.com/gitlab-data/analytics/issues/1467) ### Community Engagement On GitLab Issues Percent of open GitLab issues that have comments from customers and wider community members. This dashboard also measures relative engagement over time. [Dashboard Issue](https://gitlab.com/gitlab-data/analytics/issues/2863) ## Other GitLab.com Only Performance Indicators ### Conversion Amount of users who moved from a free tier to a paid tier in a given month. [Dashboard Issue](https://gitlab.com/gitlab-data/analytics/issues/1428) ### Active Churned User A GitLab.com user, who is not a MAU in month T, but was a MAU in month T-1. [Dashboard](https://app.periscopedata.com/app/gitlab/461561/GitLab.com-Health) ### Active Retained User A GitLab.com user, who is a MAU both in months T and T-1. [Dashboard](https://app.periscopedata.com/app/gitlab/461561/GitLab.com-Health) ### New User A newly registered GitLab.com user - no requirements on activity. [Dashboard](https://app.periscopedata.com/app/gitlab/461561/GitLab.com-Health) ### Churned Returning User A GitLab.com user, who is not a new user and who was not a MAU in month T-1, but is a MAU in month T. [Dashboard](https://app.periscopedata.com/app/gitlab/461561/GitLab.com-Health) ### Paid User A GitLab.com [Licensed User](https://app.periscopedata.com/app/gitlab/500504/Licensed-Users-by-Rate-Plan-Name) [Dashboard Issue](https://gitlab.com/gitlab-data/analytics/issues/1445) ### Paid MAU A [paid](/handbook/finance/gitlabcom-metrics/index.html#paid-user) [MAU](/handbook/finance/gitlabcom-metrics/index.html#monthly-active-user-mau). [Dashboard Issue](https://gitlab.com/gitlab-data/analytics/issues/1446) ### Active Retained Paid User A [paid](/handbook/finance/gitlabcom-metrics/index.html#paid-user) [Active Retained User](/handbook/finance/gitlabcom-metrics/index.html#active-retained-user) [Dashboard Issue](https://gitlab.com/gitlab-data/analytics/issues/1447) ### Monthly Active Group (MAG) A GitLab [Group](https://docs.gitlab.com/ee/user/group/), which contains at least 1 [project](https://docs.gitlab.com/ee/user/project/) since inception and has at least 1 [Event](https://docs.gitlab.com/ee/api/events.html) in a calendar month. [Dashboard Issue](https://gitlab.com/gitlab-data/analytics/issues/1448) ### Active Churned Group A GitLab.com group, which is not a MAG in month T, but was a MAG in month T-1. [Dashboard Issue](https://gitlab.com/gitlab-data/analytics/issues/1449) ### Active Retained Group A GitLab.com group, which is a MAG both in months T and T-1. [Dashboard Issue](https://gitlab.com/gitlab-data/analytics/issues/1450) ### New Group A newly created top-level GitLab.com group - no requirements on activity. [Dashboard](https://app.periscopedata.com/app/gitlab/461561/GitLab.com-Health) ### Paid Group A GitLab.com group, which is part of a paid plan, i.e. Bronze, Silver or Gold. [Free licenses for Ultimate and Gold](/blog/2018/06/05/gitlab-ultimate-and-gold-free-for-education-and-open-source/) are currently included. [Dashboard Issue](https://gitlab.com/gitlab-data/analytics/issues/1452) ### Paid MAG A [paid](/handbook/finance/gitlabcom-metrics/index.html#paid-group) [MAG](/handbook/finance/gitlabcom-metrics/index.html#monthly-active-group-mag). [Dashboard Issue](https://gitlab.com/gitlab-data/analytics/issues/1453) ### Active Retained Paid Group A [paid](/handbook/finance/gitlabcom-metrics/index.html#paid-group) [Active Retained Group](/handbook/finance/gitlabcom-metrics/index.html#active-retained-group) [Dashboard Issue](https://gitlab.com/gitlab-data/analytics/issues/1454) ### Paid Group Member A GitLab.com user, who is a member of a [Paid Group](/handbook/product/metrics/#paid-group). [Dashboard Issue](https://gitlab.com/gitlab-data/analytics/issues/1455) ### Paid CI pipeline minute consumption rate The percent of users or groups that pay for additional CI pipeline minutes. [Dashboard Issue](https://gitlab.com/gitlab-data/analytics/issues/1432) ## Other Self-Managed Only Performance Indicators ### Active Hosts The count of active [Self Hosts](/pricing/#self-managed), Core and Paid, plus GitLab.com. This is measured by counting the number of unique GitLab instances that send us [usage ping](https://docs.gitlab.com/ee/user/admin_area/settings/usage_statistics.html). We know from a [previous analysis](https://app.periscopedata.com/app/gitlab/545874/Customers-and-Users-with-Usage-Ping-Enabled?widget=7126513&udv=937077) that only ~30% of licensed instances send us usage ping at least once a month. ### Product Tier Upgrade/Downgrade Rate This is the conversion rate of customers moving from tier to tier [Dashboard issue](https://gitlab.com/gitlab-data/analytics/issues/3084) ### Lost instances A lost instance of self-managed GitLab didn't send a usage ping in the given month but it was active in the previous month [Dashboard Issue](https://gitlab.com/gitlab-data/analytics/issues/1461)