---
layout: handbook-page-toc
title: "Engineering Metrics"
---

{:.no_toc .hidden-md .hidden-lg}

- TOC
{:toc .hidden-md .hidden-lg}

There are many ways to develop engineering metrics. Ideally you need a representation of different aspects of the
operational data such as throughput, quality and cycle time to name a few.

The goal of gathering this data is to define what a healthy and successful team looks like and to foster a healthy dialogue amongst the engineering team and with stakeholders.
These metrics are helpful for conversations around planning, addressing technical debt, capacity to work on bugs,
identifying bottlenecks and improving engineering efficiency. Metrics are a good way to also see how change affects the team’s execution.

Engineering metrics are project or team based and are not intended to track an individual's capacity or performance.
It is important to note that if metrics are used punitively, these goals are hampered and the team psychological safety could be at risk.

As a manager you should ensure you are on top of your team's metrics, driving the right behavior, and understanding any fluctuations. Expect to be able to discuss your team's metrics with your manager on a weekly basis.

To access our current dashboard, please visit the [GitLab Insight Quality Page](https://quality-dashboard.gitlap.com/groups/gitlab-org).
We are currently working to implement these graphs into the GitLab product.

We collect all related Sisense dashboards in its own [`Engineering: Development Metrics`](https://app.periscopedata.com/app/gitlab/topic/Engineering:-Development-Metrics/ab40964d2500481090ae655bfabbc2c1) topic in Periscope to have them all in one place.

## List of Metrics

- [Throughput](/handbook/engineering/management/throughput/): This is our measure for number of MRs merged in a defined time frame (weekly and monthly).
  The current calculations includes merges in all branches and not just `master`.
  - [Engineering throughput](https://quality-dashboard.gitlap.com/groups/gitlab-org/throughputs):
    This chart shows the throughput at the `gitlab-org` group level.
  - [Group throughputs (using Plan:Portfolio Management as an example)](https://quality-dashboard.gitlap.com/groups/gitlab-org/sections/group::portfolio_management):
    The throughput graph is also available for each group.
  - Data for throughput is available at the weekly and monthly level. We show 12 weeks for weekly and 24 months for monthly views.
- [MR Rate](https://quality-dashboard.gitlap.com/groups/gitlab-org/average_mrs_per_month):
  This is our measure calculated by the total number of MRs merged in a given month divided by the total unique authors who created those MRs.
  We currently get all the unique authors from that period and not just filtered to engineering.
  - If a team is contributing to non-GitLab open-source projects when working on product management priorities, and the team spends greater than or equal to one FTE for one month on these activities, they should note this in their public group discussion, and they should also consider tracking this in their overall MR rate.
- [Bugs by Priority (monthly)](https://quality-dashboard.gitlap.com/groups/gitlab-org/bugs_by_priority): This chart shows the monthly bugs by priority over time.
- [Bugs by Severity (monthly)](https://quality-dashboard.gitlap.com/groups/gitlab-org/bugs_by_severity): This chart shows the monthly bugs by severity over time.
- [Bug Classification](https://quality-dashboard.gitlap.com/groups/gitlab-org/bug_classification): This chart shows the overall of priority and severity of current open bugs.
- [Regressions per Milestone](https://quality-dashboard.gitlap.com/groups/gitlab-org/regressions): This chart shows the number of regressions found in a release.
  - **Known issues** Data here is dependent on the `regression:x.y` label. It is possible that some regressions are not captured due to it not having a regression label.
- [Weekly Pulse Survey](/handbook/engineering/management/pulse-survey/): We are working to roll out a weekly pulse survey across all of the engineering teams.
  The goal is to provide a weekly set of data of how the individuals on the team feel in regards to 3 main areas:
  Their work, their manager, and GitLab.
  In addition to the numeric responses, a comment field is available.
  This survey is anonymous and the comments in the results are only visible by the director administering the survey.

## Data included in calculation

### Engineering wide projects

We are currently working on including all of engineering's projects. The following projects included can be seen in this [list](https://quality-dashboard.gitlap.com/groups/gitlab-org/projects).

### Security fixes on dev instance

We also capture data from our `https://dev.gitlab.org/` instance which currently includes the projects below.

- gitlabhq (CE)
- gitlab-ee
- gitaly
- gitlab-shell
- gitlab-pages
- GitLab runner

The data from dev instance is only reflected in a few group level charts, see list of known issues below.

### Known issues

- Due to the lack of team labels on `https://dev.gitlab.org/`, only the 3 group-level charts below are reflected with data from this source.
  - [Engineering throughput](https://quality-dashboard.gitlap.com/groups/gitlab-org/throughputs)
  - [Average MRs merged per engineer per Month](https://quality-dashboard.gitlap.com/groups/gitlab-org/average_mrs_per_month)
  - [Average MRs merged per engineer per Milestone](https://quality-dashboard.gitlap.com/groups/gitlab-org/average_mrs_per_milestone)
- Insufficient labelled data will be listed under `undefined` category.
- A small amount of Merge Requests are not visualized in the charts due to missing `merged_at` attribute.
  - Bug filed: [API: `merged_at` fields missing for certain Merge Requests in `merged` state](https://gitlab.com/gitlab-org/gitlab/issues/26911)

### Adding more metrics

If you or your team needs to include a new project in the metrics or add a new team, please create an issue in the [GitLab Insights issue tracker](https://gitlab.com/gitlab-org/gitlab-insights/issues).

We have a list of projects that are still not included in this [issue](https://gitlab.com/gitlab-org/gitlab-insights/issues/56#projects-not-introduced-yet).

## Universal team metrics dashboard in Sisense

Considering how one of our [development department KPIs](/handbook/engineering/development/performance-indicators/) is to have a goal of <%= performance_indicator_target('MR Rate') %>, we have a team metrics dashboard that can easily be used to see MR statistics per team.

1. Open the [dashboard](https://app.periscopedata.com/app/gitlab/570334/Universal-Engineering-Team-Metrics-Dashboard)
1. You can select in the `team_id` filter (under the headline) for which team you want to see the statistics.

## Universal section metrics dashboard in Sisense

Considering how one of our [development department KPIs](/handbook/engineering/development/performance-indicators/) is to have a goal of <%= performance_indicator_target('MR Rate') %>, we have a section metrics dashboard that can easily be used to see MR statistics per section.

1. Open the [dashboard](https://app.periscopedata.com/app/gitlab/584348/Universal-Engineering-Section-Metrics-Dashboard)
1. You can select in the `section_id` filter (under the headline) for which section you want to see the statistics.

> Note: These dashboards will be available internally to everyone at GitLab (as per default Sisense permissions). Considering that the majority of engineering MR activities are public, we do not anticipate this being an issue. In addition, anyone that can view the dashboard will also be able to toggle the filter to see individual team member's metrics. We purposefully do not chart all team members in a specific chart against each other because throughput is just one data point and is not a holistic evaluation of a team member's productivity/performance.

> Note: MRs that are merged in dev.gitlab.org are not available in Sisense: [gitlab-data/analytics#2789](https://gitlab.com/gitlab-data/analytics/issues/2789)

### Why doesn't a team member show up?

Currently we update the team member records in Sisense with a manually-executed script. This script maps members from team.yml to teams in Sisense by the `departments` list for each member. The script expects exact string matches, so if a team member is not appearing in Sisense please check their section or team is not missing or misspelled in team.yml. If you make an update to team.yml, please let a director know to rerun the script.

#### Red column in Employee Full Name Match
We currently map team members between team.yml and Bamboo through their full name. But they can be different due to various reasons so we have as a pragmatic solution for now a [Google sheet](https://docs.google.com/spreadsheets/d/17ypF9Vw6DEF_MGJUSEDg1t2WQm7miOrYQMrMpVebR8E/edit#gid=0) to map GitLab ID's to Fullnames as in Bamboo so that we can track when people were members of a specific team. Please add missing team members to the sheet with the first and last name from fields in bamboo and then let a director know to rerun the script.

### Sisense Utility Snippets

We created many SQL snippets to centralize counting and the actual structure of data for best reuse.

#### Example usage

For example paste `[development_team_mrsperengineer_by_members('manage_be')]` into a chart and it will give you all MR/engineer metrics for the team `manage_be`.

#### SQL Snippets

`[development_team_mrsperengineer_by_members(team_name)]`

Gives you all metrics directly for a chart around MR/engineer metrics for one specific team.

`[development_section_mrsperengineer_by_members(section_name)]`

Gives you all metrics directly for a chart around MR/engineer metrics for one specific section.

`[development_team_mrtotals_by_members(team_name)]`

Gives you all data for a MR total graph based on team members.

`[development_section_mrtotals_by_members(section_name)]`

Gives you all data for a MR total graph based on section members.

`[development_team_metrics_by_members(team_name)]`

Gives you all relevant queries for MR metrics based on team members. (dev_product_mrs_merged, dev_team_members, dev_counts, dev_monthly_analysis)

`[development_section_metrics_by_members(section_name)]`

Gives you all relevant queries for MR metrics based on section members. (dev_product_mrs_merged, dev_team_members, dev_counts, dev_monthly_analysis)

`[development_team_meantimetomerge_members(team_name)]`

Returns all Mean time to Merge metrics for a team.

`[development_team_members_by_team(team_name)]`

Returns all Development Team Members by team name.

`[development_team_members_by_section(section_name)]`

Returns all Development Team Members by section.


`[development_team_member_definitions]`

Currently half-automated select statement that defines all team members based on data from team.yml and GitLab API to gather GitLab user ID's.

The script is located in the [manager-tools repository](https://gitlab.com/gitlab-org/frontend/frontend-managers/manager-tools). The plan is to automate it ASAP through automatic importing, currently half manual as it is easier to iterate.

### Sisense project indicators

There are two different indicators in Sisense for Merge Request data;
`is_included_in_engineering_metrics` and `is_part_of_product`.

- `is_included_in_engineering_metrics` is a **deprecated legacy indicator** that was intended to align reporting project population in the [Quality Dashboard][quality-dashboard].
- `is_part_of_product` is the indicator that identifies projects which are included in the GitLab release. This metric is used to identify the projects for the [Average MRs/Development Engineers/month][mr-kpi].

Use `is_part_of_product` to identify MRs to include for team dashboards to ensure alignment with
department KPIs.

[quality-dashboard]: https://quality-dashboard.gitlap.com/groups/gitlab-org
[mr-kpi]: /handbook/engineering/development/performance-indicators/#average-mrs-development-engineers-month