---
layout: handbook-page-toc
title: "Release:Progressive Delivery Group"
---

## On this page
{:.no_toc .hidden-md .hidden-lg}

- TOC
{:toc .hidden-md .hidden-lg}

## Progressive Delivery Group

The Progressive Delivery Group works on the parts of GitLab within the
[Release Stage](/product/categories/#release-stage), a component of our greater [CI/CD product](/direction/ops/).

## Vision

For an understanding of what this team is going to be working on take a look at [the product
vision](/direction/release/).

## Mission

The Release:Progressive Delivery is focused on all the functionality with respect to
Continuous Delivery and Release Automation.

This team maps to [Release](/handbook/product/categories/#release) devops stage.

## Product Categories

We are responsbile for [maturation](/direction/maturity/) of several [continuous delivery product categories](/direction/ops/#progressive-delivery).

## Team Members

The following people are permanent members of the Release Team:

<%= direct_team(manager_role: 'Engineering Manager, Release:Progressive Delivery') %>
<%= direct_team(manager_role: 'Frontend Engineering Manager, Release (CD)', role_regexp: /Progressive/) %>

## Stable Counterparts

The following members of other functional teams are our stable counterparts:

<%= stable_counterparts(role_regexp: /[,&] Release/, direct_manager_role: 'Engineering Manager, Release:Progressive Delivery', other_manager_roles: ['Frontend Engineering Manager, Release (CD)']) %>

## Technologies

Like most GitLab backend teams, we spend a lot of time working in Rails on the main [GitLab CE app](https://gitlab.com/gitlab-org/gitlab-ce), but we also do a lot of work in Go which is used heavily in [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages). Familiarity with Docker and Kubernetes is also useful on our team.

## Common Links

 * [Issue Tracker](https://gitlab.com/groups/gitlab-org/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=devops%3A%3Arelease)
 * [Slack Channel](https://gitlab.slack.com/archives/s_release)
 * [Roadmap](/direction/release/)

## Planning and Process

Our planning and build process is detailed on our [planning page](./planning.html).

## How to work with us

### On issues

Issues that contribute to the release stage of the devops toolchain have the `~"devops::release"` and `~"group::Progressive Delivery"` labels.

### In Slack

The team's primary Slack channel is `#g_progressive-delivery`, and the public managers channel is `#release-managers`. We also support a number of feature channels for discussons or questions about a specific [feature area](/handbook/product/categories/#release-stage). We no longer support issue specific channels, as they are easy to lose track of and fragment the discussion. Supported channels are:

* Continuous delivery: `f_continuous_delivery`
* Incremental rollout: `f_incremental_rollout`
* Feature flags: `f_feature_flags`

### Team Workflow and Issue Boards

We use the [Progressively Delivery Planning Board](https://gitlab.com/groups/gitlab-org/-/boards/1489550?&label_name[]=group%3A%3Aprogressive%20delivery) to organize and plan upcoming features. This board details the various stages an issue can exist in as it is being readied for development. It's not necessary that an issue go through all of these stages, and it is allowed for issues to move back and forward through the workflow as they are iterated on.

The `~cicd::active` and `~workflow::scheduling` labels are used to feed the next board where EM and PM connect on scheduling upcoming issues.

We use the [Progressive Delivery Scheduling Board](https://gitlab.com/groups/gitlab-org/-/boards/1489554?label_name[]=group%3A%3Aprogressive%20delivery&label_name[]=workflow%3A%3Ascheduling) as a checkpoint to ensure that we have a well-formed issues prior to being ready for development.

Placing the `~workflow::ready for development` will automatically remove the `~workflow::scheduling` label. The following Build Board contains columns for each of the workflow labels beginning with the `Ready for Development`. The intention here is that this is the only place that developers should update issues with the relevant workflow status.

We use the [Progressive Delivery Build board](https://gitlab.com/groups/gitlab-org/-/boards/1489558?label_name[]=cicd%3A%3Aactive&label_name[]=group%3A%3Aprogressive%20delivery) to track features as they make their way from idea to production. Here, issues move more or less linearly from Ready for Development thru to Verification, but on occassion they have to return to Ready for Development as priorities shift.

Issues if over a weight of 1, will have multiple MRs. Each of these associated MRs will likely be in a different stage of development. It's important to update the issues on the build board with the lastest, most relevant state. Example, if an issue has three MRs with one in review, another in development, and one ready for development, then the we should place that issue is in `In Dev` as this is where the most activity for this issue is happening - it's the most relevant signal to the rest of the team. Because we have a one-to-many relationship between our issues and MRs, activity on the board will appear slow. Async daily standups and weekly sync meetings outlined below help with any ambiguity that this system introduces.

#### Workflow Stages

Below is a description of each stage, its meaning, and any requirements for that stage.

* `workflow::start`
  * The entry point for workflow scoped labels. From here, Issues will either move through the validation workflow, go directly to planning and scheduling, or in certain cases, go directly to the development steps.
* `workflow::problem validation`
  * This stage aims to produce a [clear and shared understanding of the customer problem](/handbook/product-development-flow/#validation-phase-2-problem-validation).
* `workflow::solution validation`
  * This output of this stage is a [clear prototype of the solution to be created](/handbook/product-development-flow/#validation-phase-3-solution-validation).
* `workflow::ready for development`
  * This stage indicates the issue is ready for engineering to begin their work.
  * Issues in this stage must have a `UX Ready` label, as well as either `frontend`, `backend` or both labels to indicate which areas will need focus.
* `workflow::in dev`
  * This stage indicates that the issue is actively being worked on by one or more developers.
* `workflow::in review`
  * This stage indicates that the issue is undergoing code review by the development team and/or undergoing design review by the UX team.
* `workflow::verification`
  * This stage indicates that everything has been merged and the issue is waiting for verification after a deploy.

### Engineering Evaluation & Estimation

In order to make sure sprints are effective as possible, it is important to ensure issues are clearly defined and broken down before the start of a sprint. In order to accomplish this, engineering evaluation will take place for issues that require it (small changes and bug fixes typically won’t require this analysis). Engineers will look for assigned issues with the `workflow::planning breakdown` label that have user stories defined, as well as functional, performance, documentation, and security acceptance criteria. See [Build phase](https://about.gitlab.com/handbook/product-development-flow/#build-phase-1-plan) of the Product Development Workflow.

Assigned engineers will work with Product, UX and other engineers to determine the implementation plan for the feature.

Once an implementation plan has been finalized, the following steps should be taken:

* The issue description should be updated to include the details of the implementation plan along with a checklist to show the planned breakdown of merge requests.
* The issue should be moved forward in the workflow to `workflow::ready for development`.
* The weight of the issue should be updated to reflect the number of merge requests estimated as part of the implementation plan. For smaller changes and bugs, the weight should be 1.
* Those updates to the issue will signal that the issue has been properly scoped and is ready to be worked on in an upcoming milestone.

#### Merge request count as a measure of issue weight

As a byproduct of the engineering evaluation process, a rough estimate of the number of merge requests required to develop a feature will be produced. This measure can be used as a way to determine issue weights. These weights can be useful during the planning process to get a rough gauge of how many features can be worked on in a milestone, and to understand how much work the team can do in a milestone. This metric also aligns with the [throughput metric](https://about.gitlab.com/handbook/engineering/management/throughput/) currently measured by engineering teams.

#### Tips: When you're having hard time to estimate MR count, consider PoC step.

If you have no idea where to begin the development due to lack of knowledge on the context or
you're less confident on the current technical proposal,
it might be better to work on Proof-of-Concept MR (PoC or Prototype) at first.
This gives you more insights on the current implementation details and potential required changes.

This step removes a lot of "if"s from your assumptions. When you're making a
technical proposal, often you don't have enough time to investigate the actual
implementation details. In such case, you're assuming like "Probably, this feature works
like ..." or "Probably, this feature and that feature are connected like ...", which
leaves some probability in your development plan, and if it turned out a wrong assumption, you
may need to rethink the whole approach, even though you're at the middle of development cycle.
PoC step derisks such a turnaround by cross-checking the technical proposal with
domain/performance/security experts and UX/PM.

Here is how to work on PoC step:

- Create a new MR with a title started from "PoC:". Explain the technical proposal
  in the MR description, that you think it's the best.
- Try to implement the feature in the merge request. Frontend and bakcend engineers
  can work on the same branch.
- If it takes too long time to make the feature actually works on your local development machine,
  your technical proposal might need to be reconsidered. You can seek the other viable approaches
  by hearing from domain experts or the other developers.
- If you see technical difficulties that seems impossible to be solved in the same milestone,
  you should raise a concern that there is a blocker thus we might
  not be able to finish the feature by the due date.
- If the change size is too large (e.g. you had to modify over 1000-1500 lines without tests),
  you should raise a concern that the issue is too large to ship within one milestone,
  and you can make a suggestion that what can be done in the release and what can be done
  in the next milestone (i.e. issue split).
- In the PoC review, you should ask domain/performance/security experts or the other engineers who are familer on the context
  to review the technical approach and actual code. This is to ensure that there are no big red flags on the proposed solution.
- In the PoC review, you should ask PM and UX to review the feature's behavior.
  This is to ensure that the feature is correctly implemented and orthogonal to our problem to solve.
- You don't need to split MRs. It can be done later.
- You don't need to write neither tests nor documentation. It can be done later.
- You don't need to fix failed pipelines. You can fix the broken tests later. (Although, it might have some useful information)
- You don't need to write well-organized code. Refactoring can be done later.
- The PoC MR is not for merge, however, you can copy and paste the changes into the actual MRs, and poish later.
- Definition of done of PoC step is collecting approvals from PM, UX, domain/performance/security experts.
- Once it's done, you can estimate the number of MRs. For example, if the change size is 1000,
  you can split it to 5 MRs with 200 LoC. As you add tests in the actual development,
  the finial change size per MR would be doubled.
- Once it's done, you can close the PoC MR.

Technically, if you've done a PoC step, there is no need to ask additional reviews
in actual MRs as long as you don't change the feature behavior. You can simply
focus on general engineering review or documentation review, only.
For example, improving code quality, refactoring code, writing tests, writing documents, etc.

Here are the examples of PoC step. [Example 1](https://gitlab.com/gitlab-org/gitlab/merge_requests/16276),
[Example 2](https://gitlab.com/gitlab-org/gitlab/merge_requests/18115)

#### Issue template for a feature development

This is an issue template for feature development. It includes some
important engineering tasks to be done in general.

<details>
<summary>Click me to collapse/fold the Issue template for a feature development.</summary>
<pre>
```
<!-- The issue begins with "Problme to solve", "Intended users", "Proposal", etc.
     Read more https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/issue_templates/Feature%20proposal.md -->

## Technical proposal

<!-- In this section, describe technical proposal for the problem to solve.
     It should give enough contexts to be able to be reviewed by domain/performance/security experts. -->

## Feature Flag

This feature is implemented behind `feature-flag-name` feature flag and disabled by default.
Once we've confirmed the feature is deemed stable, we remove the feature flag in order to publish the feature as GA.
<!-- Read more [Feature flags in development of GitLab](https://docs.gitlab.com/ee/development/feature_flags/) -->

## Planned MRs

### Backend

- [ ] [MR-1 title](MR link if it already exists)
- [ ] [MR-2 title](MR link if it already exists)

### Frontend

- [ ] [MR-1 title](MR link if it already exists)
- [ ] [MR-2 title](MR link if it already exists)

### General

- [ ] [Write a feature spec to test frontend and backend change altogether](MR link if it already exists)
- [ ] [Remove the feature flag and update documentation](MR link if it already exists) # i.e. publish the feature
```
</pre>
</details>

#### Issue template for feature evaluation (dogfooding, beta-tester program)

This is an issue template for feature evaluation. It includes some
important engineering tasks to be done in general.

When you enable a feature via feature flag and expect a significant impact on
user workflow or production load, you should create an issue with the following
template to communicate better with the affected users or SRE.

<details>
<summary>Click me to collapse/fold the Issue template for feature evaluation</summary>
<pre>
```
## Summary

<!-- The reason of feature evaluation. -->

## Feature details

<!-- Describes the feature details and expected performance or usability impact -->

## Timeline

<!-- This section describes the timeline of the evaluation.
     Example:
     - 2019-10-12 01:00 UTC The second evaluation: We will enable the feature on yyy project for X days.
     - 2019-10-11 01:00 UTC We've disabled the feature on xxx project because ...
     - 2019-10-10 01:00 UTC The first evaluation: We've enabled the feature on yyy project for X days.
-->

## How to enable the feature

<!-- Describes how to enable the feature that anyone can execute -->

## How to disable the feature

<!-- Describes how to disable the feature that anyone can execute.
     Consider, in an emergency case, someone else might need to disable the feature instead of you. -->

## Beginning of evaluation

- [ ] Announce in Slack/CompanyCall (T-minus 1 day)
- [ ] Enable the feature

## End of evaluation

- [ ] Announce in Slack/CompanyCall
- [ ] Disable the feature

## Feedback/Metrics

<!-- This section should be filled after the end of evaluation.
     You can collect metrics from user feedback or looking at Grafana, Sentry, Kibana, etc -->

### The second evaluation

<!-- The result of the second evaluation. e.g. We didn't observe any problmes. -->

### The first evaluation

<!-- The result of the first evaluation. e.g. We've found a crucial problem thus we need to fix YYY issue before the second evaluation. -->
```
</pre>
</details>


### Code Review
Code reviews follow the standard process of using the [reviewer roulette](https://docs.gitlab.com/ee/development/code_review.html#reviewer-roulette) to choose a reviewer and a maintainer. The roulette is optional, so if a merge request contains changes that someone outside our group may not fully understand in depth, it is encouraged that a member of the Release team be chosen for the preliminary review to focus on correctly solving the problem. The intent is to leave this choice to the discretion of the engineer but raise the idea that fellow Release team members will sometimes be best able to understand the implications of the features we are implementing. The maintainer review will then be more focused on quality and code standards.

This tactic also creates an environment to ask for early review on a WIP merge request where the solution might be better refined through collaboration and also allows us to share knowledge across the team.


### Async Status Updates

Since we are a [remote](/company/culture/all-remote/) company, we utilize a Slack plugin called [Geekbot](https://geekbot.io/) to coordinate various status updates. There are currently 3 status updates configured in Geekbot, one is weekly and two are daily:

#### Weekly Status Update
The **Weekly Status Update** is configured to run at noon on Fridays, and contains three questions:

1. ***What progress was made on your deliverables this week?*** (MRs and demos are good for this)

    The goal with this question is to show off the work you did, even if it's only part of a feature. This could be a whole feature, a small part of a larger feature, an API to be used later, or even just a copy change.

2. ***What do you plan to work on next week?*** (think about what you'll be able to merge by the end of the week)

    Think about what the next most important thing is to work on, and think about what part of that you can accomplish in one week. If your priorities aren't clear, talk to your manager.

3. ***Who will you need help from to accomplish your plan for next week?*** (tag specific individuals so they know ahead of time)

    This helps to ensure that the others on the team know you'll need their help and will surface any issues earlier.

#### Daily Standup

The **Daily Standup** is configured to run each morning Monday through Thursday and posts to `#g_release` Slack channel. It has just one question:

1. ***Is there anything you could use a hand with today, or any blockers?***

    This check-in is optional and can be skipped if you don't need any help or don't have any blockers. Be sure to ask for help early, your team is always happy to lend a hand.

#### Daily Social

The optional **Daily Social** is configured to run each morning and posts to #g_cicd_social. It has just one question:

1. ***What was something awesome, fun, or interesting that you did yesterday outside of work?***

    This check-in is optional and can be skipped if you don't have anything to share.