--- layout: handbook-page-toc title: "Quality Department" --- ## On this page {:.no_toc .hidden-md .hidden-lg} - TOC {:toc .hidden-md .hidden-lg} #### Child Pages ##### [On-boarding](/handbook/engineering/quality/onboarding/) ##### [Guidelines](/handbook/engineering/quality/guidelines/) ##### [Performance Indicators](/handbook/engineering/quality/performance-indicators/) ##### [Performance and Scalability](/handbook/engineering/quality/performance-and-scalability/) ##### [Project Management](/handbook/engineering/quality/project-management/) ##### [Roadmap](/handbook/engineering/quality/roadmap/) ##### [Test Engineering](/handbook/engineering/quality/test-engineering/) ##### [Triage Operations](/handbook/engineering/quality/triage-operations/) ## FY21 Direction Achieve world class enterprise grade readiness and empowering product and development teams to ship software at scale without sacrificing quality, stability or velocity. Enable Engineering and Product organizations to account for quality proactively in the planning process. Relentless focus on internal test and tooling stability for maximum productivity of our Engineering organization. The quality of the product is our collective responsibility, we the quality department makes sure everyone is aware of what the quality of the product is, empirically. To execute on this, we categorize our direction into the following areas. ### Culture of quality * Review product requirements and designs to identify risk and steer the team on quality strategy. * Review code in collaboration with counterpart teams to ensure adequate test coverage. * Partner with development teams to ensure that testability is built in. * Be a champion for better software design, promote proper testing practices and bug prevention strategies. * Be a sounding-board for our end users and non-engineering stakeholders by integrating their feedback into the product development process. ### Test automation coverage * Expand our test coverage across all test levels. * Use the data from test results and bugs to improve test gaps in our test suites. * Groom the [test pyramid](https://martinfowler.com/bliki/TestPyramid.html) coverage and ensure the right tests run are at the right place. * Improve functional, performance and third-party service integration test coverage. * Improve test results reporting mechanisms with clear and easy to understand test results for everyone. * Coach developers (internal & external) on contributing to our test scenarios. ### Tooling reliability and efficiency * Ensure maximum reliability in our internal tooling. * Optimize continuous integration pipelines for maximum efficiency. * Improve on the duration and de-duplication of test suites in all levels. * Improve stability by eliminating flaky tests and transient failures. ### Productivity, metrics and workflow optimization * Build dashboards capturing metrics in defects, test stability, tooling efficiency and workflow health. * Make metrics-driven suggestions to improve engineering processes and velocity. * Build productivity tooling, bots, label and triage helpers to speed up overall Engineering workflows. * Build automated tools to ensure the consistency and speed of our issues and merge requests workflows. ### Self-managed usecase * Build GitLab self-managed reference architecture environments. * Build test tooling and test processes that validates GitLab reference environments. * Capture self-managed customer usage and requirements to improve testing and validation. ## Quality Engineering structure ### Teams within Quality Engineering ##### [Dev QE team](dev-qe-team) ##### [Ops & CI/CD QE team](ops-qe-team) ##### [Growth QE team](growth-qe-team) ##### [Secure & Enablement QE team](secure-enablement-qe-team) ##### [Engineering Productivity team](engineering-productivity-team) ### Department members <% director_role = 'Director of Quality Engineering' roles_regexp = /(Engineer in Test|Engineering Productivity|Quality Engineering)/ %> #### Director and managers <%= direct_team(role_regexp: roles_regexp, manager_role: director_role) %> #### Individual contributors <%= stable_counterparts(role_regexp: roles_regexp, direct_manager_role: director_role) %> ### Stable counterparts Every Software Engineer in Test (SET) takes part in building our product as a DRI in [GitLab's Product Quad DRIs](/handbook/product/product-management/process/#pm-em-ux-and-set-quad-dris). They work alongside Development, Product, and UX in the [Product Development Workflow](/handbook/product-development-flow/#build-phase-1-plan). As stable counterparts, SETs should be considered critical members of the core team between Product Designers, Engineering Managers and Product Managers. * SETs should receive invites and participate in all relevant product group collaborations (meeting recordings, retro issues, planning issues, etc). * SETs should operate proactively, not waiting for other stable counterparts to provide them direction. The area a Software Engineer in Test is responsible for is defined in the [Product Stages and Groups](/handbook/product/categories/#hierarchy) and part of their title in [team org chart](/company/team/org-chart/). Every Quality Engineering Manager is aligned with an Engineering Director in the Development Department. They work at a higher level and align cross-team efforts which maps to a [Development Department section](/handbook/product/categories/#hierarchy). The area a Quality Engineering Manager is responsible for is defined in the [Product Stages and Groups](/handbook/product/categories/#hierarchy) and part of their title in [team org chart](/company/team/org-chart/). This is with the exception of the Engineering Productivity team which is based on the [span of control](/handbook/leadership/#management-group). Full-stack Engineering Productivity Engineers develop features both internal and external that improves the efficiency of engineers and development processes. Their work is separate from the regular release kickoff features per [areas of responsibility](/handbook/engineering/quality/engineering-productivity-team#areas-of-responsibility). ### Staffing planning We staff our department with the following gearing ratios: * **One Software Engineer in Test for each product group** * 1:1 ratio of Software Engineer in Test to [Product Groups](/handbook/product/categories/#hierarchy). * **Note:** The ratio may change in some product groups due to the complexity of that product area. * Approximately a 1:8 ratio of Software Engineer in Test to Development Department Engineers. * Approximately 1:1 ratio of Software Engineer in Test to Product Managers. * **One Quality Engineering Manager for each product section** * 1:1 ratio of Quality Engineering Manager to [Development Department Sections](/handbook/product/categories/#hierarchy). * Approximately a 1:1 ratio of Quality Engineering Manager to Development Department Directors. * **One Engineering Productivity Engineer for each product stage** * 1:1 ratio of Engineering Productivity Engineers to [Development Department Stage](/handbook/product/categories/#hierarchy). * Approximately a 1:40 ratio of Engineering Productivity Engineers to Development Department Engineers. ## Quality Engineering processes ### Task management and tracking We have top level boards (at the `gitlab-org` level) to communicate what is being worked on for all teams in quality engineering. Each board has a cut-line on every column that is owned by an individual. Tasks can be moved vertically to be above or below the cut-line. The cut-line is used to determine team member capacity, it is assigned to the `Backlog` milestone. The board itself pulls from `any milestone` as a catch-all so we have insights into past, current and future milestones. The cut-line also serves as a healthy discussion between engineers and their manager in their 1:1s. Every task on the board should be sized according to our [weight definitions](/handbook/engineering/quality/guidelines/#weights). #### How to use the board and cut-line {:.no_toc} * Items above the cut-line are issues in-progress and have current priority. * Items under the cut-line are not being actively worked on. * Engineers should self-update content in their column, in addition to being aware of their overall assignments before coming to their 1:1s * Managers should be aware of their overall team assignments. Please review your boards and groom them frequently according to the department goals and business needs. * Highlight blockers and tasks that are under in weight. Consider adjusting the weights to communicate the challenges/roadblocks broadly. Use `~"workflow::blocked"` to indicate a blocked issue. * Weight adjustments are a healthy discussion. Sometimes an issue maybe overweight or underweight, this calibration should be an continuous process. Nothing is perfect, we take learnings as feedback to future improvements. * We aim to have roughly 15 weights assigned to any person at a given time to ensure that engineers are not overloaded and prevent burnout. The number may change due to on-boarding period and etc. [Discussion on the intent and how to use the board](https://drive.google.com/open?id=1w3z9u_VCvMSpg7OOWvOW17mEMpezi6jH) #### Team boards {:.no_toc} * [Dev QE Team](https://gitlab.com/groups/gitlab-org/-/boards/425899) * [Ops & CI/CD QE Team](https://gitlab.com/groups/gitlab-org/-/boards/978348) * [Enablement QE Team](https://gitlab.com/groups/gitlab-org/-/boards/978354) * [Growth QE Team](https://gitlab.com/groups/gitlab-org/-/boards/1512645) * [Engineering Productivity QE Team](https://gitlab.com/groups/gitlab-org/-/boards/978615) The boards serve as a single pane of glass view for each team and help in communicating the overall status broadly, transparently and asynchronously. ### Quality department pipeline triage rotation Every member in the Quality Department shares the responsibility of analyzing the daily QA tests against `master` and `staging` branches. More details can be seen [here](/handbook/engineering/quality/guidelines/#quality-department-pipeline-triage-rotation) ### Development and infrastructure collaboration We currently have 2 venues of collaboration with [Development](/handbook/engineering/development/) and [Infrastructure](/handbook/engineering/infrastructure/) departments. #### Availability and performance grooming To mitigate performance issues, Quality Engineering will triage and groom performance issues for Product Management and Development via a weekly [Availability & Performance Grooming](https://gitlab.com/groups/gitlab-org/-/boards/1233204). The goal is to make the performance of various aspects of our application empirical with tests, environments, and metrics. Quality Engineering will ensure that performance issues are identified and/or created on the board with the label `~performance-grooming`. These issues that are surfaced to the grooming meeting will be severitized according to our definitions. ##### Identifying issues {:.no_toc} Quality Engineering will focus in identifying issues in the following areas: * Existing customer impacting performance bugs [in our issue tracker](https://gitlab.com/groups/gitlab-org/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=performance&label_name[]=customer&label_name[]=bug). * Early warning on performance degradation which have not had customer exposure but poses a risk in the future. ##### Grooming {:.no_toc} A manager in the Quality Engineering department will lead grooming with issues populated before hand in the board. Issues are walked through from high to low severity covering `~S1`, `~S2` and `~S3` performance bugs. Deliverable of grooming each issue: * A product manager of a stage or group is assigned to prioritize performance mitigation with a scheduled milestone. * Long running issues that do not have meaningful activity can be considered for closure if no longer actionable. * If additional test coverage is needed, a follow up issue to close the performance test gap will be created in [Performance test issue tracker](https://gitlab.com/gitlab-org/quality/performance). Please see the [Development department's Infrastructure and Quality collaboration](/handbook/engineering/development/#continuous-delivery-infrastructure-and-quality-collaboration) handbook section. #### Development requests Quality Engineering will track productivity, metric and process automation improvement work items in the [Development-Quality](https://gitlab.com/groups/gitlab-org/-/boards/1262515) board to service the [Development](/handbook/engineering/development/) department. Requirements and requests are to be created with the label `~dev-quality`. The head of both departments will review and groom the board on an on-going basis. Issues will be assigned and worked on by an Engineer in the [Engineering Productivity team](engineering-productivity-team) team and [communicated broadly](/handbook/engineering/quality/triage-operations/#communicate-early-and-broadly-about-expected-automation-impact) when each work item is completed. ### Meetings We try to have as few meetings as possible. We currently have 3 recurring meetings for the whole department. Everyone in the Department is free to join and the agenda is available to everyone in the company. Every meeting is also recorded. 1. **Quality Department Weekly**: This is where the whole department comes together weekly to discuss our day-to-day challenges, propose automation framework improvements and catchup on important announcements. This is also a place to connect socially with the rest of the department. This meeting happens in 2 parts to accommodate our team members across multiple timezones. * Part 1 - Wednesday 0730 UTC * Part 2 - Wednesday 2030 UTC (adjusts per PDT) 1. **Engineering Productivity Team Weekly**: The Engineering Productivity team meets weekly to discuss engineering wide process improvements. This meeting is scheduled for every Tuesday at 1300 UTC. 1. **Quality Engineering Staff Weekly**: This weekly brings together Quality Engineering's management team to discuss directional plans, long-term initiatives, hiring goals and address issues that require attention. This meeting is scheduled for every Wednesday at 1430 UTC (adjusts per PDT). ### Team retrospective The Quality team holds an asynchronous retrospective for each release. The process is automated and notes are captured in [Quality retrospectives](https://gitlab.com/gl-retrospectives/quality/) (GITLAB ONLY) ### Release process overview Moved to [release documentation](https://gitlab.com/gitlab-org/release/docs/). ## Quality Engineering initiatives ### Triage Efficiency Due to the volume of issues, one team cannot handle the triage process. We have invented [Triage Reports](/handbook/engineering/quality/triage-operations/#triage-reports) to scale the triage process within Engineering horizontally. More on our [Triage Operations](/handbook/engineering/quality/triage-operations/) ### Test Automation Framework The GitLab test automation framework is distributed across three projects: * [GitLab QA], the test orchestration tool. * The scenarios and spec files within the GitLab codebase under `/qa` in both GitLab [CE] and [EE]. #### Architecture overview See the [GitLab QA Documentation](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs) and [current architecture overview](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md). #### Installation and execution * Install and set up the [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit) * Install and run [GitLab QA] to kick off test execution. * The spec files (test cases) can be found in the [GitLab CE/EE codebase](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa) ### Performance and Scalability The Quality Department is committed to ensuring that self-managed customers have performant and scalable configurations. To that end, we are focused on creating a variety of tested and certified [Reference Architectures]. Additionally, we have developed the [GitLab Performance Toolkit], which provides several tools for measuring the performance of any GitLab instance. We use the Toolkit every day to monitor for potential performance degradations, and this tool can also be used by GitLab customers to directly test their on-premise instances. More information is available on our [Performance and Scalability](/handbook/engineering/quality/performance-and-scalability/) page. ## Other related pages * [Issue Triage Policies](/handbook/engineering/quality/issue-triage/) * [Performance of GitLab](/handbook/engineering/performance/) * [Monitoring of GitLab.com](/handbook/engineering/monitoring/) * [Production Readiness Guide](https://gitlab.com/gitlab-com/infrastructure/blob/master/.gitlab/issue_templates/production_readiness.md) [GitLab QA]: https://gitlab.com/gitlab-org/gitlab-qa [GitLab Insights]: https://gitlab.com/gitlab-org/gitlab-insights [GitLab Performance Toolkit]: https://gitlab.com/gitlab-org/quality/performance [GitLab Triage]: https://gitlab.com/gitlab-org/gitlab-triage [CE]: https://gitlab.com/gitlab-org/gitlab-ce [EE]: https://gitlab.com/gitlab-org/gitlab-ee [Reference Architectures]: https://docs.gitlab.com/ee/administration/high_availability/README.html#reference-architecture