--- layout: handbook-page-toc title: "Category Maturity Scorecards" --- ## On this page {:.no_toc .hidden-md .hidden-lg} - TOC {:toc .hidden-md .hidden-lg} ## Intro and Goal This document outlines the process for completing a Category Maturity Scorecard. We use this process to grade the [maturity of our products](/direction/maturity/) based on user performance and feedback. If you have questions or find that this process doesn’t meet the needs of your users or product, reach out to the UX Researcher for your group. The goal of this process is to produce a maturity grade in as objective a manner as is possible given time and resource constraints. For this reason, the process is more rigorous than our normal methods of user testing. This is so we can produce a maturity grade in which we have confidence. That grade should be as free of subjective judgement as possible, and instead rely on observed metrics and self-reported user sentiment. The grade should be consistent with the standard we've established and able to be compared to other grades for other product areas. To facilitate this, we've made this process prescriptive so that it can be consistently applied by all Product Designers on all areas of our product. ## Job(s) to be Done The first step of the Category Maturity Scorecard process is to identify the job(s) to be done for your product. Each Product Designer works with their Product Manager to identify Primary and Secondary JTBDs within the category. If there is uncertainty on whether those jobs are appropriate, that is a separate research question and should be handled with its own research study. Category Maturity Scorecards are about judging the quality of experiences against a defined job to be done. ## Defining and recruiting users The next step is to decide what type of users are appropriate to put through your scenarios. You’ll want to do this early on before you even have your scenarios fully defined because recruiting users can take a good amount of time. The Product Designer and Product Manager will devise a user criteria for the JTBD(s). This will be a set of descriptions for the user(s) you're describing in your job(s) so you know who you should be recruiting for testing. You should aim to use the fewest number of descriptors possible. Your goal is to ensure you recruit appropriate users, not create a persona or narrative. To balance expediency with getting a variety of perspectives, we currently conduct this Category Maturity Scorecard research process with five participants. The criteria can be based on an existing persona or not, but needs to be specific enough to be turned into a screener survey. [Create a screener survey in Qualtrics](/handbook/engineering/ux/ux-research-training/recruiting-participants/#craft-your-screener-user-interviews-and-usability-testing) that your prospective participants will fill out to let you know if they're eligible. You can use the Category Maturity Scorecard screener survey as a template and copy it to create your screener. The template survey includes a question asking people if they consent to having their session recorded. **Due to the analysis required for Category Maturity Scorecards, participants must answer yes to this question in order to participate.** Once your screener survey is complete, open a Recruiting request issue in the [UX Research project](https://gitlab.com/gitlab-org/ux-research/) and assign it to the relevant [Research Coordinator](https://about.gitlab.com/company/team/?department=ux-research-team). The Coordinator will review your screener and begin recruiting users. ## Turning JTBD into research scenarios You need to turn your job(s) to be done into one or more research scenarios. You can create one scenario to cover one or more JTBD, or you can create multiple. Create as many scenarios as you need to adequately account for the concepts covered in the job(s). It's important that the scenarios we pose to users in this evaluation process are as open-ended as possible and do not lead them in any particular direction. Read through your job(s) to be done and assess if they make sense as a research scenario. ### Things to look for #### Is there a clear action for the user to complete in the task? It's important that your scenario has an action for the user to complete that has a well defined completion criteria. Take for example the following JTBD: *When I am investigating an event, I want to have as much of the contextual and detailed information as possible, so I can make an informed decision when responding.* This may work as a job for our purposes, but isn't specific enough to work as a suitable research scenario. As written, it's not clear what event the user is investigating, and what the user is supposed to do during or after they investigate. The JTBD could be reframed like this: > Scenario: You just got an email alert about a potential [enter a specific type of a threat here]. Investigate the issue further and once you feel you have enough information, decide if and how to respond. This scenario has a clear action for the user to execute and a well defined completion point. #### Is the job to be done comprehensive without being overly detailed? A job to be done should be able to account for multiple permutations of a single higher level action. Take for example this JTBD for configuring issue notifications and to-dos: > JTBD: When requirements, implementation methods, or capacity changes, I want to be notified quickly so that I may adjust resources, due dates, or edit scope in order to keep my roadmap updated, on-track, and visible to stakeholders and customers. This job is certainly comprehensive, but when thinking in terms of a research scenario, it leads participants in all directions. It describes three different things changing, and three different actions to take as a result. To better frame this as a research scenario, you should pick one action and reaction: > Scenario: You want to ensure that you'll be notified when requirements change on one of your issues so that you can edit due dates and keep your roadmap up to date. ## Prepare your testing environment Once you know what scenario(s) you’ll be putting your participants through, it’s important to work out what sort of interface you’ll use. Some questions to ask yourself: * Is there a reason you can’t simply use a GitLab.com account? * Do you require a self-managed instance? * Is there a meaningful difference between the experience on .com versus self-managed? * Do your scenarios require any external actions (such as an alert to be thrown) or can a participant complete everything on their own? * Does your scenario require interacting with anything besides a GitLab web-based interface, such as an email or a command-line interface? It’s important to thoroughly plan how a participant will go about completing your scenario(s), especially if you answered *yes* to any of the questions above. Involve technical counterparts early in the process if you have any uncertainty on how to enable users to go through your desired workflow(s). ### Using prototypes You may find it’s necessary to use prototyping tools to simulate certain aspects of a workflow that would be difficult or impossible to re-create in production. In cases where this **will not have a meaningful effect on the experience** this isn’t a big deal. An example would be showing users an alert email where they’re meant to read some copy and click a single CTA which takes them out of the prototype and into a production interface. However, you should not use prototypes for workflows where a user is meant to have a high degree of freedom in their usage, especially if there are multiple paths to completion of a scenario. Using a prototype in such a situation would not be a fair evaluation of that experience. ## Documenting your scenario It’s important once you decide on your scenario(s) and user criteria that you go through the flow(s) of your scenario(s) yourself. You should go through all possible ways a user could complete your scenario. Typically, there is a primary flow that users will most often use, and sometimes additional secondary flows that are less common or less efficient. You should document the **primary flow** that can be used to complete a scenario. Write down each step and record the number of **meaningful clicks** (see below) needed to reach completion. This will serve as a reference for the correct way to go through the flow for those reading your findings as well as for the analysis you’ll do of your participants’ performance. If one of your participants ends up completing a scenario using a method other than the primary flow, you'll need to document that flow and count the number of meaningful clicks in order to complete your analysis. Once you've gone through your scenario and documented the steps, have a co-worker complete the scenario as well. Ideally this person wouldn't work directly on the product featured in the scenario so that they don't have an expert level understanding of how it works. Use this to uncover any issues with how you've formulated your scenario. ## Writing a testing guide Given that Category Maturity Scorecards are meant to be a standardized process, moderators should aim to follow this testing guide as closely as possible. The moderator will typically be a Product Designer, but this is not strictly required. You are encouraged to have any relevant stakeholders attend the sessions, but it is very important they remain silent until at least the *Freeform discussion* portion of the session. ### Introduction * Introduce yourself, other stakeholders * Re-iterate recording consent and start recording * "We’d like for you to use part of our product and then give us some feedback. We’ll be giving you an activity to complete, but first I'd like to learn a little bit more about you." ### Participant introduction * Tell me about your role and your day-to-day responsibilities **_If participant is an experienced GitLab user:_** * How long have you been using GitLab? * How often do you use GitLab? * What features of GitLab do you use regularly? ### Scenario(s) #### Introduction * "In just a minute I’ll give you the activity to complete. If you’re not clear on the activity I can help clarify it, but I won’t be able to answer questions or provide any help once you begin." * "Complete the activity however you see fit. You can go anywhere and access any support material you feel you need, on GitLab or elsewhere. Act as if you were on your own trying to complete the activity for your own purposes." * "If you get stuck or don’t know what to do, try to solve your problem. If you reach a point where you have exhausted all possibilities and you feel there is nothing more you can do to complete the activity, let me know." * "As you work through the activity I'll give you, please narrate your thought process. Whatever comes to mind, just say that outloud." * "I’m going to give you the activity prompt and then once you feel like you understand it, I'll have a question for you." #### Pre-scenario * Give the participant the scenario. * Answer any clarifying questions * Only clarify the prompt. * Do not give them information that is beyond the prompt, they should seek this information out themselves. * Do not lead them towards any particular solutions. * If you’re not sure if their question falls under one of the bullets above, ask them to elaborate and provide more details. * If the participant asks a question that may jeopardize the integrity of the scenario you can try saying something like, _"I'm sorry, I'm unable to answer that because it may give too much away about the activity."_ * Once they understand the scenario, ask the *ease of completion* question: * **How difficult or easy do you expect it will be to complete this task?** * Very easy * Somewhat easy * Not difficult or easy * Somewhat difficult * Very difficult * *(Scale should be flipped for each participant. Start with ‘Very easy’ for one participant, then ‘Very difficult’ for the next, and so on.)* * **Why?** #### During the scenario * Mute your audio. * Take notes as they progress through the activity. #### Post-scenario * **Ask about confidence:** How confident are you that you've successfully completed the task? * Absolutely confident * Very confident * Somewhat confident * Not confident at all * **Ask ease of completion question again:** How difficult or easy was it to complete this task? Why? * Use the same response order as you used the first time. If you started with ‘Very easy’ before the scenarios, start with that this time. #### Scorecards with multiple scenarios If your Scorecard has multiple scenarios, repeat the pre-scenario through post scenario portions for each one. ### Discussion * Now that you've finished the activity/activities, how do you think it went? * Probe on each activity if there are multiple * **(If not ‘Absolutely confident’ on completion for a scenario)** You said you weren’t absolutely confident you successfully completed the _(Nth)_ activity. Why not? * **(If a pre- and post-ease of completion rating(s) are different)** You rated the _(Nth)_ activity (easier/more difficult) compared to before you did it. What made completing the _(Nth)_ activity (easier/more difficult) than you predicted? * **(If pre- and post-ease of completion rating(s) are the same)** Was completing this task (easier or more difficult) than you predicted? Why? #### Freeform discussion At this point you are free to ask the participant any questions you have about how they executed the scenarios. These questions should help you and your team understand any issues they experienced and provide any clarifications you may need. Any insights gleaned here won’t be part of the Scorecard rating but will end up in the *Recommendations* portion of the findings. ## Analysis ### Post-session debriefing It’s important that the moderator and any stakeholders don’t leave the call once the session is concluded. Instead, remove the participant and remain on the call. Use this time for the group to debrief on what they just experienced. The notetaker(s) should take notes on this discussion. * Have each person talk about what they feel were the major findings of the session. * Mention any issues with the session or things that should be done differently in future sessions. * For Category Maturity Scorecards, any changes you make to how you conduct sessions should only be made to ensure you’re following this procedure. Don’t deviate from this process. If there are interesting things you learn and wish to follow up on, do so in a later study. * Allow anyone to ask any questions about the content covered or otherwise say things they feel need to be said before the session concludes. ### Heuristic evaluation Answer the following question for each scenario a participant completes: * **How well did the participant complete the scenario?** * Successfully * Successfully, but with some unnecessary steps or difficulty * Partially successful * Unsuccessful #### Response rubric * **Successfully:** Workflow is smooth and painless. Clear path to reach goal. Any extra interactions are very minor and have no meaningful effect on a participants’ progress. * **Successfully, but with some unnecessary steps or difficulty:** Workflow is not smooth, but participant still completes the scenario. There are extra interactions that delay or confuse the participant. * **Partially successful:** There are two ways a scenario would be considered partially successful. * The scenario has multiple discrete steps and the participant completes some but not all. The results of the actions taken by the participants should be persistent, such that if they were to leave and come back later the state of the feature would remain the same. * The participant completes the workflow but in a manner that doesn’t actually satisfy the conditions of a scenario. * **Unsuccessful:** The participant gives up on completing the task without any partial success. #### Number of meaningful clicks Count the number of **meaningful clicks** a participant takes to complete each scenario. **_Meaningful clicks_** means clicks that are: * Within a GitLab interface or GitLab content. Clicks outside of these interfaces should not be counted. * Clicks are meant to trigger an action. Things like stray/nervous clicks, text highlighting, tabbing/mousing from one form field to another, or other clicks that are obviously not meant to produce a result should not be counted. **For interactions using a command-line interface, count the number of commands issued.** A command-line interface used to complete the scenario is considered a GitLab interface, even if the program being used is not created by GitLab. ### Grading **A (Exceeds Expectations):** * At least 80% of participants must complete all scenarios with **no more than 20% more clicks above the minimum** * At least 80% of participants must rate their post-scenario confidence as **Absolutely confident** * At least 80% of participant’s post-scenario ease of completion rating **must be higher than their pre-scenario rating**, or both ratings must be **Very easy** * Designer must designate that at least 80% of participants completed all scenarios **Successfully** **B (Meets Expectations):** * At least 80% of participants must complete all scenarios with **no more than 40% more clicks above the minimum** * At least 80% of participants must rate their post-scenario confidence as **Absolutely confident** * At least 80% of participants’ post-scenario ease of completion rating **must be the same or higher than their pre-scenario rating** * Designer must designate that at least 40% of participants completed all scenarios **Successfully** **C (Average):** * No clicks rating * At least 80% of participants must rate their post-scenario confidence as **Somewhat confident** or higher * At least 80% of participants’ post-scenario ease of completion rating **must be at most one step lower than their pre-scenario rating** * Designer must designate that at least 40% of participants completed all scenarios **Successfully, but with some unnecessary steps or difficulty** **D (Presentable):** * No clicks rating * At least 40% of participants must rate their post-scenario confidence as **Somewhat confident** or higher * At least 80% of participants’ post-scenario ease of completion rating are **at most two steps lower than their pre-scenario rating** * Designer designates that at most 60% of participants were **Unsuccessful** in completing their scenario(s). **F (Poor):** * No clicks rating * Post-scenario confidence rating fails to meet all higher criteria * 80% of participants’ post-scenario ease of completion rating are **two or more steps lower than their pre-scenario rating** * Designer designates that at minimum 80% of participants were **Unsuccessful** in completing their scenario(s) ## Documenting findings ### Creating an issue Create an issue using the Category Maturity Scorecard template. Follow the instructions in that issue to document each participants’ performance as well as the final grade. ### Recommendations In order to document changes you feel should be made based on this evaluation, create a Recommendations issue using the [issue template](https://gitlab.com/gitlab-org/gitlab-design/blob/master/.gitlab/issue_templates/UX%20Scorecard%20Part%202.md).