Reading Time: 14 mins

Assuring product quality? Here’s the scorecard you need

QA scorecard

Assuring product quality? Here’s the scorecard you need

We’ve consulted companies from across the industries for years and realize time and again that setting up an intensive testing process, reviewing it, and making improvements periodically is a sure-shot way to release products that resonate with the end-users and all the stakeholders involved.

One of the product quality owners at a firm, for instance, said they had been in the industry for a long time and had been quite adaptive to the trends such as test automation, setting up a continuous integration/continuous delivery model, etc., and yet they were still finding P0 P1 defects escaping into the production.

Though setting up a product QA looks reassuring, the end results were not what they were looking for.

We agree that no product or software is defect-free. But it can defect less.

And that’s just one example.

There are companies with different approaches to software quality, some doing so great, and others have distorted views of quality by measuring wrong metrics and resulting in bad decisions.

Big or small, incremental changes happen when you establish a basic software quality scorecard that measures the STLC process right from requirements till you hit the deploy button.

QA score card: What’s it?

A software QA scorecard is similar to your fit bit watches: Like the panel displays how many steps you have taken, sleep hours and other vital information, a product QA scorecard shows the KPI of your product quality – the metrics that will present how well your product quality is.

In short: Quickly identify areas of improvement, inefficient processes, or underperforming teams with QA scorecards.

Who uses it?

Organizations of any size, be it small or big, use QA Scorecards to improve the team’s performance and their QA strategies. An organization can use the QA Scorecard template with a list of checklists as the guideline for the team to meet the Quality and the standards as expected by their customers on a regular basis/cycle.

QA scorecards become helpful to the following people in general:

  • QA analysts
  • Team leaders
  • Managers
  • Supervisors

Engineering teams can prepare a QA scorecard by themselves or seek external help based on the need.

However, the QA team by themselves cannot assume the quality/performance of the team without evaluating against any checklists, guidelines that the team/organization is following.

QA score card benefits: 

Following are the benefits that the QA team can infer from QA score card: 

  • Helps identify the areas of improvement in the quality roadmap 
  • Clear understanding of the status or quality based on the feedback 
  • Set up a benchmark and teams can learn best practices as well 
  • Helps to monitor progress of the testing activities. 

Building a QA scorecard

As mentioned earlier, QA Score Card is a basically a template with the list of standard questions based on the project or the business. It consists of a limited set of questions, let’s say 10 to 20 to determine the quality.

Every question posed to the team comes with a briefing and detailed explanation of the expectations and standards to be met to maintain the quality, both functional & non-functional. Also, the weighing of the questions differs based on what the team wants to achieve.

For example, here’s an example of QA scorecard questionnaire you can use to measure your entire QA activities

Activities Definition
Requirements
Clarity Are the requirements provided clear with detailed acceptance criteria?
Impact analysis Is impact analysis done for each story and tracked?
Dev Testing
Unit & Integration tests Are unit & integration tests automated and coverage analyzed?
Static code analysis Is static code analysis done to uncover issues even before execution?
DevOps (CICD)
Automated pipelines Is the build process automated along with execution of automated tests?
Archiving artifacts Are the builds archived & history maintained?
Test Strategy / Planning
Types of testing Are required types of testing identified and estimated during the Sprint/Release planning?
Risk Are the risks related to Schedule \ Effort \ Environment \ Resources foreseen during planning?
Test Design
Traceable to requirement Are the tests designed traceable to the requirements?
Coverage Are the peer review happening to ensure coverage?
Test Automation
Planning & Execution Is test automation planned, designed & executed consistently for different types of tests?
Quality Is the effectiveness of the automation measured?
Test Artifacts
Results Are the test results captured and tracked to the requirement?
Sign-off Is release Go-No Go substantiated with test sign-off report?
Functional Testing
Positive + Negative flows Are the functional tests covering both positive & negative flows?
Complex combinations Are the functional tests covering complex combination of flows?
Non Functional Testing
Planning & Execution Are non-functional tests planned and executed as part of release testing?
Test Data & Environment Test Data & Environment specific for non-functional tests available?
Regression Testing
Impact areas per ticket Are regressions test based on impact analysis identified and executed for each ticket?
Full per Sprint/Release Complete regression test suite executed after code freeze for each sprint / release?
Production Readiness Testing
Customer like E2E Are End to End test cases similar to customer test identified & executed?
Test Data & Environment & Resource Is Test environment along with proper test data, pheripherals/devices and other integrations available for the E2E testing?
Defect Analysis
Parameters collected Are the various parameters like Issue source, environment, component / functional area, root cause, source / identified & fixed version collected?
Analyzed & Acted Are the metrics derived based on the above parameters, analyzed and converted to actionalable items?

The scoring is usually done in numbers from 0-5 or with basic Yes/No option to arrive at the score. The final score is provided in terms of percentage out of 100. 

Example of a QA scorecard:

example of a qa scorecard

As you can see, this team scores well on CI/CD processes, but they still lag in the Production readiness testing and setting up a defect analysis system. Assuring a product/software quality depends on all of these parameters put together.

Scoring in detail:

Activities Target Achieved
Requirements
Clarity 5 4
Impact analysis 5 4
Dev Testing
Unit & Integration tests 5 4
Static code analysis 5 3
DevOps (CICD)
Automated pipelines 5 5
Archiving artifacts 5 5
Test Strategy / Planning
Types of Testing 5 4
Risk 5 3
Test Design
Traceable to requirement 5 4
Coverage 5 2
Test Automation
Planning & Execution 5 4
Quality 5 3
Test Artifacts
Results 5 4
Sign-off 5 3
Functional Testing
Positive + Negative flows 5 2
Complex combinations 5 1
Non Functional Testing
Planning & Execution 5 4
Test Data & Environment 5 4
Regression Testing
Impact areas per ticket 5 4
Full per Sprint/Release 5 4
Production Readiness Testing
Customer like E2E 5 0
Test Data & Environment & Resource 5 0
Defect Analysis
Parameters collected 5 3
Analyzed & Acted 5 2
120 76
63.33%

The objective of any product quality team is – continuous improvement and performance. A team can make use of this QA scorecard for two different types of projects.

  • Existing project
  • New project

Let’s take an example of an external QA vendor working with a customer who wants to arrive at the QA score for their product, the vendor’s team will have to come up with a clear goal before constructing the QA scorecard questionnaire.

In most cases, the customer wants to know the reason behind the issues in production, and missed bugs/defects. In such cases, the vendor will kickstart with an initial engagement, wherein they will coordinate closely with the customer's team to get a grasp of their working model, test process, and work culture and arrive at the set of questions that will help meet the goals of the customer.

The benefit of doing this is to help understand if the bug leakage to production happens due to poor QA or all of their engineering practices combined right from the requirement to release.

Arriving at the final score involves weekly meetings with the team, studying their requirements, speaking with their engineering team, processes they follow, and going through test artifacts and defect tracking tools.

Analyzing and presenting the score usually spans 8 weeks followed by the sign-off process to the client.

For teams that are just starting out to build a product, the team can have the QA scorecard template as a checklist from the initial state to maintain the quality of the product.

Doing so would help them to bridge the process and knowledge gaps, and areas they need to work on to improve their performance and be successful in terms of delivering a seamless user experience.

Criteria for QA Score Card

QA scorcard criteria

What can QA teams infer from the scorecard?

QA teams infer the following from the  Scorecard:

  • Based on the score, they identify areas of improvement viz., process, documentation
  • Root cause of the issue
  • Quality level of the product
  • Evaluate themselves against the standard procedures/metrics

Ways to improve QA Scorecard

Just like any practice, quality involves continuous improvement. Arriving at the QA scorecard should not be viewed as a one-time process. Following are a few ways to put the QA scorecard in a continuous improvement mode.

  • Monitor systematically
  • Collaborate among teams (Dev, QA, Ops)
  • Rewards and recognition within the team
  • Transparency and continuous feedback system

The article is co-authored by Vidhya Shanmugham, Test Lead @Zuci Systems. 

Keerthika

Keerthi Veerappan

An INFJ personality wielding brevity in speech and writing. Marketer @ Zucisystems.