What We Overlook When Building Our End-to-End Test Strategy
It’s a no-brainer that documenting a neatly articulated test strategy aids in the process of assuring the software quality. There are so many blogs on the subject, and there are core principles that should be followed when approaching a test strategy.
Typically, a test strategy should have:
Clear overview of the project
Risks associated with the project
It should also help identify areas where additional testing may be required. Writing an effective test strategy is skill testers gain with experience. Hence, the possibility of missing any test activity is very low when a proper test strategy is defined.
End-End Test Strategy
End-End Test Strategy lays out the test coverage for the entire application and ensures that the flow of data is maintained for all kinds of end-user actions.
Testers should always ensure that the End-End Test Strategy covers everything a client needs and, at times, can even go one step further than required to provide them with a satisfactory product.
This article will provide an overview of what should be included in a test strategy, what we overlook when building a test strategy, and how to create one that will work for your project.
Formulating an E2E test strategy
A picture representing the components of test strategy
#Testing Scope and Overview
This should contain the main Scope of the project and the overview information about who will be accessing this document, and it includes the details such as: who will review and approve these documents.
People who will usually review the test strategy:
Test Team Lead
Quality Analyst Manager
The next component in test strategy is about the testing approach. The following might help you get started.
Role and responsibility across QA team
Change management processes
Types of Testing that can be presented to the client
Defect handling mechanisms
The testing environment specifies – environments in which testing types will take place and also configurations.
It should specify instructions on creating test data and the data recovery process. A weak test strategy with incomplete information can result in final deployments plagued by defects and errors.
An essential piece of a test strategy document as it bears the information on the test automation and management tools to perform test execution.
Determines the ratio of open-source and commercial tools to assure non-functional parameters like security, performance, usability, and the number of users who’ll support.
This part of the test strategy defines the release management plan and the version history to ensure that all the modifications that happened in that release are included.
This lends itself to building a management process to determine where the new build should be available when it will be delivered, who will deploy it, how to stop the release in the event of errors, and so on.
As the name mentions, it determines how many resources will be involved in the testing process and their responsibilities for the task assigned to them.
Risk analysis – to estimate the possible risks that could occur during test execution and a clear plan to mitigate the risk and a contingency plan when risks happen in real-time.
#Reviews and Approvals
This is the last piece of information in the test strategy document where the project management and development team reviews all these activities and sign off.
A summary of review changes should be traced at the beginning of the document, along with an approved date, name, and comment.
The next section walks you through a different discussion on test strategy by Harini Mohan, our Test Lead.
What do testers overlook when building a test strategy?
Test estimation and test planning involve a lot of dependencies which may cause variance in test estimation.
When building a test strategy, managers and testers should be aware of the coordination and miscommunication challenges and plan a suitable strategy for a project.
Our Senior test engineer, Harini Mohan, shares some of the factors that test managers overlook when building a test strategy.
An application with changing requirements causes inaccurate test estimates. It’s imperative to have in place a good change management process to keep track of the requirements to minimize business risks.
A screenshot highlighting the change management practices of our client:
No formal change management board/committee established
Members of the CAB should evaluate change requests. The CAB members make recommendations that are based on the impact on existing services, the cost of the change, and other relevant factors. The CAB members should be chosen to ensure that business and technical positions are adequately represented.
A formal approach to prioritization needs to be written down based on criteria’s such as ROI, Increased revenue, Improved profitability, Cost to implement, Improved customer service, Increased competition, and Resources needed to implement.
No formal approval processes
A formal approval process is needed to minimize business risks and to review the overall change implementation steps followed as part of the release. Auditing change steps, such as code review, end-to-end testing, sign-offs, etc., is important before the change can be pushed to customers.
No test data/Wrong test data
Test data is the most pre-requisite to be considered before starting the test execution. Some projects don’t have proper test data, resulting in chaos in testing.
Therefore, testers must continuously explore, learn and apply the most efficient approaches for data collection, generation, maintenance, automation, and comprehensive data management for any functional and non-functional testing type.
Some project managers forget to allocate time for collaboration in their test strategy, which, especially in an offshore/onsite model, would result in more effort being spent on the project than planned.
Ambiguity in planning
Lack or ambiguity during the planning stage sets up for a failure upfront. When enough care is not put in the planning stage, there’s a possibility of arriving at an inefficient test strategy resulting in investing a lot of time and effort in the wrong places and high-risk areas getting left out.
Test strategy in agile
In an agile world where testers work on multiple projects/user stories in two-week sprints, it’s natural to think if we need a test strategy for such short sprints. If we create a test strategy document for each project, a lot of time would be spent iterating the documentation.
So, we identified some questions from testing communities on this topic and tried to answer them.
Disclaimer: You do you and what’s best for your team
Is a “per project” test strategy document the right approach in an agile environment?
In the Waterfall environment, you must be familiar with all the heavy documentation associated with the testing process. You are aware of the format and content of those documents, and as we know, it is a hugely time-consuming task to develop all those documents.
But the strategy you put together describes your test approaches, your test reporting methods, your strategy for managing environments, your strategy for reporting bugs, key stakeholders and decision-makers, etc.
However, a lean test strategy should work for both waterfall and agile models. The document can be for the whole project instead of sprints in the agile model. Only then does it make sense for the team leaders or managers to have enough information, make plans for test designs, etc.
Would a global test strategy document be the right approach which is more of a static company-wide test strategy document, acting more as governance, as opposed to doing this for each project?
The idea of a global document needs to be revised, as the teams should be able to decide what is appropriate for their project at any point.
Is a test strategy document even necessary at all?
OfCourse Yes! Every project must have a test strategy as it helps set up the testing framework for any project.
Test automation framework is the architectural backbone of any automation testing as it structures the entire workflow with a set of guidelines, practices, and tools that streamline and standardize the process of automating tests.
Despite being in an Agile-driven SDLC, test automation often remains treated as a distinct process. The seamless integration of the test automation team and developers largely depends on selecting the right test automation partner. This article attempts to give key factors that be considered and taken care of before choosing your test automation vendor.
Flaky tests are a common challenge faced by development teams in their quest to ensure software quality. These intermittent tests produce inconsistent results that may pass or fail unpredictably, even with no changes to the code.
A user's expected behavior when interacting with an application is documented and designed into the application using the agile software development methodology known as Behavior-driven Development (BDD). BDD assists in avoiding bloat, excessive code, unnecessary features, and lack of focus by advising developers to focus solely on the desired behaviors of an app or program.