Running Non-Functional Tests In Continuous Testing Mode – Part 3

The often stated challenges in considering non-functional tests in any model (let alone continuous testing) are:

• Non-functional tests are inconsistently defined and poorly planned
• Non-functional tests are often treated as lower priority
• Lack of suitable skillset
• Cost of tools & environment involved tend to be prohibitive

This further gets complicated when considering non-functional tests in a continuous testing context:

• Non-functional test asserts are more fragile and requires substantial maintenance in rapid iterative model
• Automated analysis tends to be more objective; whereas for most of the non-functional test we need subjective analysis
• Test environment, test data and test duration plays a key role in obtaining closer to accurate test results. Maintaining a production like environment with realistic test data for detailed non-functional tests requires mature DevOps processes and significant investment in setting up the environment

With that being said, considering automation of basic non-functional tests (would it be appropriate to call them checks?) for load, latency, throughput, security and adding them to the Continuous Testing suite helps to a great extent. The question is why? To answer it, such tests can give us early feedback on the problem areas and provides us the flexibility to choose the subset of tests we want to run.

Continuous Testing mode

SQL injection vulnerabilities, for which an example was given in the previous part of the blog is one such test to consider. We also commonly see that known weaknesses and misconfigurations such as lack of the HttpOnly flag on session cookies or use of known weak SSL suites and ciphers can be automated and run in Continuous Testing mode. Such tests can be automated using browser automation tools such as Selenium Webdriver in combination with others like OWASP ZAP.

Similarly, DB & component / service level load tests and performance regressing tests for business critical user journey can help in identifying the “performance degradation” point for the “system under construction” can be ideal candidates for continuous testing. More intense/ specific tests like Ethical hacking, capacity planning and stress testing can be designed and run based on the feedback received. In short, understanding both application technical stack & business risk, concise / limited non-functional test suite can be added to continuous test suite for frequent repeated feedback cycle; whereas targeted non-functional tests can be saved for less frequent / one time milestone.

The goal behind running non-functional tests in continuous testing mode is “to be able to detect the exact moment when someone enters a line of code that affects the non-functional aspects of the system.” To get there, we need a strong Continuous Testing model not just with the right kind of tests but also with the right processes, components and tools.

Continuous Delivery is a discipline that is successful only when those involved feel responsible and accountable. A typical Continuous Delivery pipeline looks similar to the one below from dzone.com

As you can see Continuous Testing sits within “L3 – Continuous Delivery” and as mentioned above, it is not just the kind of tests alone that make Continuous Testing successful. It is about following a process based approach that is in line with the organization’s needs and quality expectations and constantly improving it based on the results obtained.

“Continuous Improvement” is what true practitioners focus on!

,

Read more

>

Running Non-Functional Tests in Continuous Testing Mode – Part 2

Running Non-Functional Tests in Continuous Testing Mode

Similar posts

A Simple approach to Handle Test Automation Failures