Continuous Testing

Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.Part of the Pipeline: Why Continuous Testing Is Essential
by Adam Auerbach, TechWell Insights August 2015The Relationship between Risk and Continuous Testing: An Interview with Wayne Ariola
by Cameron Philipp-Edmonds, Stickyminds December 2015 Continuous testing was originally proposed as a way of reducing waiting time for feedback to developers by introducing development environment-triggered tests as well as more traditional developer/tester-triggered tests.

For Continuous testing, the scope of testing extends from validating bottom-up requirements or user stories to assessing the system requirements associated with overarching business goals.DevOps: Are You Pushing Bugs to Clients Faster
by Wayne Ariola and Cynthia Dunlop, PNSQC October 2015

Adoption drivers

In the 2010s, software has become a key business differentiator.DevOps and QA: What’s the real cost of quality?
by Ericka Chickowski, DevOps.com June 2015 As a result, organizations now expect software development teams to deliver more, and more innovative, software within shorter delivery cycles.Kinks persist in Continuous Workflows
by Lisa Morgan, SD Times September 2014 To meet these demands, teams have turned to lean approaches, such as Agile, DevOps, and Continuous Delivery, to try to speed up the systems development life cycle (SDLC).Continuous Testing: Think Different
by Ian Davis, Visual Studio Magazine September 2011 After accelerating other aspects of the delivery pipeline, teams typically find that their testing process is preventing them from achieving the expected benefits of their SDLC acceleration initiative. Testing and the overall quality process remain problematic for several key reasons.

* Traditional testing processes are too slow. Iteration length has changed from months to weeks or days with the rising popularity of Agile, DevOps, and Continuous Delivery. Traditional methods of testing, which rely heavily on manual testing and automated GUI tests that require frequent updating, cannot keep pace.The Forrester Wave™ Evaluation Of Functional Test Automation (FTA) Is Out And It's All About Going Beyond GUI Testing
by Diego Lo Giudice, Forrester Research April 23, 2015 At this point, organizations tend to recognize the need to extend their test automation efforts.Continuous Development Brings Changes for Software Testers
by Amy Reichert, SearchSoftwareQuality September 2014
* Even after more automation is added to the existing test process, managers still lack adequate insight into the level of risk associated with an application at any given point in time. Understanding these risks is critical for making the rapid go/no go decisions involved in Continuous Delivery processes.Zeichick’s Take: Forget 'Continuous Integration'—the Buzzword is now 'Continuous Testing'
by Alan Zeichick, SD Times February 2014 If tests are developed without an understanding of what the business considers to be an acceptable level of risk, it is possible to have a release candidate that passes all the available tests, but which the business leaders would not consider to be ready for release.Buy the Wrong Software? A Fix Can Cost $700,000 A Conversation with voke’s Theresa Lanowitz
by Dom Nicastro , CMS Wire October 2014 For the test results to accurately indicate whether each release candidate meets business expectations, the approach to designing tests must be based on the business's tolerance for risks related to security, performance, reliability, and compliance. In addition to having unit tests that check code at a very granular bottom-up level, there is a need for a broader suite of tests to provide a top-down assessment of the release candidate's business risk.
* Even if testing is automated and tests effectively measure the level of business risk, teams without a coordinated end-to-end quality process tend to have trouble satisfying the business expectations within today's compressed delivery cycles. Trying to remove risks at the end of each iteration has been shown to be significantly slower and more resource-intensive than building quality into the product through defect prevention strategies such as development testing.

Organizations adopt Continuous Testing because they recognize that these problems are preventing them from delivering quality software at the desired speed. They recognize the growing importance of software as well as the rising cost of software failure, and they are no longer willing to make a tradeoff between time, scope, and quality.Theresa Lanowitz Talks Extreme Test Automation at STAREAST 2014
by Beth Romanik, TechWell Insights May 2014Guest View: What’s keeping you from Continuous?
by Noel Wurst, SD Times November 2015

Goals and benefits

The goal of continuous testing is to provide fast and continuous feedback regarding the level of business risk in the latest build or release candidate. This information can then be used to determine if the software is ready to progress through the delivery pipeline at any given time.Manage the Business Risks of Application Development with Continuous Testing
by Wayne Ariola, CM Crossroads September 2014

Since testing begins early and is executed continuously, application risks are exposed soon after they are introduced. Development teams can then prevent those problems from progressing to the next stage of the SDLC. This reduces the time and effort that need to be spent finding and fixing defects. As a result, it is possible to increase the speed and frequency at which quality software (software that meets expectations for an acceptable level of risk) is delivered, as well as decrease technical debt.Shift Left and Put Quality First
by Adam Auerbach, TechWell Insights October 2014

Moreover, when software quality efforts and testing are aligned with business expectations, test execution produces a prioritized list of actionable tasks (rather than a potentially overwhelming number of findings that require manual review). This helps teams focus their efforts on the quality tasks that will have the greatest impact, based on their organization's goals and priorities.

Additionally, when teams are continuously executing a broad set of continuous tests throughout the SDLC, they amass metrics regarding the quality of the process as well as the state of the software. The resulting metrics can be used to re-examine and optimize the process itself, including the effectiveness of those tests. This information can be used to establish a feedback loop that helps teams incrementally improve the process. Frequent measurement, tight feedback loops, and continuous improvement are key principles of DevOps.Practices for DevOps and Continuous Delivery
by Ben Linders, InfoQ July 2015

Scope of testing

Continuous testing includes the validation of both functional requirements and non-functional requirements.

For testing functional requirements (functional testing), Continuous Testing often involves unit tests, API testing, integration testing, and system testing. For testing non-functional requirements ( non-functional testing - to determine if the application meets expectations around performance, security, compliance, etc.), it involves practices such as static code analysis, security testing, performance testing, etc.The Power of Continuous Performance Testing
by Don Prather, Stickyminds August 2015 Tests should be designed to provide the earliest possible detection (or prevention) of the risks that are most critical for the business or organization that is releasing the software.The Importance of Shifting Right in DevOps
by Bob Aiello, CM Crossroads December 2014

Teams often find that in order to ensure that test suite can run continuously and effectively assesses the level of risk, it's necessary to shift focus from GUI testing to API testing because 1) APIs (the "transaction layer") are considered the most stable interface to the system under test, and 2) GUI tests require considerable rework to keep pace with the frequent changes typical of accelerated release processes; tests at the API layer are less brittle and easier to maintain.Produce Better Software by Using a Layered Testing Strategy
by Sean Kenefick, Gartner January 7, 2014

Tests are executed during or alongside continuous integration—at least daily.Experiences from Continuous Testing at Siemens Healthcare
by Ben Linders, InfoQ February 2015 For teams practicing continuous delivery, tests are commonly executed many times a day, every time that the application is updated in to the version control system.Testing in a Continuous Delivery World
by Rob Marvin, SD Times June 2014

Ideally, all tests are executed across all non-production test environments. To ensure accuracy and consistency, testing should be performed in the most complete, production-like environment possible. Strategies for increasing test environment stability include virtualization software (for dependencies your organization can control and image) service virtualization (for dependencies beyond your scope of control or unsuitable for imaging), and test data management.DevOps- Not a Market, but a Tool-Centric Philosophy That Supports a Continuous Delivery Value Chain
by Laurie F. Wurster, Ronni J. Colville, Jim Duggan, Gartner February, 2015

Common practices

* Testing should be a collaboration of Development, QA, and Operations—aligned with the priorities of the line of business—within a coordinated, end-to-end quality process.Keep your Software Healthy During Agile Development
by Adrian Bridgwater, ComputerWeekly November 2013
* Tests should be logically-componentized, incremental, and repeatable; results must be deterministic and meaningful.
* All tests need to be run at some point in the build pipeline, but not all tests need to be run all the time since some tests are more resource expensive ( integration tests) than other (unit tests).
* Eliminate test data and environment constraints so that tests can run constantly and consistently in production-like environments.
* To minimize false positives, minimize test maintenance, and more effectively validate use cases across modern systems with multitier architectures, teams should emphasize API testing over GUI testing.

Challenges/roadblocks

Since modern applications are highly distributed, test suites that exercise them typically require access to dependencies that are not readily available for testing (e.g., third-party services, mainframes that are available for testing only in limited capacity or at inconvenient times, etc.) Moreover, with the growing adoption of Agile and parallel development processes, it is common for end-to-end functional tests to require access to dependencies that are still evolving or not yet implemented. This problem can be addressed by using service virtualization to simulate the application under test's (AUT's) interactions with the missing or unavailable dependencies. It can also be used to ensure that data, performance, and behavior is consistent across the various test runs.

One reason teams avoid continuous testing is that their infrastructure is not scalable enough to continuously execute the test suite. This problem can be addressed by focusing the tests on the business's priorities, splitting the test base, and parallelizing the testing with application release automation tools.

Continuous testing vs automated testing

The goal of Continuous Testing is to apply "extreme automation" to stable, production-like test environments. Automation is essential for Continuous Testing.Extreme automation, meet the pre-production life cycle
by Alexandra Weber Morales, SD Times January 2014 But automated testing is not the same as Continuous Testing.

Automated testing involves automated, CI-driven execution of whatever set of tests the team has accumulated. Moving from automated testing to continuous testing involves executing a set of tests that is specifically designed to assess the business risks associated with a release candidate, and to regularly execute these tests in the context of stable, production-like test environments. Some differences between automated and continuous testing:

* With automated testing, a test failure may indicate anything from a critical issue to a violation of a trivial naming standard. With continuous testing, a test failure always indicates a critical business risk.
* With continuous testing, a test failure is addressed via a clear workflow for prioritizing defects vs. business risks and addressing the most critical ones first.
* With continuous testing, each time a risk is identified, there is a process for exposing all similar defects that might already have been introduced, as well as preventing this same problem from recurring in the future.

Predecessors

Since the 1990s, Continuous test-driven development has been used to provide programmers rapid feedback on whether the code they added a) functioned properly and b) unintentionally changed or broke existing functionality. This testing, which was a key component of Extreme Programming, involves automatically executing unit tests (and sometimes acceptance tests or smoke tests) as part of the automated build, often many times a day. These tests are written prior to implementation; passing tests indicate that implementation is successful.Continuous Integration (original version)
by Martin Fowler, DevOps.com September 2000

Continuous testing tools

Research firms Forrester Research and Gartner made Continuous Testing a primary consideration in their annual evaluations of test automation tools. Gartner published an updated version of the research in 2019.

Gartner evaluated 10 tools that met their criteria for enterprise-grade test automation tools. The evaluation involved inquiries with Gartner clients, surveys of tool users, vendor responses to Gartner questions, vendor product demonstrations. Gartner required tools to support native Windows desktop application testing and Android or iOS testing support as well as support 3 of the following: responsive web applications, mobile applications, package applications, API/web services. The results of the 2019 Magic Quadrant research are:Magic Quadrant for Software Test Automation
Gartner, November 25, 2019
* Leaders: Eggplant, SmartBear Software, Tricentis
* Challengers: IBM, Micro Focus
* Visionaries: Broadcom, Parasoft
* Niche players: froglogic, Ranorex, Worksoft

In 2020, Forrester Research evaluated 15 tools that met their criteria for enterprise-grade test functional automation tools. Forrester determined 26 criteria based on past research, user needs, and expert interviews, then evaluated products versus that criteria based on vendor responses to Forrester questions, vendor product demonstrations, and customer interviews. Forrester required tools to have cross-browser, mobile, UI, and API testing capabilities. The results of the 2020 Forrester wave are:
* Leaders: ACCELQ, Eggplant, Parasoft, Tricentis
* Strong performers: Broadcom, IBM, Mabl, Micro Focus, Perforce, Sauce Labs, SmartBear Software
* Contenders: Cyara, Expiretest, Worksoft
* Challengers: Ranorex

See also

* Continuous delivery
* Continuous integration
* DevOps
* Release management
* Service virtualization
* Software testing
* Test automation

Further reading

*
*
*
*
*Rosalind Radcliffe (2021)
''Enterprise Bug Busting, From Testing through CI/CD to Deliver Business Results''
Accelerated Strategies Press. ISBN 978-1-09838-149-3.

References

{{Authority control

Software testing