Robustness testing is any
quality assurance methodology focused on testing the
robustness
Robustness is the property of being strong and healthy in constitution. When it is transposed into a system, it refers to the ability of tolerating perturbations that might affect the system’s functional body. In the same line ''robustness'' ca ...
of
software
Software is a set of computer programs and associated software documentation, documentation and data (computing), data. This is in contrast to Computer hardware, hardware, from which the system is built and which actually performs the work.
...
. Robustness testing has also been used to describe the process of verifying the robustness (i.e. correctness) of test cases in a test process.
ANSI
The American National Standards Institute (ANSI ) is a private non-profit organization that oversees the development of voluntary consensus standards for products, services, processes, systems, and personnel in the United States. The organ ...
and
IEEE
The Institute of Electrical and Electronics Engineers (IEEE) is a 501(c)(3) professional association for electronic engineering and electrical engineering (and associated disciplines) with its corporate office in New York City and its operati ...
have defined robustness as the degree to which a system or component can function correctly in the presence of invalid inputs or stressful environmental conditions.
The term "robustness testing" was first used by the Ballista project at
Carnegie Mellon University
Carnegie Mellon University (CMU) is a private research university in Pittsburgh, Pennsylvania. One of its predecessors was established in 1900 by Andrew Carnegie as the Carnegie Technical Schools; it became the Carnegie Institute of Technology ...
. They performed testing of
operating system
An operating system (OS) is system software that manages computer hardware, software resources, and provides common daemon (computing), services for computer programs.
Time-sharing operating systems scheduler (computing), schedule tasks for ef ...
s for dependability based on the data types of
POSIX
The Portable Operating System Interface (POSIX) is a family of standards specified by the IEEE Computer Society for maintaining compatibility between operating systems. POSIX defines both the system- and user-level application programming inte ...
API, producing complete
system crash
''System Crash'' is a Canadian youth sketch comedy television series, which aired on YTV from March 14, 1999 to December 9, 2001.
The series centred on a group of students in a junior high school media club, telling the events of their ficti ...
es in some systems. The term was also used by
OUSPG The Oulu University Secure Programming Group (OUSPG) is a research group at the University of Oulu that studies, evaluates and develops methods of implementing and testing application and system software in order to prevent, discover and eliminate ...
and VTT researchers taking part in the PROTOS project in the context of software security testing. Eventually the term ''
fuzzing
In programming and software development, fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program. The program is then monitored for exception ...
'' (which security people use for mostly non-intelligent and random robustness testing) extended to also cover model-based robustness testing.
Methods
Fault injection
Fault injection is a testing method that can be used for checking robustness of systems. During the process, testing engineers inject faults into systems and observe the system's resiliency. Test engineers can develop efficient methods which aid fault injection to find critical faults in the system.
[Moradi, Mehrdad, Bentley James Oakes, Mustafa Saraoglu, Andrey Morozov, Klaus Janschek, and Joachim Denil. "Exploring Fault Parameter Space Using Reinforcement Learning-based Fault Injection." (2020).]
See also
*
Robustness (computer science)
In computer science, robustness is the ability of a computer system to cope with errors during execution1990. IEEE Standard Glossary of Software Engineering Terminology, IEEE Std 610.12-1990 defines robustness as "The degree to which a system o ...
*
Fault injection
In computer science, fault injection is a testing technique for understanding how computing systems behave when stressed in unusual ways. This can be achieved using physical- or software-based means, or using a hybrid approach. Widely studied phys ...
References
Software testing
{{Software-eng-stub