Metamorphic testing (MT) is a property-based

software testing Software testing is the act of checking whether software satisfies expectations. Software testing can provide objective, independent information about the Quality (business), quality of software and the risk of its failure to a User (computin ...

technique, which can be an effective approach for addressing the

test oracle In software testing, a test oracle (or just oracle) is a provider of information that describes correct output based on the input of a test case. Testing with an oracle involves comparing actual results of the system under test (SUT) with the ex ...

problem and test case generation problem. The test oracle problem is the difficulty of determining the expected outcomes of selected

test case In software engineering, a test case is a specification of the inputs, execution conditions, testing procedure, and expected results that define a single test to be executed to achieve a particular software testing objective, such as to exercise ...

s or to determine whether the actual outputs agree with the expected outcomes. ''Metamorphic relations'' (MRs) are necessary properties of the intended functionality of the software, and must involve ''multiple'' executions of the software. Consider, for example, a program that implements sin ''x'' correct to 100

significant figures Significant figures, also referred to as significant digits, are specific digits within a number that is written in positional notation that carry both reliability and necessity in conveying a particular quantity. When presenting the outcom ...

; a metamorphic relation for sine functions is " sin (''π − x'') = sin ''x''". Thus, even though the expected value of sin ''x''₁ for the source test case ''x''₁ = 1.234 correct to the required accuracy is not known, a follow-up test case ''x''₂ = ''π'' − 1.234 can be constructed. We can verify whether the actual outputs produced by the program under test from the source test case and the follow-up test case are consistent with the MR in question. Any inconsistency (after taking rounding errors into consideration) indicates a

failure Failure is the social concept of not meeting a desirable or intended objective, and is usually viewed as the opposite of success. The criteria for failure depends on context, and may be relative to a particular observer or belief system. On ...

of the program, caused by a fault in the implementation. MRs are not limited to programs with numerical inputs or

equality Equality generally refers to the fact of being equal, of having the same value. In specific contexts, equality may refer to: Society * Egalitarianism, a trend of thought that favors equality for all people ** Political egalitarianism, in which ...

relations. As an example, when testing a booking website, a

web search A search engine is a software system that provides hyperlinks to web pages, and other relevant information on the Web in response to a user's query. The user enters a query in a web browser or a mobile app, and the search results are typically ...

for accommodation in Sydney, Australia, returns 1,671 results; are the results of this search correct and complete? This is a test oracle problem. Based on a metamorphic relation, we may filter the price range or star rating and apply the search again; it should return a subset of the previous results. A violation of this expectation would similarly reveal a failure of the system. Metamorphic testing was invented by T.Y. Chen in a technical report in 1998. Since then, more than 150 international researchers and practitioners have applied the technique to real-life applications. Some examples include

web services A web service (WS) is either: * a service offered by an electronic device to another electronic device, communicating with each other via the Internet, or * a server running on a computer device, listening for requests at a particular port over a n ...

computer graphics Computer graphics deals with generating images and art with the aid of computers. Computer graphics is a core technology in digital photography, film, video games, digital art, cell phone and computer displays, and many specialized applications. ...

embedded systems An embedded system is a specialized computer system—a combination of a computer processor, computer memory, and input/output peripheral devices—that has a dedicated function within a larger mechanical or electronic system. It is em ...

simulation A simulation is an imitative representation of a process or system that could exist in the real world. In this broad sense, simulation can often be used interchangeably with model. Sometimes a clear distinction between the two terms is made, in ...

and

modeling A model is an informative representation of an object, person, or system. The term originally denoted the Plan_(drawing), plans of a building in late 16th-century English, and derived via French language, French and Italian language, Italian ult ...

machine learning Machine learning (ML) is a field of study in artificial intelligence concerned with the development and study of Computational statistics, statistical algorithms that can learn from data and generalise to unseen data, and thus perform Task ( ...

decision support A decision support system (DSS) is an information system that supports business or organizational decision-making activities. DSSs serve the management, operations and planning levels of an organization (usually mid and higher management) and ...

bioinformatics Bioinformatics () is an interdisciplinary field of science that develops methods and Bioinformatics software, software tools for understanding biological data, especially when the data sets are large and complex. Bioinformatics uses biology, ...

components Component may refer to: In engineering, science, and technology Generic systems *System components, an entity with discrete structure, such as an assembly or software module, within a system considered at a particular level of analysis * Lumped e ...

numerical analysis Numerical analysis is the study of algorithms that use numerical approximation (as opposed to symbolic computation, symbolic manipulations) for the problems of mathematical analysis (as distinguished from discrete mathematics). It is the study of ...

compilers In computing, a compiler is a computer program that translates computer code written in one programming language (the ''source'' language) into another language (the ''target'' language). The name "compiler" is primarily used for programs tha ...

, and even

quantum computing A quantum computer is a computer that exploits quantum mechanical phenomena. On small scales, physical matter exhibits properties of wave-particle duality, both particles and waves, and quantum computing takes advantage of this behavior using s ...

The first major survey of the field of MT was conducted in 2016. It was followed by another major survey in 2018, which highlights the challenges and opportunities and clarifies common misunderstandings. Although MT was initially proposed as a

software verification Software verification is a discipline of software engineering, programming languages, and theory of computation whose goal is to assure that software satisfies the expected requirements. Broad scope and classification A broad definition of verif ...

technique, it was later developed into a

paradigm In science and philosophy, a paradigm ( ) is a distinct set of concepts or thought patterns, including theories, research methods, postulates, and standards for what constitute legitimate contributions to a field. The word ''paradigm'' is Ancient ...

that covers verification, validation, and other types of

software quality In the context of software engineering, software quality refers to two related but distinct notions: * Software's functional quality reflects how well it complies with or conforms to a given design, based on functional requirements or specificat ...

assessment. MT can be applied independently, and can also be combined with other static and dynamic software analysis techniques such as proving and

debugging In engineering, debugging is the process of finding the Root cause analysis, root cause, workarounds, and possible fixes for bug (engineering), bugs. For software, debugging tactics can involve interactive debugging, control flow analysis, Logf ...

. In August 2018,

Google Google LLC (, ) is an American multinational corporation and technology company focusing on online advertising, search engine technology, cloud computing, computer software, quantum computing, e-commerce, consumer electronics, and artificial ...

acquired GraphicsFuzz, a startup from

Imperial College London Imperial College London, also known as Imperial, is a Public university, public research university in London, England. Its history began with Prince Albert of Saxe-Coburg and Gotha, Prince Albert, husband of Queen Victoria, who envisioned a Al ...

, to apply metamorphic testing to graphics

device drivers In the context of an operating system, a device driver is a computer program that operates or controls a particular type of device that is attached to a computer or automaton. A driver provides a software interface to hardware devices, enabli ...

for Android

smartphones A smartphone is a mobile phone with advanced computing capabilities. It typically has a touchscreen interface, allowing users to access a wide range of applications and services, such as web browsing, email, and social media, as well as mult ...

References

External links

* * * {{cite conference , title=Automated testing of real-life self-driving systems and beyond: Keynote speech , author=Zhi Quan (George) Zhou , conference=

IEEE The Institute of Electrical and Electronics Engineers (IEEE) is an American 501(c)(3) organization, 501(c)(3) public charity professional organization for electrical engineering, electronics engineering, and other related disciplines. The IEEE ...

International Conference on Artificial Intelligence Testing (AI Test 2019) , location= San Francisco East Bay, CA, USA , doi = 10.5281/zenodo.2636979, date=9 April 2019 Software testing

See also

References

External links