Coverage criteria
To measure what percentage of code has been executed by a test suite, one or more ''coverage criteria'' are used. These are usually defined as rules or requirements, which a test suite must satisfy.Basic coverage criteria
There are a number of coverage criteria, but the main ones are: * Function coveragehas each function (or subroutine) in the program been called? * Statement coveragehas each statement in the program been executed? * Edge coveragehas every edge in the control-flow graph been executed? ** Branch coveragehas each branch (also called the DD-path) of each control structure (such as in ''if'' and ''case'' statements) been executed? For example, given an ''if'' statement, have both the ''true'' and ''false'' branches been executed? (This is a subset of edge coverage.) * Condition coveragehas each Boolean sub-expression evaluated both to true and false? (Also called predicate coverage.) For example, consider the following C function:foo was called at least once.
* ''Statement coverage'' for this function will be satisfied if it was called for example as foo(1,1), because in this case, every line in the function would be executed—including z = x;.
* ''Branch coverage'' will be satisfied by tests calling foo(1,1) and foo(0,1) because, in the first case, both if conditions are met and z = x; is executed, while in the second case, the first condition, (x>0), is not satisfied, which prevents the execution of z = x;.
* ''Condition coverage'' will be satisfied with tests that call foo(1,0), foo(0,1), and foo(1,1). These are necessary because in the first case, (x>0) is evaluated to true, while in the second, it is evaluated to false. At the same time, the first case makes (y>0) false, the second case does not evaluate (y>0) (because of the lazy-evaluation of the Boolean operator), the third case makes it true.
In programming languages that do not perform short-circuit evaluation, condition coverage does not necessarily imply branch coverage. For example, consider the following Pascal code fragment:
a=true, b=false
* a=false, b=true
However, this set of tests does not satisfy branch coverage since neither case will meet the if condition.
Fault injection may be necessary to ensure that all conditions and branches of exception-handling code have adequate coverage during testing.
Modified condition/decision coverage
A combination of function coverage and branch coverage is sometimes also called decision coverage. This criterion requires that every point of entry and exit in the program has been invoked at least once, and every decision in the program has taken on all possible outcomes at least once. In this context, the decision is a Boolean expression comprising conditions and zero or more Boolean operators. This definition is not the same as branch coverage,Position Paper CAST-10 (June 2002).Multiple condition coverage
This criterion requires that all combinations of conditions inside each decision are tested. For example, the code fragment from the previous section will require eight tests:Parameter value coverage
Parameter value coverage (PVC) requires that in a method taking parameters, all the common values for such parameters be considered. The idea is that all common possible values for a parameter are tested. For example, common values for a string are: 1)Other coverage criteria
There are further coverage criteria, which are used less often: * Linear Code Sequence and Jump (LCSAJ) coverage a.k.a. JJ-Path coverage has every LCSAJ/JJ-path been executed?M. R. Woodward, M. A. Hennell, "On the relationship between two control-flow coverage criteria: all JJ-paths and MCDC", Information and Software Technology 48 (2006) pp. 433-440 * Path coverageHas every possible route through a given part of the code been executed? * Entry/exit coverageHas every possible call and return of the function been executed? * Loop coverageHas every possible loop been executed zero times, once, and more than once? * State coverageHas each state in a finite-state machine been reached and explored? * Data-flow coverageHas each variable definition and its usage been reached and explored?Ting Su, Ke Wu, Weikai Miao, Geguang Pu, Jifeng He, Yuting Chen, and Zhendong Su. "A Survey on Data-Flow Testing". ACM Comput. Surv. 50, 1, Article 5 (March 2017), 35 pages. Safety-critical or dependable applications are often required to demonstrate 100% of some form of test coverage. For example, the ECSS-E-ST-40C standard demands 100% statement and decision coverage for two out of four different criticality levels; for the other ones, target coverage values are up to negotiation between supplier and customer.ECSS-E-ST-40C: Space engineering - Software. ECSS Secretariat, ESA-ESTEC. March, 2009 However, setting specific target values - and, in particular, 100% - has been criticized by practitioners for various reasons (cf.C. Prause, J. Werner, K. Hornig, S. Bosecker, M. Kuhrmann (2017):In practice
The target software is built with special options or libraries and run under a controlled environment, to map every executed function to the function points in the source code. This allows testing parts of the target software that are rarely or never accessed under normal conditions, and helps reassure that the most important conditions (function points) have been tested. The resulting output is then analyzed to see what areas of code have not been exercised and the tests are updated to include these areas as necessary. Combined with other test coverage methods, the aim is to develop a rigorous, yet manageable, set of regression tests. In implementing test coverage policies within a software development environment, one must consider the following: * What are coverage requirements for the end product certification and if so what level of test coverage is required? The typical level of rigor progression is as follows: Statement, Branch/Decision, Modified Condition/Decision Coverage (MC/DC), LCSAJ ( Linear Code Sequence and Jump) * Will coverage be measured against tests that verify requirements levied on the system under test ( DO-178B)? * Is the object code generated directly traceable to source code statements? Certain certifications, (i.e. DO-178B Level A) require coverage at the assembly level if this is not the case: "Then, additional verification should be performed on the object code to establish the correctness of such generated code sequences" ( DO-178B) para-6.4.4.2. Software authors can look at test coverage results to devise additional tests and input or configuration sets to increase the coverage over vital functions. Two common forms of test coverage are statement (or line) coverage and branch (or edge) coverage. Line coverage reports on the execution footprint of testing in terms of which lines of code were executed to complete the test. Edge coverage reports which branches or code decision points were executed to complete the test. They both report a coverage metric, measured as a percentage. The meaning of this depends on what form(s) of coverage have been used, as 67% branch coverage is more comprehensive than 67% statement coverage. Generally, test coverage tools incur computation and logging in addition to the actual program thereby slowing down the application, so typically this analysis is not done in production. As one might expect, there are classes of software that cannot be feasibly subjected to these coverage tests, though a degree of coverage mapping can be approximated through analysis rather than direct testing. There are also some sorts of defects which are affected by such tools. In particular, someNotable code coverage tools
Hardware manufacturers
* Aldec * Mentor Graphics * Silvaco * SynopsysSoftware
* LDRA Testbed * Parasoft=
= PHP
= * PHPUnit, also need Xdebug to make coverage reportsUsage in industry
Test coverage is one consideration in the safety certification of avionics equipment. The guidelines by which avionics gear is certified by theSee also
* Cyclomatic complexity * Intelligent verification * Linear code sequence and jump * Modified condition/decision coverage * Mutation testing * Regression testing * Software metric * Static program analysis * White-box testing * Java code coverage toolsReferences
{{DEFAULTSORT:Code Coverage Software metrics Software testing tools