In
statistics
Statistics (from German language, German: ', "description of a State (polity), state, a country") is the discipline that concerns the collection, organization, analysis, interpretation, and presentation of data. In applying statistics to a s ...
, the Holm–Bonferroni method, also called the Holm method or Bonferroni–Holm method, is used to counteract the problem of
multiple comparisons
Multiple comparisons, multiplicity or multiple testing problem occurs in statistics when one considers a set of statistical inferences simultaneously or estimates a subset of parameters selected based on the observed values.
The larger the numbe ...
. It is intended to control the
family-wise error rate
In statistics, family-wise error rate (FWER) is the probability of making one or more false discoveries, or type I errors when performing multiple hypotheses tests.
Familywise and experimentwise error rates
John Tukey developed in 1953 the conce ...
(FWER) and offers a simple test
uniformly more powerful than the
Bonferroni correction
In statistics, the Bonferroni correction is a method to counteract the multiple comparisons problem.
Background
The method is named for its use of the Bonferroni inequalities.
Application of the method to confidence intervals was described by ...
. It is named after
Sture Holm, who codified the method, and
Carlo Emilio Bonferroni.
Motivation
When considering several hypotheses, the problem of
multiplicity arises: the more hypotheses are tested, the higher the probability of obtaining
Type I error
Type I error, or a false positive, is the erroneous rejection of a true null hypothesis in statistical hypothesis testing. A type II error, or a false negative, is the erroneous failure in bringing about appropriate rejection of a false null hy ...
s (
false positive
A false positive is an error in binary classification in which a test result incorrectly indicates the presence of a condition (such as a disease when the disease is not present), while a false negative is the opposite error, where the test resu ...
s). The Holm–Bonferroni method is one of many approaches for controlling the FWER, i.e., the probability that one or more Type I errors will occur, by adjusting the rejection criterion for each of the individual hypotheses.
Formulation
The method is as follows:
* Suppose you have
p-values, sorted into order lowest-to-highest
, and their corresponding hypotheses
(null hypotheses). You want the FWER to be no higher than a certain pre-specified
significance level .
* Is
? If so, reject
and continue to the next step, otherwise EXIT.
* Is
? If so, reject
also, and continue to the next step, otherwise EXIT.
* And so on: for each P value, test whether
. If so, reject
and continue to examine the larger P values, otherwise EXIT.
This method ensures that the FWER is at most
, in the strong sense.
Rationale
The simple
Bonferroni correction
In statistics, the Bonferroni correction is a method to counteract the multiple comparisons problem.
Background
The method is named for its use of the Bonferroni inequalities.
Application of the method to confidence intervals was described by ...
rejects only null hypotheses with ''p''-value less than or equal to
, in order to ensure that the FWER, i.e., the risk of rejecting one or more true null hypotheses (i.e., of committing one or more type I errors) is at most
. The cost of this protection against type I errors is an increased risk of failing to reject one or more false null hypotheses (i.e., of committing one or more type II errors).
The Holm–Bonferroni method also controls the FWER at
, but with a lower increase of type II error risk than the classical Bonferroni method. The Holm–Bonferroni method sorts the ''p''-values from lowest to highest and compares them to nominal alpha levels of
to
(respectively), namely the values
.
* The index
identifies the first ''p''-value that is ''not'' low enough to validate rejection. Therefore, the null hypotheses
are rejected, while the null hypotheses
are not rejected.
* If
then no ''p''-values were low enough for rejection, therefore no null hypotheses are rejected.
* If no such index
could be found then all ''p''-values were low enough for rejection, therefore all null hypotheses are rejected (none are accepted).
Proof
Let
be the family of hypotheses sorted by their p-values
. Let
be the set of indices corresponding to the (unknown) true null hypotheses, having
members.
Claim: If we wrongly reject some true hypothesis, there is a true hypothesis for which is at most .
First note that, in this case, there is at least one true hypothesis, so . Let be such that is the first rejected true hypothesis. Then are all rejected false hypotheses. It follows that and, hence, (1). Since is rejected, it must be by definition of the testing procedure. Using (1), we conclude that , as desired.
So let us define the random event
. Note that, for
, since
is a true null hypothesis, we have that
. Subadditivity of the probability measure implies that
. Therefore, the probability to reject a true hypothesis is at most
.
Alternative proof
The Holm–Bonferroni method can be viewed as a
closed testing procedure
In statistics, the closed testing procedure is a general method for performing more than one hypothesis test simultaneously.
The closed testing principle
Suppose there are ''k'' hypotheses ''H''1,..., ''H'k'' to be tested and the overall type ...
,
with the Bonferroni correction applied locally on each of the intersections of null hypotheses.
The closure principle states that a hypothesis
in a family of hypotheses
is rejected – while controlling the FWER at level
– if and only if all the sub-families of the intersections with
are rejected at level
.
The Holm–Bonferroni method is a ''shortcut procedure'', since it makes
or less comparisons, while the number of all intersections of null hypotheses to be tested is of order
.
It controls the FWER in the strong sense.
In the Holm–Bonferroni procedure, we first test
. If it is not rejected then the intersection of all null hypotheses
is not rejected too, such that there exists at least one intersection hypothesis for each of elementary hypotheses
that is not rejected, thus we reject none of the elementary hypotheses.
If
is rejected at level
then all the intersection sub-families that contain it are rejected too, thus
is rejected.
This is because
is the smallest in each one of the intersection sub-families and the size of the sub-families is at most
, such that the Bonferroni threshold larger than
.
The same rationale applies for
. However, since
already rejected, it sufficient to reject all the intersection sub-families of
without
. Once
holds all the intersections that contains
are rejected.
The same applies for each
.
Example
Consider four null hypotheses
with unadjusted p-values
,
,
and
, to be tested at significance level
. Since the procedure is step-down, we first test
, which has the smallest p-value
. The p-value is compared to
, the null hypothesis is rejected and we continue to the next one. Since
we reject
as well and continue. The next hypothesis
is not rejected since
. We stop testing and conclude that
and
are rejected and
and
are not rejected while controlling the family-wise error rate at level
. Note that even though
applies,
is not rejected. This is because the testing procedure stops once a failure to reject occurs.
Extensions
Holm–Šidák method
When the hypothesis tests are not negatively dependent, it is possible to replace
with:
:
resulting in a slightly more powerful test.
Weighted version
Let
be the ordered unadjusted p-values. Let
,
correspond to
. Reject
as long as
:
Adjusted ''p''-values
The adjusted
''p''-values for Holm–Bonferroni method are:
:
In the earlier example, the adjusted ''p''-values are
,
,
and
. Only hypotheses
and
are rejected at level
.
Similar adjusted ''p''-values for Holm-Šidák method can be defined recursively as
, where
. Due to the inequality
for
, the Holm-Šidák method will be more powerful than Holm–Bonferroni method.
The weighted adjusted ''p''-values are:
:
A hypothesis is rejected at level α if and only if its adjusted ''p''-value is less than α. In the earlier example using equal weights, the adjusted ''p''-values are 0.03, 0.06, 0.06, and 0.02. This is another way to see that using α = 0.05, only hypotheses one and four are rejected by this procedure.
Alternatives and usage
The Holm–Bonferroni method is "uniformly" more powerful than the classic
Bonferroni correction
In statistics, the Bonferroni correction is a method to counteract the multiple comparisons problem.
Background
The method is named for its use of the Bonferroni inequalities.
Application of the method to confidence intervals was described by ...
, meaning that it is always at least as powerful.
There are other methods for controlling the FWER that are more powerful than Holm–Bonferroni. For instance, in the
Hochberg procedure, rejection of
is made after finding the ''maximal'' index
such that
. Thus, The Hochberg procedure is uniformly more powerful than the Holm procedure. However, the Hochberg procedure requires the hypotheses to be
independent
Independent or Independents may refer to:
Arts, entertainment, and media Artist groups
* Independents (artist group), a group of modernist painters based in Pennsylvania, United States
* Independentes (English: Independents), a Portuguese artist ...
or under certain forms of positive dependence, whereas Holm–Bonferroni can be applied without such assumptions. A similar step-up procedure is the Hommel procedure, which is uniformly more powerful than the Hochberg procedure.
Naming
Carlo Emilio Bonferroni did not take part in inventing the method described here. Holm originally called the method the "sequentially rejective Bonferroni test", and it became known as Holm–Bonferroni only after some time. Holm's motives for naming his method after Bonferroni are explained in the original paper: "The use of the Boole inequality within multiple inference theory is usually called the Bonferroni technique, and for this reason we will call our test the sequentially rejective Bonferroni test."
References
{{DEFAULTSORT:Holm-Bonferroni Method
Statistical tests
Multiple comparisons