In a scientific study, post hoc analysis (from
Latin
Latin ( or ) is a classical language belonging to the Italic languages, Italic branch of the Indo-European languages. Latin was originally spoken by the Latins (Italic tribe), Latins in Latium (now known as Lazio), the lower Tiber area aroun ...
''
post hoc
''Post hoc'' (sometimes written as ''post-hoc'') is a Latin phrase, meaning "after this" or "after the event".
''Post hoc'' may refer to:
* ''Post hoc'' analysis or ''post hoc'' test, statistical analyses that were not specified before the data w ...
'', "after this") consists of
statistical analyses that were specified after the data were seen. They are usually used to uncover specific differences between three or more group means when an
analysis of variance
Analysis of variance (ANOVA) is a family of statistical methods used to compare the Mean, means of two or more groups by analyzing variance. Specifically, ANOVA compares the amount of variation ''between'' the group means to the amount of variati ...
(ANOVA) test is significant. This typically creates a
multiple testing problem because each potential analysis is effectively a
statistical test
A statistical hypothesis test is a method of statistical inference used to decide whether the data provide sufficient evidence to reject a particular hypothesis. A statistical hypothesis test typically involves a calculation of a test statistic. ...
. Multiple testing procedures are sometimes used to compensate, but that is often difficult or impossible to do precisely. Post hoc analysis that is conducted and interpreted without adequate consideration of this problem is sometimes called ''
data dredging
Data dredging, also known as data snooping or ''p''-hacking is the misuse of data analysis to find patterns in data that can be presented as statistically significant, thus dramatically increasing and understating the risk of false positives. Th ...
'' (''p''-hacking) by critics because the statistical associations that it finds are often spurious.
Post hoc analyses are not inherently bad or good;
rather, the main requirement for their
ethical use is simply that their results not be
mispresented as the original hypothesis.
Modern editions of scientific manuals have clarified this point; for example,
APA style
APA style (also known as APA format) is a writing style and format for academic documents such as Scientific journal, scholarly journal articles and books. It is commonly used for citing sources within the field of Behavioral sciences, behavior ...
now specifies that "hypotheses should now be stated in three groupings: preplanned–primary, preplanned–secondary, and exploratory (post hoc). Exploratory hypotheses are allowable, and there should be no pressure to disguise them as if they were preplanned."
Common post hoc tests
Some common post hoc tests include:
*
Holm-Bonferroni Procedure
*
Newman-Keuls
*
Rodger’s Method
*
Scheffé’s Method
*
Tukey’s Test (see also: Studentized Range Distribution)
However, with the exception of Scheffès Method, these tests should be specified "a priori" despite being called "post-hoc" in conventional usage. For example, a difference between means could be significant with the Holm-Bonferroni method but not with the Turkey Test and vice versa. It would be poor practice for a data analyst to choose which of these tests to report based on which gave the desired result.
Causes
Sometimes the temptation to engage in post hoc analysis is motivated by a desire to produce positive results or see a project as successful. In the case of pharmaceutical research, there may be significant financial consequences to a failed trial.{{citation needed, date=November 2023
See also
*
HARKing
HARKing (hypothesizing after the results are known) is an acronym coined by social psychologist Norbert Kerr that refers to the questionable research practice of "presenting a post hoc hypothesis in the introduction of a research report as if it w ...
*
Testing hypotheses suggested by the data
In statistics, hypotheses suggested by a given dataset, when tested with the same dataset that suggested them, are likely to be accepted even when they are not true. This is because circular reasoning (double dipping) would be involved: somethi ...
*
Nemenyi test
*
Outcome switching
References
Data analysis
Multiple comparisons
Clinical research
Medical statistics