The garden of forking paths is a problem in frequentist hypothesis testing through which researchers can unintentionally produce false positives for a tested hypothesis, through leaving themselves too many degrees of freedom. In contrast to fishing expeditions such as
data dredging
Data dredging, also known as data snooping or ''p''-hacking is the misuse of data analysis to find patterns in data that can be presented as statistically significant, thus dramatically increasing and understating the risk of false positives. Th ...
where only expected or apparently-significant results are published, this allows for a similar effect even when only one experiment is run, through a series of choices about how to implement methods and analyses, which are themselves informed by the data as it is observed and processed.
History
Exploring a forking decision-tree while analyzing data was at one point grouped with the
multiple comparisons problem
Multiple comparisons, multiplicity or multiple testing problem occurs in statistics when one considers a set of statistical inferences simultaneously or estimates a subset of parameters selected based on the observed values.
The larger the numbe ...
as an example of poor statistical method. However Gelman and Loken demonstrated that this can happen implicitly by researchers aware of best practices who only make a single comparison and only evaluate their data once.
The fallacy is believing an analysis to be free of multiple comparisons despite having had enough degrees of freedom in choosing the method, after seeing some or all of the data, to produce similarly-grounded false positives. Degrees of freedom can include choosing among main effects or interactions, methods for data exclusion, whether to combine different studies, and method of data analysis.
Multiverse analysis
A
multiverse analysis is an approach that acknowledges the multitude of analytical paths available when analyzing data. The concept is inspired by the metaphorical "garden of forking paths," which represents the multitude of potential analyses that could be conducted on a single dataset. In a multiverse analysis, researchers systematically vary their analytical choices to explore a range of possible outcomes from the same raw data.
This involves altering variables such as data inclusion/exclusion criteria, variable transformations, outlier handling, statistical models, and hypothesis tests to generate a spectrum of results that could have been obtained given different analytic decisions.
The key benefits of a multiverse analysis include.
* Transparency. It makes the analytical process more transparent by openly discussing the impact of different analytic choices on the results.
* Robustness. By examining how conclusions vary across a range of analytical scenarios, researchers can assess the robustness of their findings. If a conclusion holds across many plausible analyses, it is considered more robust and less likely to be a product of arbitrary decision-making.
* Identifying Consequential Decisions. It helps identify which analytical decisions most strongly influence the outcomes, guiding researchers towards more informed methodological choices in future studies.
This approach is valuable in fields where research findings are sensitive to the methods of data analysis, such as psychology,
neuroscience,
economics, and social sciences. Multiverse analysis aims to mitigate issues related to reproducibility and replicability by revealing how different analytical choices can lead to different conclusions from the same dataset. Thus, it encourages a more nuanced understanding of data analysis, promoting integrity and credibility in scientific research.
Concepts that are closely related to multiverse analysis are specification-curve analysis and the assessment of vibration of effects.
See also
*
Statistical hypothesis testing
A statistical hypothesis test is a method of statistical inference used to decide whether the data provide sufficient evidence to reject a particular hypothesis. A statistical hypothesis test typically involves a calculation of a test statistic. T ...
*
Researcher degrees of freedom
Researcher degrees of freedom is a concept referring to the inherent flexibility involved in the process of designing and conducting a scientific experiment, and in analyzing its results. The term reflects the fact that researchers can choose betw ...
References
Fallacies
Statistical hypothesis testing
{{statistics-stub