Foundations Of Statistics

The Foundations of Statistics are the mathematical and philosophical bases for statistical methods. These bases are the theoretical frameworks that ground and justify methods of statistical inference, estimation, hypothesis testing, uncertainty quantification, and the interpretation of statistical conclusions. Further, a foundation can be used to explain statistical paradoxes, provide descriptions of statistical laws, and guide the application of statistics to real-world problems.

Different statistical foundations may provide different, contrasting perspectives on the analysis and interpretation of data, and some of these contrasts have been subject to centuries of debate. Examples include the Bayesian inference versus frequentist inference; the distinction between Fisher's ''significance testing'' and the Neyman- Pearson ''hypothesis testing''; and whether the likelihood principle holds.

Certain frameworks may be preferred for specific applications, such as the use of Bayesian methods in fitting complex ecological models.

Bandyopadhyay & Forster identify four statistical paradigms: classical statistics (error statistics), Bayesian statistics, likelihood-based statistics, and information-based statistics using the Akaike Information Criterion. More recently, Judea Pearl reintroduced formal mathematics by attributing causality in statistical systems that addressed the fundamental limitations of both Bayesian and Neyman-Pearson methods, as discussed in his book '' Causality''.

Fisher's "significance testing" vs. Neyman–Pearson "hypothesis testing"

During the 20th century, the development of classical statistics led to the emergence of two competing foundations for inductive statistical testing. The merits of these models were extensively debated. Although a hybrid approach combining elements of both methods is commonly taught and utilized, the philosophical questions raised during the debate still remain unresolved.

Significance testing

Publications by Fisher, like " Statistical Methods for Research Workers" in 1925 and " The Design of Experiments" in 1935, contributed to the popularity of significance testing, which is a probabilistic approach to deductive inference. In practice, a statistic is computed based on the experimental data and the probability of obtaining a value greater than that statistic under a default or "null" model is compared to a predetermined threshold. This threshold represents the level of discord required (typically established by convention). One common application of this method is to determine whether a treatment has a noticeable effect based on a comparative experiment. In this case, the null hypothesis corresponds to the absence of a treatment effect, implying that the treated group and the control group are drawn from the same population. Statistical significance measures probability and does not address practical significance. It can be viewed as a criterion for the statistical signal-to-noise ratio. It is important to note that the test cannot prove the hypothesis (of no treatment effect), but it can provide evidence against it.

The Fisher significance test involves a single hypothesis, but the choice of the test statistic requires an understanding of relevant directions of deviation from the hypothesized model.

Hypothesis testing

Neyman and Pearson collaborated on the problem of selecting the most appropriate hypothesis based solely on experimental evidence, which differed from significance testing. Their most renowned joint paper, published in 1933, introduced the Neyman-Pearson lemma, which states that a ratio of probabilities serves as an effective criterion for hypothesis selection (with the choice of the threshold being arbitrary). The paper demonstrated the optimality of the Student's t-test, one of the significance tests. Neyman believed that hypothesis testing represented a generalization and improvement of significance testing. The rationale for their methods can be found in their collaborative papers.

Hypothesis testing involves considering multiple hypotheses and selecting one among them, akin to making a multiple-choice decision. The absence of evidence is not an immediate factor to be taken into account. The method is grounded in the assumption of repeated sampling from the same population (the classical frequentist assumption), although Fisher criticized this assumption.

Grounds of disagreement

The duration of the dispute allowed for a comprehensive discussion of various fundamental issues in the field of statistics.

An example exchange from 1955–1956

Fisher's attack

Repeated sampling of the same population
* Such sampling is the basis of frequentist probability
* Fisher preferred fiducial inference
Type II errors
* Which result from an alternative hypothesis
Inductive behavior
* (Vs inductive reasoning)

Neyman's rebuttal

Fisher's attack on inductive behavior has been largely successful because he selected the field of battle. While ''operational decisions'' are routinely made on a variety of criteria (such as cost), ''scientific conclusions'' from experimentation are typically made based on probability alone.
Fisher's theory of fiduciary inference is flawed
* Paradoxes are common

A purely probabilistic theory of tests requires an alternative hypothesis. Fisher's attacks on Type II errors have faded with time. In the intervening years, statistics have separated the exploratory from the confirmatory. In the current environment, the concept of Type II errors are used in power calculations for confirmatory hypothesis tests' sample size determination.

Discussion

Fisher's attack based on frequentist probability failed but was not without result. He identified a specific case (2×2 table) where the two schools of testing reached different results. This case is one of several that are still troubling. Commentators believe that the "right" answer is context-dependent. Fiducial probability has not fared well, being virtually without advocates, while frequentist probability remains a mainstream interpretation.

Fisher's attack on inductive behavior has been largely successful because he selected the field of battle. While ''operational decisions'' are routinely made on a variety of criteria (such as cost), ''scientific conclusions'' from experimentation are typically made based on probability alone.

During this exchange, Fisher also discussed the requirements for inductive inference, specifically criticizing cost functions that penalize erroneous judgments. Neyman countered by mentioning the use of such functions by Gauss and Laplace. These arguments occurred 15 years ''after'' textbooks began teaching a hybrid theory of statistical testing.

Fisher and Neyman held different perspectives on the foundations of statistics (though they both opposed the Bayesian viewpoint):
* The interpretation of probability
** The disagreement between Fisher's inductive reasoning and Neyman's inductive behavior reflected the Bayesian-Frequentist divide. Fisher was willing to revise his opinion (reaching a provisional conclusion) based on calculated probability, while Neyman was more inclined to adjust his observable behavior (making a decision) based on computed costs.
* The appropriate formulation of scientific questions, with a particular focus on modelling
* Whether it is justifiable to reject a hypothesis based on a low probability without knowing the probability of an alternative
* Whether a hypothesis could ever be accepted based solely on data
** In mathematics, deduction proves, while counter-examples disprove.
** In the Popperian philosophy of science, progress is made when theories are disproven.
* Subjectivity: Although Fisher and Neyman endeavored to minimize subjectivity, they both acknowledged the significance of "good judgment." Each accused the other of subjectivity.
** Fisher ''subjectively'' selected the null hypothesis.
** Neyman-Pearson ''subjectively'' determined the criterion for selection (which was not limited to probability).
** Both ''subjectively'' established numeric thresholds.

Fisher and Neyman diverged in their attitudes and, perhaps, their language. Fisher was a scientist and an intuitive mathematician, and inductive reasoning came naturally to him. Neyman, on the other hand, was a rigorous mathematician who relied on deductive reasoning rather than probability calculations based on experiments. Hence, there was an inherent clash between applied and theoretical approaches (between science and mathematics).

Related history

In 1938, Neyman relocated to the West Coast of the United States of America, effectively ending his collaboration with Pearson and their work on hypothesis testing. Subsequent developments in the field were carried out by other researchers.

By 1940, textbooks began presenting a hybrid approach that combined elements of significance testing and hypothesis testing. However, none of the main contributors were directly involved in the further development of the hybrid approach currently taught in introductory statistics.

Statistics subsequently branched out into various directions, including decision theory, Bayesian statistics, exploratory data analysis, robust statistics, and non-parametric statistics. Neyman-Pearson hypothesis testing made significant contributions to decision theory, which is widely employed, particularly in statistical quality control. Hypothesis testing also extended its applicability to incorporate prior probabilities, giving it a Bayesian character. While Neyman-Pearson hypothesis testing has evolved into an abstract mathematical subject taught at the post-graduate level, much of what is taught and used in undergraduate education under the umbrella of hypothesis testing can be attributed to Fisher.

Contemporary opinion

There have been no major conflicts between the two classical schools of testing in recent decades, although occasional criticism and disputes persist. However, it is highly unlikely that one theory of statistical testing will completely supplant the other in the foreseeable future.

The hybrid approach, which combines elements from both competing schools of testing, can be interpreted in different ways. Some view it as an amalgamation of two mathematically complementary ideas, while others see it as a flawed union of philosophically incompatible concepts. Fisher's approach had certain philosophical advantages, while Neyman and Pearson emphasized rigorous mathematics. Hypothesis testing remains a subject of controversy for some users, but the most widely accepted alternative method, confidence intervals, is based on the same mathematical principles.

Due to the historical development of testing, there is no single authoritative source that fully encompasses the hybrid theory as it is commonly practiced in statistics. Additionally, the terminology used in this context may lack consistency. Empirical evidence indicates that individuals, including students and instructors in introductory statistics courses, often have a limited understanding of the meaning of hypothesis testing.

Summary

* The interpretation of probability remains unresolved, although fiduciary probability is not widely embraced.
* Neither of the test methods has been completely abandoned, as they are extensively utilized for different objectives.
* Textbooks have integrated both test methods into the framework of hypothesis testing.
** Some mathematicians argue, with a few exceptions, that significance tests can be considered a specific instance of hypothesis tests.
** On the other hand, some perceive these problems and methods as separate or incompatible.
* The ongoing dispute has harmed statistical education.

Bayesian inference versus frequentist inference

Two distinct interpretations of probability have existed for a long time, one based on objective evidence and the other on subjective degrees of belief. The debate between Gauss and Laplace could have taken place more than 200 years ago, giving rise to two competing schools of statistics. Classical inferential statistics emerged primarily during the second quarter of the 20th century, largely in response to the controversial principle of indifference used in Bayesian probability at that time. The resurgence of Bayesian inference was a reaction to the limitations of frequentist probability, leading to further developments and reactions.

While the philosophical interpretations have a long history, the specific statistical terminology is relatively recent. The terms "Bayesian" and "frequent" became standardized in the second half of the 20th century. However, the terminology can be confusing, as the "classical" interpretation of probability aligns with Bayesian principles, while "classical" statistics follow the frequentist approach. Moreover, even within the term "frequentist," there are variations in interpretation, differing between philosophy and physics.

The intricate details of philosophical probability interpretations are explored elsewhere. In the field of statistics, these alternative interpretations ''allow'' for the analysis of different datasets using distinct methods based on various models, aiming to achieve slightly different objectives. When comparing the competing schools of thought in statistics, pragmatic criteria beyond philosophical considerations are taken into account.

Major contributors

Fisher and Neyman were significant figures in the development of frequentist (classical) methods. While Fisher had a unique interpretation of probability that differed from Bayesian principles, Neyman adhered strictly to the frequentist approach. In the realm of Bayesian statistical philosophy, mathematics, and methods, de Finetti, Jeffreys, and Savage emerged as notable contributors during the 20th century. Savage played a crucial role in popularizing de Finetti's ideas in English-speaking regions and establishing rigorous Bayesian mathematics. In 1965, Dennis Lindley's two-volume work titled "Introduction to Probability and Statistics from a Bayesian Viewpoint" played a vital role in introducing Bayesian methods to a wide audience. For three generations, statistics have progressed significantly, and the views of early contributors are not necessarily considered authoritative in present times.

Contrasting approaches

Frequentist inference

The earlier description briefly highlights frequentist inference, which encompasses Fisher's "significance testing" and Neyman-Pearson's "hypothesis testing." Frequentist inference incorporates various perspectives and allows for scientific conclusions, operational decisions, and parameter estimation with or without confidence intervals.

Bayesian inference

A classical frequency distribution provides information about the probability of the observed data. By applying Bayes' theorem, a more abstract concept is introduced, which involves estimating the probability of a hypothesis (associated with a theory) given the data. This concept, formerly referred to as "inverse probability," is realized through Bayesian inference. Bayesian inference involves updating the probability estimate for a hypothesis as new evidence becomes available. It explicitly considers both the evidence and prior beliefs, enabling the incorporation of multiple sets of evidence.

Comparisons of characteristics

Frequentists and Bayesians employ distinct probability models. Frequentist typically view parameters as fixed but unknown, whereas Bayesians assign probability distributions to these parameters. As a result, Bayesian discuss probabilities that frequentist do not acknowledge. Bayesian consider the probability of a theory, whereas true frequentists can only assess the evidence's consistency with the theory. For instance, a frequentist does not claim a 95% probability that the true value of a parameter falls within a confidence interval; rather, they state that 95% of confidence intervals encompass the true value.

Mathematical results

Both the frequentist and Bayesian schools are subject to mathematical critique, and neither readily embraces such criticism. For instance, Stein's paradox highlights the intricacy of determining a "flat" or "uninformative" prior probability distribution in high-dimensional spaces. While Bayesians perceive this as tangential to their fundamental philosophy, they find frequentist plagued with inconsistencies, paradoxes, and unfavorable mathematical behavior. Frequentist traveller can account for most of these issues. Certain "problematic" scenarios, like estimating the weight variability of a herd of elephants based on a single measurement (Basu's elephants), exemplify extreme cases that defy statistical estimation. The principle of likelihood has been a contentious area of debate.

Statistical results

Both the frequentist and Bayesian schools have demonstrated notable accomplishments in addressing practical challenges. Classical statistics, with its reliance on mechanical calculators and specialized printed tables, boasts a longer history of obtaining results. Bayesian methods, on the other hand, have shown remarkable efficacy in analyzing sequentially sampled information, such as radar and sonar data. Several Bayesian techniques, as well as certain recent frequentist methods like the bootstrap, necessitate the computational capabilities that have become widely accessible in the past few decades. There is an ongoing discourse regarding the integration of Bayesian and frequentist approaches, although concerns have been raised regarding the interpretation of results and the potential diminishment of methodological diversity.

Philosophical results

Bayesians share a common stance against the limitations of frequent, but they are divided into various philosophical camps (empirical, hierarchical, objective, personal, and subjective), each emphasizing different aspects. A philosopher of statistics from the frequentist perspective has observed a shift from the statistical domain to philosophical interpretations of probability over the past two generations. Some perceive that the successes achieved with Bayesian applications do not sufficiently justify the associated philosophical framework. Bayesian methods often develop practical models that deviate from traditional inference and have minimal reliance on philosophy. Neither the frequentist nor the Bayesian philosophical interpretations of probability can be considered entirely robust. The frequentist view is criticized for being overly rigid and restrictive, while the Bayesian view can encompass both objective and subjective elements, among others.

Illustrative quotations

* "Carefully used, the frequentist approach yields broadly applicable if sometimes clumsy answers"
* "To insist on unbiased requenttechniques may lead to negative (but unbiased) estimates of variance; the use of p-values in multiple tests may lead to blatant contradictions; conventional 0.95  confidence regions may consist of the whole real line. No wonder that mathematicians find it often difficult to believe that conventional statistical methods are a branch of mathematics."
* "Bayesianism is a neat and fully principled philosophy, while frequentist is a grab-bag of opportunistic, individually optimal, methods."
* "In multiparameter problems flat priors can yield very bad answers"
* " Bayes' rule says there is a simple, elegant way to combine current information with prior experience to state how much is known. It implies that sufficiently good data will bring previously disparate observers to an agreement. It makes full use of available information, and it produces decisions having the least possible error rate."
* "Bayesian statistics is about making probability statements, frequentist statistics is about evaluating probability statements."
* "Statisticians are often put in a setting reminiscent of Arrow’s paradox, where we are asked to provide estimates that are informative and unbiased and confidence statements that are correct conditional on the data and also on the underlying true parameter." (These are conflicting requirements.)
* "Formal inferential aspects are often a relatively small part of statistical analysis"
* "The two philosophies, Bayesian and frequent, are more orthogonal than antithetical."
* "A hypothesis that may be true is rejected because it has failed to predict observable results that have not occurred. This seems a remarkable procedure."

Summary

* Bayesian theory has a mathematical advantage.
** Frequentist probability has existence and consistency problems.
** But finding good priors to apply Bayesian theory remains (very?) difficult.
* Both theories have impressive records of successful application.
* Neither the philosophical interpretation of probability nor its support is robust.
* There is increasing scepticism about the connection between application and philosophy.
* Some statisticians are recommending active collaboration (beyond a cease-fire).

The likelihood principle

In common usage, likelihood is often considered synonymous with probability. However, according to statistics, this is not the case. In statistics, probability refers to variable data given a fixed hypothesis, whereas likelihood refers to variable hypotheses given a fixed set of data. For instance, when making repeated measurements with a ruler under fixed conditions, each set of observations corresponds to a probability distribution, and the observations can be seen as a sample from that distribution, following the frequentist interpretation of probability. On the other hand, a set of observations can also arise from sampling various distributions based on different observational conditions. The probabilistic relationship between a fixed sample and a variable distribution stemming from a variable hypothesis is referred to as likelihood, representing the Bayesian view of probability. For instance, a set of length measurements may represent readings taken by observers with specific characteristics and conditions.

Likelihood is a concept that was introduced and developed by Fisher over a span of more than 40 years, although earlier references to the concept exist and Fisher's support for it was not wholehearted. The concept was subsequently accepted and substantially revised by Jeffreys. In 1962, Birnbaum "proved" the likelihood principle based on premises that were widely accepted among statisticians, although his proof has been subject to dispute by statisticians and philosophers. Notably, by 1970, Birnbaum had rejected one of these premises (the conditionality principle) and had also abandoned the likelihood principle due to their incompatibility with the frequentist "confidence concept of statistical evidence." The likelihood principle asserts that all the information in a sample is contained within the likelihood function, which is considered a valid probability distribution by Bayesians but not by frequent.

Certain significance tests employed by frequentists are not consistent with the likelihood principle. Bayesian, on the other hand, embrace the principle as it aligns with their philosophical standpoint (perhaps in response to frequentist discomfort). The likelihood approach is compatible with Bayesian statistical inference, where the posterior Bayes distribution for a parameter is derived by multiplying the prior distribution by the likelihood function using Bayes' Theorem. Frequentist interpret the likelihood principle unfavourably, as it suggests a lack of concern for the reliability of evidence. The likelihood principle, according to Bayesian statistics, implies that information about the experimental design used to collect evidence does not factor into the statistical analysis of the data. Some Bayesian, including Savage, acknowledge this implication as a vulnerability.

The likelihood principle's staunchest proponents argue that it provides a more solid foundation for statistics compared to the alternatives presented by Bayesian and frequentist approaches. These supporters include some statisticians and philosophers of science. While Bayesian recognize the importance of likelihood for calculations, they contend that the posterior probability distribution serves as the appropriate basis for inference.

Modelling

Inferential statistics relies on statistical models. Classical hypothesis testing, for instance, has often relied on the assumption of data normality. To reduce reliance on this assumption, robust and nonparametric statistics have been developed. Bayesian statistics, on the other hand, interpret new observations based on prior knowledge, assuming continuity between the past and present. The experimental design assumes some knowledge of the factors to be controlled, varied, randomized, and observed. Statisticians are aware of the challenges in establishing causation, often stating that "correlation does not imply causation," which is more of a limitation in modelling than a mathematical constraint.

As statistics and data sets have become more complex, questions have arisen regarding the validity of models and the inferences drawn from them. There is a wide range of conflicting opinions on modelling.

Models can be based on scientific theory or ad hoc data analysis, each employing different methods. Advocates exist for each approach. Model complexity is a trade-off and less subjective approaches such as the Akaike information criterion and Bayesian information criterion aim to strike a balance.

Concerns have been raised even about simple regression models used in the social sciences, as a multitude of assumptions underlying model validity are often neither mentioned nor verified. In some cases, a favorable comparison between observations and the model is considered sufficient.

Traditional observation-based models often fall short in addressing many significant problems, requiring the utilization of a broader range of models, including algorithmic ones. "If the model is a poor emulation of nature, the conclusions may be wrong."

Modelling is frequently carried out inadequately, with improper methods employed, and the reporting of models is often subpar.

Given the lack of a strong consensus on the philosophical review of statistical modeling, many statisticians adhere to the cautionary words of George Box: "''All models are wrong, but some are useful.''"

Other reading

For a concise introduction to the fundamentals of statistics, refer to ''Stuart, A.; old, J.K. (1994). "Ch. 8 – Probability and statistical inference" in Kendall's Advanced Theory of Statistics, Volume I: Distribution Theory (6th ed.), published by Edward Arnold''.

In his book ''Statistics as Principled Argument'', Robert P. Abelson presents the perspective that statistics serve as a standardized method for resolving disagreements among scientists, who could otherwise engage in endless debates about the merits of their respective positions. From this standpoint, statistics can be seen as a form of rhetoric. However, the effectiveness of statistical methods depends on the consensus among all involved parties regarding the chosen approach.

See also

*Philosophy of statistics
* History of statistics
* Philosophy of probability
*Philosophy of mathematics
*Philosophy of science
*Evidence
* Likelihoodist statistics
*Probability interpretations
* Founders of statistics

Footnotes

Citations

References

*
*
* The text is a collection of essays.
*
*
*
*
* University of Houston lecture notes?
*
* Translation of the 1937 French original with later notes added.
* Preliminary version of an article for the International Encyclopedia of the Social and Behavioral Sciences.
*
*
*
*
*
*
*
*
*
*
* – A joke escalated into a serious discussion of Bayesian problems by 5  authors (Gelman, Bernardo, Kadane, Senn, Wasserman) on pages 445-478.
*
*
*
* – A working paper that explains the difference between Fisher's evidential ''p''-value and the Neyman–Pearson type  I error rate $\alpha$.
*
*
*
*
*
*
*
*
* Working paper contains numerous quotations from the sources of the dispute.
*
*
*
*
*
*
*
*
*
*
* – Lecture notes? University of Illinois at Chicago

Further reading

*
*
* .
*
* – Bayesian.
* .

External links

*
* {{cite book , series=Stanford Encyclopedia of Philosophy , url=http://plato.stanford.edu/entries/statistics/ , title=Philosophy of statistics , year=2022 , publisher=Stanford University , place=Palo Alto, CA

Philosophy of statistics