HOME

TheInfoList



Click Here for Items Related To - CLs Method (particle Physics)
OR:
CLs Method (particle Physics) on:   [Wikipedia]   [Google]   [Amazon]

In
particle physics Particle physics or high energy physics is the study of fundamental particles and forces that constitute matter and radiation. The fundamental particles in the universe are classified in the Standard Model as fermions (matter particles) and ...
, CLs represents a
statistical Statistics (from German: '' Statistik'', "description of a state, a country") is the discipline that concerns the collection, organization, analysis, interpretation, and presentation of data. In applying statistics to a scientific, industr ...
method for setting ''upper limits'' (also called ''exclusion limits'') on model
parameter A parameter (), generally, is any characteristic that can help in defining or classifying a particular system (meaning an event, project, object, situation, etc.). That is, a parameter is an element of a system that is useful, or critical, when ...
s, a particular form of
interval estimation In statistics, interval estimation is the use of sample data to estimate an '' interval'' of plausible values of a parameter of interest. This is in contrast to point estimation, which gives a single value. The most prevalent forms of interval ...
used for parameters that can take only non-negative values. Although CLs are said to refer to Confidence Levels, "The method's name is ... misleading, as the CLs exclusion region is not a
confidence interval In frequentist statistics, a confidence interval (CI) is a range of estimates for an unknown parameter. A confidence interval is computed at a designated ''confidence level''; the 95% confidence level is most common, but other levels, such as ...
." It was first introduced by physicists working at the LEP experiment at CERN and has since been used by many
high energy physics Particle physics or high energy physics is the study of fundamental particles and forces that constitute matter and radiation. The fundamental particles in the universe are classified in the Standard Model as fermions (matter particles) and b ...
experiments. It is a
frequentist Frequentist inference is a type of statistical inference based in frequentist probability, which treats “probability” in equivalent terms to “frequency” and draws conclusions from sample-data by means of emphasizing the frequency or pr ...
method in the sense that the properties of the limit are defined by means of error probabilities, however it differs from standard confidence intervals in that the stated confidence level of the interval is not equal to its
coverage probability In statistics, the coverage probability is a technique for calculating a confidence interval which is the proportion of the time that the interval contains the true value of interest. For example, suppose our interest is in the mean number of mo ...
. The reason for this deviation is that standard upper limits based on a most powerful test necessarily produce empty intervals with some fixed probability when the parameter value is zero, and this property is considered undesirable by most physicists and statisticians. Upper limits derived with the CLs method always contain the zero value of the parameter and hence the coverage probability at this point is always 100%. The definition of CLs does not follow from any precise theoretical framework of statistical inference and is therefore described sometimes as ''ad hoc''. It has however close resemblance to concepts of ''statistical evidence'' proposed by the statistician
Allan Birnbaum Allan Birnbaum (May 27, 1923 – July 1, 1976) was an American statistician who contributed to statistical inference, foundations of statistics, statistical genetics, statistical psychology, and history of statistics. Life and career Birnbaum was ...
.


Definition

Let ''X'' be a
random sample In statistics, quality assurance, and survey methodology, sampling is the selection of a subset (a statistical sample) of individuals from within a statistical population to estimate characteristics of the whole population. Statisticians attem ...
from a
probability distribution In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomeno ...
with a real non-negative
parameter A parameter (), generally, is any characteristic that can help in defining or classifying a particular system (meaning an event, project, object, situation, etc.). That is, a parameter is an element of a system that is useful, or critical, when ...
\theta \in ,\infty). A ''CLs'' upper limit for the parameter ''θ'', with confidence level 1-\alpha', is a statistic (i.e., observable random variable) \theta_(X) which has the property: The inequality is used in the definition to account for cases where the distribution of ''X'' is discrete and an equality can not be achieved precisely. If the distribution of ''X'' is continuous Continuity or continuous may refer to: Mathematics * Continuity (mathematics), the opposing concept to discreteness; common examples include ** Continuous probability distribution or random variable in probability and statistics ** Continuous g ...
then this should be replaced by an equality. Note that the definition implies that the
coverage probability In statistics, the coverage probability is a technique for calculating a confidence interval which is the proportion of the time that the interval contains the true value of interest. For example, suppose our interest is in the mean number of mo ...
\mathbb( \theta_(X) \geq \theta ">\theta) is always larger than 1-\alpha'. An equivalent definition can be made by considering a type-I error probability (\alpha) of the test (i.e., \theta_0 is rejected when \theta_(X) < \theta_0) and the denominator to the statistical power">power Power most often refers to: * Power (physics), meaning "rate of doing work" ** Engine power, the power put out by an engine ** Electric power * Power (social and political), the ability to influence people or events ** Abusive power Power may ...
 (1-\beta). The criterion for rejecting H_0 thus requires that the ratio \alpha/(1-\beta) will be smaller than \alpha'. This can be interpreted intuitively as saying that \theta_0 is excluded because it is \alpha' less likely to observe such an extreme outcome as ''X'' when \theta_0 is true than it is when the alternative \theta=0 is true. The calculation of the upper limit is usually done by constructing a
test statistic A test statistic is a statistic (a quantity derived from the sample) used in statistical hypothesis testing.Berger, R. L.; Casella, G. (2001). ''Statistical Inference'', Duxbury Press, Second Edition (p.374) A hypothesis test is typically specifie ...
q_\theta(X) and finding the value of \theta for which : \frac = \alpha' . where q_\theta^* is the observed outcome of the experiment.


Usage in high energy physics

Upper limits based on the CLs method were used in numerous publications of experimental results obtained at particle accelerator experiments such as LEP, the
Tevatron The Tevatron was a circular particle accelerator (active until 2011) in the United States, at the Fermi National Accelerator Laboratory (also known as ''Fermilab''), east of Batavia, Illinois, and is the second highest energy particle collider ...
and the
LHC The Large Hadron Collider (LHC) is the world's largest and highest-energy particle collider. It was built by the European Organization for Nuclear Research (CERN) between 1998 and 2008 in collaboration with over 10,000 scientists and hundr ...
, most notable in the searches for new particles.


Origin

The original motivation for CLs was based on a conditional probability calculation suggested by physicist G. Zech for an event counting experiment. Suppose an experiment consists of measuring n events coming from signal and background processes, both described by
Poisson distribution In probability theory and statistics, the Poisson distribution is a discrete probability distribution that expresses the probability of a given number of events occurring in a fixed interval of time or space if these events occur with a known ...
s with respective rates s and b, namely n \sim \text(s+b). b is assumed to be known and s is the parameter to be estimated by the experiment. The standard procedure for setting an upper limit on s given an experimental outcome n^* consists of excluding values of s for which \mathbb(n \leq n^*, s+b) \leq \alpha, which guarantees at least 1-\alpha coverage. Consider, for example, a case where b=3 and n^*=0 events are observed, then one finds that s+b \geq 3 is excluded at 95% confidence level. But this implies that s \geq 0 is excluded, namely all possible values of s. Such a result is difficult to interpret because the experiment cannot essentially distinguish very small values of s from the background-only hypothesis, and thus declaring that such small values are excluded (in favor of the background-only hypothesis) seems inappropriate. To overcome this difficulty Zech suggested conditioning the probability that n \leq n^* on the observation that n_b \leq n^*, where n_b is the (unmeasurable) number of background events. The reasoning behind this is that when n_b is small the procedure is more likely to produce an error (i.e., an interval that does not cover the true value) than when n_b is large, and the distribution of n_b itself is independent of s. That is, not the over-all error probability should be reported but the conditional probability given the knowledge one has on the number of background events in the sample. This conditional probability is :\mathbb(n \leq n^* , n_b \leq n^* , s+b) = \frac = \frac. which correspond to the above definition of CLs. The first equality just uses the definition of
Conditional probability In probability theory, conditional probability is a measure of the probability of an event occurring, given that another event (by assumption, presumption, assertion or evidence) has already occurred. This particular method relies on event B occu ...
, and the second equality comes from the fact that if n \leq n^* \Rightarrow n_b \leq n^* and the number of background events is by definition independent of the signal strength.


Generalization of the conditional argument

Zech's conditional argument can be formally extended to the general case. Suppose that q(X) is a
test statistic A test statistic is a statistic (a quantity derived from the sample) used in statistical hypothesis testing.Berger, R. L.; Casella, G. (2001). ''Statistical Inference'', Duxbury Press, Second Edition (p.374) A hypothesis test is typically specifie ...
from which the confidence interval is derived, and let : p_ = \mathbb( q(X) > q^* , \theta) where q* is the outcome observed by the experiment. Then p_ can be regarded as an unmeasurable (since \theta is unknown) random variable, whose distribution is uniform between 0 and 1 independent of \theta. If the test is unbiased then the outcome q* implies : p_ \leq \mathbb( q(X) > q^* , 0 ) \equiv p_0^* from which, similarly to conditioning on n_b in the previous case, one obtains :\mathbb(q(X) \geq q^* , p_\theta \leq p_0^* , \theta) = \frac = \frac = \frac.


Relation to foundational principles

The arguments given above can be viewed as following the spirit of the
conditionality principle The conditionality principle is a Fisherian principle of statistical inference that Allan Birnbaum formally defined and studied in his 1962 JASA article. Informally, the conditionality principle can be taken as the claim that experiments which wer ...
of statistical inference, although they express a more generalized notion of conditionality which do not require the existence of an
ancillary statistic An ancillary statistic is a measure of a sample whose distribution (or whose pmf or pdf) does not depend on the parameters of the model. An ancillary statistic is a pivotal quantity that is also a statistic. Ancillary statistics can be used to co ...
. The
conditionality principle The conditionality principle is a Fisherian principle of statistical inference that Allan Birnbaum formally defined and studied in his 1962 JASA article. Informally, the conditionality principle can be taken as the claim that experiments which wer ...
however, already in its original more restricted version, formally implies the
likelihood principle In statistics, the likelihood principle is the proposition that, given a statistical model, all the evidence in a sample relevant to model parameters is contained in the likelihood function. A likelihood function arises from a probability density ...
, a result famously shown by Birnbaum. CLs does not obey the
likelihood principle In statistics, the likelihood principle is the proposition that, given a statistical model, all the evidence in a sample relevant to model parameters is contained in the likelihood function. A likelihood function arises from a probability density ...
, and thus such considerations may only be used to suggest plausibility, but not theoretical completeness from the foundational point of view. (The same however can be said on any frequentist method if the
conditionality principle The conditionality principle is a Fisherian principle of statistical inference that Allan Birnbaum formally defined and studied in his 1962 JASA article. Informally, the conditionality principle can be taken as the claim that experiments which wer ...
is regarded as necessary). Birnbaum himself suggested in his 1962 paper that the CLs ratio \alpha/(1-\beta) should be used as a measure of the strength of ''statistical evidence'' provided by significance tests, rather than \alpha alone. This followed from a simple application of the
likelihood principle In statistics, the likelihood principle is the proposition that, given a statistical model, all the evidence in a sample relevant to model parameters is contained in the likelihood function. A likelihood function arises from a probability density ...
: if the outcome of an experiment is to be only reported in a form of a "accept"/"reject" decision, then the overall procedure is equivalent to an experiment that has only two possible outcomes, with probabilities \alpha,(1-\beta) and 1-\alpha,(\beta) under H_1,(H_2). The
likelihood ratio The likelihood function (often simply called the likelihood) represents the probability of random variable realizations conditional on particular values of the statistical parameters. Thus, when evaluated on a given sample, the likelihood functi ...
associated with the outcome "reject H_1" is therefore \alpha/(1-\beta) and hence should determine the evidential interpretation of this result. (Since, for a test of two simple hypotheses, the likelihood ratio is a compact representation of the
likelihood function The likelihood function (often simply called the likelihood) represents the probability of random variable realizations conditional on particular values of the statistical parameters. Thus, when evaluated on a given sample, the likelihood funct ...
). On the other hand, if the likelihood principle is to be followed consistently, then the likelihood ratio of the original outcome should be used and not \alpha/(1-\beta), making the basis of such an interpretation questionable. Birnbaum later described this as having "at most heuristic, but not substantial, value for evidential interpretation". A more direct approach leading to a similar conclusion can be found in Birnbaum's formulation of the ''Confidence principle'', which, unlike the more common version, refers to error probabilities of both kinds. This is stated as follows:
"A concept of statistical evidence is not plausible unless it finds 'strong evidence for H_2 as against H_1' with small probability (\alpha) when H_1 is true, and with much larger probability \ (1 - \beta)\ when H_2 is true."
Such definition of confidence can naturally seem to be satisfied by the definition of CLs. It remains true that both this and the more common (as associated with the
Neyman Neyman is a surname. Notable people with the surname include: * Abraham Neyman (born 1949), Israeli mathematician *Benny Neyman (1951–2008), Dutch singer * Jerzy Neyman (1894–1981), Polish mathematician; Neyman construction and Neyman–Pearson ...
-
Pearson Pearson may refer to: Organizations Education *Lester B. Pearson College, Victoria, British Columbia, Canada *Pearson College (UK), London, owned by Pearson PLC *Lester B. Pearson High School (disambiguation) Companies *Pearson PLC, a UK-based int ...
theory) versions of the confidence principle are incompatible with the likelihood principle, and therefore no frequentist method can be regarded as a truly complete solution to the problems raised by considering conditional properties of confidence intervals.


Calculation in the large sample limit

If certain regularity conditions are met, then a general likelihood function will become a
Gaussian function In mathematics, a Gaussian function, often simply referred to as a Gaussian, is a function of the base form f(x) = \exp (-x^2) and with parametric extension f(x) = a \exp\left( -\frac \right) for arbitrary real constants , and non-zero . It i ...
in the large sample limit. In such case the CLs upper limit at confidence level 1-\alpha' (derived from the
uniformly most powerful test In statistical hypothesis testing, a uniformly most powerful (UMP) test is a statistical hypothesis testing, hypothesis test which has the greatest Statistical power, power 1 - \beta among all possible tests of a given Type I and type II errors, si ...
) is given by : \theta_ = \hat\theta + \sigma\Phi^(1 - \alpha'\Phi(\hat\theta / \sigma ) ) , where \Phi is the standard normal cumulative distribution, \hat\theta is the
maximum likelihood In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed sta ...
estimator of \theta and \sigma is its standard deviation; the latter might be estimated from the inverse of the
Fisher information In mathematical statistics, the Fisher information (sometimes simply called information) is a way of measuring the amount of information that an observable random variable ''X'' carries about an unknown parameter ''θ'' of a distribution that model ...
matrix or by using the "Asimov" data set. This result happens to be equivalent to a
Bayesian Thomas Bayes (/beɪz/; c. 1701 – 1761) was an English statistician, philosopher, and Presbyterian minister. Bayesian () refers either to a range of concepts and approaches that relate to statistical methods based on Bayes' theorem, or a follower ...
credible interval In Bayesian statistics, a credible interval is an interval within which an unobserved parameter value falls with a particular probability. It is an interval in the domain of a posterior probability distribution or a predictive distribution. The ...
if a uniform prior for \theta is used.


References


Further reading

* * * {{cite arXiv , author=Robert D. Cousins , title=Negatively Biased Relevant Subsets Induced by the Most-Powerful One-Sided Upper Confidence Limits for a Bounded Physical Parameter, eprint=1109.2023 , year=2011 , class=physics.data-an


External links


The Particle Data Group (PDG) review of statistical methods
Statistical intervals Experimental particle physics