HOME

TheInfoList



Click Here for Items Related To - CLs Upper Limits
OR:
CLs Upper Limits on:   [Wikipedia]   [Google]   [Amazon]

In
particle physics Particle physics or high energy physics is the study of fundamental particles and forces that constitute matter and radiation. The fundamental particles in the universe are classified in the Standard Model as fermions (matter particles) an ...
, CLs represents a
statistical Statistics (from German: ''Statistik'', "description of a state, a country") is the discipline that concerns the collection, organization, analysis, interpretation, and presentation of data. In applying statistics to a scientific, industria ...
method for setting ''upper limits'' (also called ''exclusion limits'') on model
parameter A parameter (), generally, is any characteristic that can help in defining or classifying a particular system (meaning an event, project, object, situation, etc.). That is, a parameter is an element of a system that is useful, or critical, when ...
s, a particular form of
interval estimation In statistics, interval estimation is the use of sample data to estimate an '' interval'' of plausible values of a parameter of interest. This is in contrast to point estimation, which gives a single value. The most prevalent forms of interval es ...
used for parameters that can take only non-negative values. Although CLs are said to refer to Confidence Levels, "The method's name is ... misleading, as the CLs exclusion region is not a
confidence interval In frequentist statistics, a confidence interval (CI) is a range of estimates for an unknown parameter. A confidence interval is computed at a designated ''confidence level''; the 95% confidence level is most common, but other levels, such as 9 ...
." It was first introduced by physicists working at the LEP experiment at
CERN The European Organization for Nuclear Research, known as CERN (; ; ), is an intergovernmental organization that operates the largest particle physics laboratory in the world. Established in 1954, it is based in a northwestern suburb of Gene ...
and has since been used by many high energy physics experiments. It is a frequentist method in the sense that the properties of the limit are defined by means of error probabilities, however it differs from standard confidence intervals in that the stated confidence level of the interval is not equal to its
coverage probability In statistics, the coverage probability is a technique for calculating a confidence interval which is the proportion of the time that the interval contains the true value of interest. For example, suppose our interest is in the mean number of mon ...
. The reason for this deviation is that standard upper limits based on a most powerful test necessarily produce empty intervals with some fixed probability when the parameter value is zero, and this property is considered undesirable by most physicists and statisticians. Upper limits derived with the CLs method always contain the zero value of the parameter and hence the coverage probability at this point is always 100%. The definition of CLs does not follow from any precise theoretical framework of
statistical inference Statistical inference is the process of using data analysis to infer properties of an underlying probability distribution, distribution of probability.Upton, G., Cook, I. (2008) ''Oxford Dictionary of Statistics'', OUP. . Inferential statistical ...
and is therefore described sometimes as ''ad hoc''. It has however close resemblance to concepts of ''statistical evidence'' proposed by the statistician
Allan Birnbaum Allan Birnbaum (May 27, 1923 – July 1, 1976) was an American statistician who contributed to statistical inference, foundations of statistics, statistical genetics, statistical psychology, and history of statistics. Life and career Birnbaum w ...
.


Definition

Let ''X'' be a random sample from a
probability distribution In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomenon i ...
with a real non-negative
parameter A parameter (), generally, is any characteristic that can help in defining or classifying a particular system (meaning an event, project, object, situation, etc.). That is, a parameter is an element of a system that is useful, or critical, when ...
\theta \in ,\infty). A ''CLs'' upper limit for the parameter ''θ'', with confidence level 1-\alpha', is a statistic (i.e., observable random variable) \theta_(X) which has the property: The inequality is used in the definition to account for cases where the distribution of ''X'' is discrete and an equality can not be achieved precisely. If the distribution of ''X'' is continuous Continuity or continuous may refer to: Mathematics * Continuity (mathematics), the opposing concept to discreteness; common examples include ** Continuous probability distribution or random variable in probability and statistics ** Continuous ...
then this should be replaced by an equality. Note that the definition implies that the
coverage probability In statistics, the coverage probability is a technique for calculating a confidence interval which is the proportion of the time that the interval contains the true value of interest. For example, suppose our interest is in the mean number of mon ...
\mathbb( \theta_(X) \geq \theta ">\theta) is always larger than 1-\alpha'. An equivalent definition can be made by considering a type-I error probability (\alpha) of the test (i.e., \theta_0 is rejected when \theta_(X) < \theta_0) and the denominator to the statistical power">power (1-\beta). The criterion for rejecting H_0 thus requires that the ratio \alpha/(1-\beta) will be smaller than \alpha'. This can be interpreted intuitively as saying that \theta_0 is excluded because it is \alpha' less likely to observe such an extreme outcome as ''X'' when \theta_0 is true than it is when the alternative \theta=0 is true. The calculation of the upper limit is usually done by constructing a test statistic q_\theta(X) and finding the value of \theta for which : \frac = \alpha' . where q_\theta^* is the observed outcome of the experiment.


Usage in high energy physics

Upper limits based on the CLs method were used in numerous publications of experimental results obtained at particle accelerator experiments such as LEP, the Tevatron and the
LHC The Large Hadron Collider (LHC) is the world's largest and highest-energy particle collider. It was built by the European Organization for Nuclear Research (CERN) between 1998 and 2008 in collaboration with over 10,000 scientists and hundre ...
, most notable in the searches for new particles.


Origin

The original motivation for CLs was based on a conditional probability calculation suggested by physicist G. Zech for an event counting experiment. Suppose an experiment consists of measuring n events coming from signal and background processes, both described by Poisson distributions with respective rates s and b, namely n \sim \text(s+b). b is assumed to be known and s is the parameter to be estimated by the experiment. The standard procedure for setting an upper limit on s given an experimental outcome n^* consists of excluding values of s for which \mathbb(n \leq n^*, s+b) \leq \alpha, which guarantees at least 1-\alpha coverage. Consider, for example, a case where b=3 and n^*=0 events are observed, then one finds that s+b \geq 3 is excluded at 95% confidence level. But this implies that s \geq 0 is excluded, namely all possible values of s. Such a result is difficult to interpret because the experiment cannot essentially distinguish very small values of s from the background-only hypothesis, and thus declaring that such small values are excluded (in favor of the background-only hypothesis) seems inappropriate. To overcome this difficulty Zech suggested conditioning the probability that n \leq n^* on the observation that n_b \leq n^*, where n_b is the (unmeasurable) number of background events. The reasoning behind this is that when n_b is small the procedure is more likely to produce an error (i.e., an interval that does not cover the true value) than when n_b is large, and the distribution of n_b itself is independent of s. That is, not the over-all error probability should be reported but the conditional probability given the knowledge one has on the number of background events in the sample. This conditional probability is :\mathbb(n \leq n^* , n_b \leq n^* , s+b) = \frac = \frac. which correspond to the above definition of CLs. The first equality just uses the definition of Conditional probability, and the second equality comes from the fact that if n \leq n^* \Rightarrow n_b \leq n^* and the number of background events is by definition independent of the signal strength.


Generalization of the conditional argument

Zech's conditional argument can be formally extended to the general case. Suppose that q(X) is a test statistic from which the confidence interval is derived, and let : p_ = \mathbb( q(X) > q^* , \theta) where q* is the outcome observed by the experiment. Then p_ can be regarded as an unmeasurable (since \theta is unknown) random variable, whose distribution is uniform between 0 and 1 independent of \theta. If the test is unbiased then the outcome q* implies : p_ \leq \mathbb( q(X) > q^* , 0 ) \equiv p_0^* from which, similarly to conditioning on n_b in the previous case, one obtains :\mathbb(q(X) \geq q^* , p_\theta \leq p_0^* , \theta) = \frac = \frac = \frac.


Relation to foundational principles

The arguments given above can be viewed as following the spirit of the
conditionality principle The conditionality principle is a Fisherian principle of statistical inference that Allan Birnbaum formally defined and studied in his 1962 JASA article. Informally, the conditionality principle can be taken as the claim that experiments which we ...
of statistical inference, although they express a more generalized notion of conditionality which do not require the existence of an ancillary statistic. The
conditionality principle The conditionality principle is a Fisherian principle of statistical inference that Allan Birnbaum formally defined and studied in his 1962 JASA article. Informally, the conditionality principle can be taken as the claim that experiments which we ...
however, already in its original more restricted version, formally implies the likelihood principle, a result famously shown by Birnbaum. CLs does not obey the likelihood principle, and thus such considerations may only be used to suggest plausibility, but not theoretical completeness from the foundational point of view. (The same however can be said on any frequentist method if the
conditionality principle The conditionality principle is a Fisherian principle of statistical inference that Allan Birnbaum formally defined and studied in his 1962 JASA article. Informally, the conditionality principle can be taken as the claim that experiments which we ...
is regarded as necessary). Birnbaum himself suggested in his 1962 paper that the CLs ratio \alpha/(1-\beta) should be used as a measure of the strength of ''statistical evidence'' provided by significance tests, rather than \alpha alone. This followed from a simple application of the likelihood principle: if the outcome of an experiment is to be only reported in a form of a "accept"/"reject" decision, then the overall procedure is equivalent to an experiment that has only two possible outcomes, with probabilities \alpha,(1-\beta) and 1-\alpha,(\beta) under H_1,(H_2). The likelihood ratio associated with the outcome "reject H_1" is therefore \alpha/(1-\beta) and hence should determine the evidential interpretation of this result. (Since, for a test of two simple hypotheses, the likelihood ratio is a compact representation of the
likelihood function The likelihood function (often simply called the likelihood) represents the probability of random variable realizations conditional on particular values of the statistical parameters. Thus, when evaluated on a given sample, the likelihood funct ...
). On the other hand, if the likelihood principle is to be followed consistently, then the likelihood ratio of the original outcome should be used and not \alpha/(1-\beta), making the basis of such an interpretation questionable. Birnbaum later described this as having "at most heuristic, but not substantial, value for evidential interpretation". A more direct approach leading to a similar conclusion can be found in Birnbaum's formulation of the ''Confidence principle'', which, unlike the more common version, refers to error probabilities of both kinds. This is stated as follows:
"A concept of statistical evidence is not plausible unless it finds 'strong evidence for H_2 as against H_1' with small probability (\alpha) when H_1 is true, and with much larger probability \ (1 - \beta)\ when H_2 is true."
Such definition of confidence can naturally seem to be satisfied by the definition of CLs. It remains true that both this and the more common (as associated with the Neyman-
Pearson Pearson may refer to: Organizations Education *Lester B. Pearson College, Victoria, British Columbia, Canada *Pearson College (UK), London, owned by Pearson PLC *Lester B. Pearson High School (disambiguation) Companies *Pearson PLC, a UK-based int ...
theory) versions of the confidence principle are incompatible with the likelihood principle, and therefore no frequentist method can be regarded as a truly complete solution to the problems raised by considering conditional properties of confidence intervals.


Calculation in the large sample limit

If certain regularity conditions are met, then a general likelihood function will become a Gaussian function in the large sample limit. In such case the CLs upper limit at confidence level 1-\alpha' (derived from the
uniformly most powerful test In statistical hypothesis testing, a uniformly most powerful (UMP) test is a hypothesis test which has the greatest power 1 - \beta among all possible tests of a given size ''α''. For example, according to the Neyman–Pearson lemma, the likelih ...
) is given by : \theta_ = \hat\theta + \sigma\Phi^(1 - \alpha'\Phi(\hat\theta / \sigma ) ) , where \Phi is the standard normal cumulative distribution, \hat\theta is the
maximum likelihood In statistics, maximum likelihood estimation (MLE) is a method of estimation theory, estimating the Statistical parameter, parameters of an assumed probability distribution, given some observed data. This is achieved by Mathematical optimization, ...
estimator of \theta and \sigma is its
standard deviation In statistics, the standard deviation is a measure of the amount of variation or dispersion of a set of values. A low standard deviation indicates that the values tend to be close to the mean (also called the expected value) of the set, while ...
; the latter might be estimated from the inverse of the Fisher information matrix or by using the "Asimov" data set. This result happens to be equivalent to a Bayesian
credible interval In Bayesian statistics, a credible interval is an interval within which an unobserved parameter value falls with a particular probability. It is an interval in the domain of a posterior probability distribution or a predictive distribution. The ...
if a uniform
prior Prior (or prioress) is an ecclesiastical title for a superior in some religious orders. The word is derived from the Latin for "earlier" or "first". Its earlier generic usage referred to any monastic superior. In abbeys, a prior would be l ...
for \theta is used.


References


Further reading

* * * {{cite arXiv , author=Robert D. Cousins , title=Negatively Biased Relevant Subsets Induced by the Most-Powerful One-Sided Upper Confidence Limits for a Bounded Physical Parameter, eprint=1109.2023 , year=2011 , class=physics.data-an


External links


The Particle Data Group (PDG) review of statistical methods
Statistical intervals Experimental particle physics