In
estimation theory
Estimation theory is a branch of statistics that deals with estimating the values of parameters based on measured empirical data that has a random component. The parameters describe an underlying physical setting in such a way that their value ...
and
statistics, the Cramér–Rao bound (CRB) expresses a lower bound on the
variance of
unbiased estimator
In statistics, the bias of an estimator (or bias function) is the difference between this estimator's expected value and the true value of the parameter being estimated. An estimator or decision rule with zero bias is called ''unbiased''. In stat ...
s of a deterministic (fixed, though unknown) parameter, the variance of any such estimator is at least as high as the inverse of the
Fisher information
In mathematical statistics, the Fisher information (sometimes simply called information) is a way of measuring the amount of information that an observable random variable ''X'' carries about an unknown parameter ''θ'' of a distribution that mode ...
. Equivalently, it expresses an upper bound on the
precision (the inverse of variance) of unbiased estimators: the precision of any such estimator is at most the Fisher information.
The result is named in honor of
Harald Cramér and
C. R. Rao
Calyampudi Radhakrishna Rao FRS (born 10 September 1920), commonly known as C. R. Rao, is an Indian-American mathematician and statistician. He is currently professor emeritus at Pennsylvania State University and Research Professor at the ...
,
but has independently also been derived by
Maurice Fréchet,
Georges Darmois
Georges Darmois (24 June 1888 – 3 January 1960) was a French mathematician and statistician. He pioneered in the theory of sufficiency, in stellar statistics, and in factor analysis. He was also one of the first French mathematicians to teach ...
,
as well as
Alexander Aitken
Alexander Craig "Alec" Aitken (1 April 1895 – 3 November 1967) was one of New Zealand's most eminent mathematicians. In a 1935 paper he introduced the concept of generalized least squares, along with now standard vector/matrix notation fo ...
and
Harold Silverstone.
An unbiased estimator that achieves this lower bound is said to be (fully) ''
efficient''. Such a solution achieves the lowest possible
mean squared error
In statistics, the mean squared error (MSE) or mean squared deviation (MSD) of an estimator (of a procedure for estimating an unobserved quantity) measures the average of the squares of the errors—that is, the average squared difference between ...
among all unbiased methods, and is therefore the
minimum variance unbiased (MVU) estimator. However, in some cases, no unbiased technique exists which achieves the bound. This may occur either if for any unbiased estimator, there exists another with a strictly smaller variance, or if an MVU estimator exists, but its variance is strictly greater than the inverse of the Fisher information.
The Cramér–Rao bound can also be used to bound the variance of estimators of given bias. In some cases, a biased approach can result in both a variance and a
mean squared error
In statistics, the mean squared error (MSE) or mean squared deviation (MSD) of an estimator (of a procedure for estimating an unobserved quantity) measures the average of the squares of the errors—that is, the average squared difference between ...
that are the unbiased Cramér–Rao lower bound; see
estimator bias.
Statement
The Cramér–Rao bound is stated in this section for several increasingly general cases, beginning with the case in which the parameter is a
scalar
Scalar may refer to:
*Scalar (mathematics), an element of a field, which is used to define a vector space, usually the field of real numbers
*Scalar (physics), a physical quantity that can be described by a single element of a number field such a ...
and its estimator is
unbiased
Bias is a disproportionate weight ''in favor of'' or ''against'' an idea or thing, usually in a way that is closed-minded, prejudicial, or unfair. Biases can be innate or learned. People may develop biases for or against an individual, a group, ...
. All versions of the bound require certain regularity conditions, which hold for most well-behaved distributions. These conditions are listed
later in this section.
Scalar unbiased case
Suppose
is an unknown deterministic parameter that is to be estimated from
independent observations (measurements) of
, each from a distribution according to some
probability density function . The
variance of any ''unbiased'' estimator
of
is then bounded by the
reciprocal
Reciprocal may refer to:
In mathematics
* Multiplicative inverse, in mathematics, the number 1/''x'', which multiplied by ''x'' gives the product 1, also known as a ''reciprocal''
* Reciprocal polynomial, a polynomial obtained from another pol ...
of the
Fisher information
In mathematical statistics, the Fisher information (sometimes simply called information) is a way of measuring the amount of information that an observable random variable ''X'' carries about an unknown parameter ''θ'' of a distribution that mode ...
:
:
where the Fisher information
is defined by
:
and
is the
natural logarithm of the
likelihood function for a single sample
and
denotes the
expected value
In probability theory, the expected value (also called expectation, expectancy, mathematical expectation, mean, average, or first moment) is a generalization of the weighted average. Informally, the expected value is the arithmetic mean of a ...
with respect to the density
of
. If
is twice differentiable and certain regularity conditions hold, then the Fisher information can also be defined as follows:
:
The
efficiency
Efficiency is the often measurable ability to avoid wasting materials, energy, efforts, money, and time in doing something or in producing a desired result. In a more general sense, it is the ability to do things well, successfully, and without ...
of an unbiased estimator
measures how close this estimator's variance comes to this lower bound; estimator efficiency is defined as
:
or the minimum possible variance for an unbiased estimator divided by its actual variance.
The Cramér–Rao lower bound thus gives
:
.
General scalar case
A more general form of the bound can be obtained by considering a biased estimator
, whose expectation is not
but a function of this parameter, say,
. Hence
is not generally equal to 0. In this case, the bound is given by
:
where
is the derivative of
(by
), and
is the Fisher information defined above.
Bound on the variance of biased estimators
Apart from being a bound on estimators of functions of the parameter, this approach can be used to derive a bound on the variance of biased estimators with a given bias, as follows. Consider an estimator
with bias
, and let
. By the result above, any unbiased estimator whose expectation is
has variance greater than or equal to
. Thus, any estimator
whose bias is given by a function
satisfies
:
The unbiased version of the bound is a special case of this result, with
.
It's trivial to have a small variance − an "estimator" that is constant has a variance of zero. But from the above equation we find that the
mean squared error
In statistics, the mean squared error (MSE) or mean squared deviation (MSD) of an estimator (of a procedure for estimating an unobserved quantity) measures the average of the squares of the errors—that is, the average squared difference between ...
of a biased estimator is bounded by
:
using the standard decomposition of the MSE. Note, however, that if
this bound might be less than the unbiased Cramér–Rao bound
. For instance, in the
example of estimating variance below,
.
Multivariate case
Extending the Cramér–Rao bound to multiple parameters, define a parameter column
vector
:
with probability density function
which satisfies the two
regularity conditions below.
The
Fisher information matrix
In mathematical statistics, the Fisher information (sometimes simply called information) is a way of measuring the amount of information that an observable random variable ''X'' carries about an unknown parameter ''θ'' of a distribution that model ...
is a
matrix with element
defined as
:
Let
be an estimator of any vector function of parameters,
, and denote its expectation vector