Efficient estimator

In statistics, efficiency is a measure of quality of an estimator, of an experimental design, or of a hypothesis testing procedure. Essentially, a more efficient estimator, needs fewer input data or observations than a less efficient one to achieve the Cramér–Rao bound.
An ''efficient estimator'' is characterized by having the smallest possible variance, indicating that there is a small deviance between the estimated value and the "true" value in the L2 norm sense.

The relative efficiency of two procedures is the ratio of their efficiencies, although often this concept is used where the comparison is made between a given procedure and a notional "best possible" procedure. The efficiencies and the relative efficiency of two procedures theoretically depend on the sample size available for the given procedure, but it is often possible to use the asymptotic relative efficiency (defined as the limit of the relative efficiencies as the sample size grows) as the principal comparison measure.

Estimators

The efficiency of an unbiased estimator, ''T'', of a parameter ''θ'' is defined as

:$e(T) = \frac$

where $\mathcal(\theta)$ is the Fisher information of the sample. Thus ''e''(''T'') is the minimum possible variance for an unbiased estimator divided by its actual variance. The Cramér–Rao bound can be used to prove that ''e''(''T'') ≤ 1.

Efficient estimators

An efficient estimator is an estimator that estimates the quantity of interest in some “best possible” manner. The notion of “best possible” relies upon the choice of a particular loss function — the function which quantifies the relative degree of undesirability of estimation errors of different magnitudes. The most common choice of the loss function is quadratic, resulting in the mean squared error criterion of optimality.

In general, the spread of an estimator around the parameter θ is a measure of estimator efficiency and performance. This performance can be calculated by finding the mean squared error. More formally, let ''T'' be an estimator for the parameter ''θ''. The mean squared error of ''T'' is the value \operatorname(T)=E T-\theta)^2/math>, which can be decomposed as a sum of its variance and bias:

:
\begin
\operatorname(T) & = \operatorname E T-\theta)^2\operatorname E T-\operatorname E[T\operatorname E[T">.html" ;"title="T-\operatorname E[T">T-\operatorname E[T\operatorname E[T\theta)^2] \\[5pt]
& =\operatorname E[(T-\operatorname E[T])^2]+2E[T-E[T(\operatorname E \theta)+(\operatorname E \theta)^2 \\[5pt]
& =\operatorname(T)+(\operatorname E \theta)^2
\end

An estimator ''T''₁ performs better than an estimator ''T''₂ if $\operatorname(T_1) < \operatorname(T_2)$. For a more specific case, if ''T''₁ and ''T₂ ''are two unbiased estimators for the same parameter θ, then the variance can be compared to determine performance. In this case, ''T''₂ is ''more efficient'' than ''T''₁ if the variance of ''T''₂ is ''smaller'' than the variance of ''T''₁, i.e. $\operatorname(T_1)>\operatorname(T_2)$ for all values of ''θ''. This relationship can be determined by simplifying the more general case above for mean squared error; since the expected value of an unbiased estimator is equal to the parameter value, \operatorname E \theta. Therefore, for an unbiased estimator, $\operatorname(T)=\operatorname(T)$, as the (\operatorname E \theta)^2 term drops out for being equal to 0.

If an unbiased estimator of a parameter ''θ'' attains $e(T) = 1$ for all values of the parameter, then the estimator is called efficient.

Equivalently, the estimator achieves equality in the Cramér–Rao inequality for all ''θ''. The Cramér–Rao lower bound is a lower bound of the variance of an unbiased estimator, representing the "best" an unbiased estimator can be.

An efficient estimator is also the minimum variance unbiased estimator (MVUE). This is because an efficient estimator maintains equality on the Cramér–Rao inequality for all parameter values, which means it attains the minimum variance for all parameters (the definition of the MVUE). The MVUE estimator, even if it exists, is not necessarily efficient, because "minimum" does not mean equality holds on the Cramér–Rao inequality.

Thus an efficient estimator need not exist, but if it does, it is the MVUE.

Finite-sample efficiency

Suppose is a parametric model and are the data sampled from this model. Let be an estimator for the parameter ''θ''. If this estimator is unbiased (that is, ), then the Cramér–Rao inequality states the variance of this estimator is bounded from below:
:
\operatorname ,T\, \geq\ \mathcal_\theta^,

where $\scriptstyle\mathcal_\theta$ is the Fisher information matrix of the model at point ''θ''. Generally, the variance measures the degree of dispersion of a random variable around its mean. Thus estimators with small variances are more concentrated, they estimate the parameters more precisely. We say that the estimator is a finite-sample efficient estimator (in the class of unbiased estimators) if it reaches the lower bound in the Cramér–Rao inequality above, for all . Efficient estimators are always minimum variance unbiased estimators. However the converse is false: There exist point-estimation problems for which the minimum-variance mean-unbiased estimator is inefficient.

Historically, finite-sample efficiency was an early optimality criterion. However this criterion has some limitations:
* Finite-sample efficient estimators are extremely rare. In fact, it was proved that efficient estimation is possible only in an exponential family, and only for the natural parameters of that family.
* This notion of efficiency is sometimes restricted to the class of unbiased estimators. (Often it isn't.) Since there are no good theoretical reasons to require that estimators are unbiased, this restriction is inconvenient. In fact, if we use mean squared error as a selection criterion, many biased estimators will slightly outperform the “best” unbiased ones. For example, in multivariate statistics for dimension three or more, the mean-unbiased estimator, sample mean, is inadmissible: Regardless of the outcome, its performance is worse than for example the James–Stein estimator.
* Finite-sample efficiency is based on the variance, as a criterion according to which the estimators are judged. A more general approach is to use loss functions other than quadratic ones, in which case the finite-sample efficiency can no longer be formulated.

As an example, among the models encountered in practice, efficient estimators exist for: the mean ''μ'' of the normal distribution (but not the variance ''σ''²), parameter ''λ'' of the Poisson distribution, the probability ''p'' in the binomial or multinomial distribution.

Consider the model of a normal distribution with unknown mean but known variance: The data consists of ''n'' independent and identically distributed observations from this model: . We estimate the parameter ''θ'' using the sample mean of all observations:
: $T(X) = \frac1n \sum_^n x_i\ .$
This estimator has mean ''θ'' and variance of , which is equal to the reciprocal of the Fisher information from the sample. Thus, the sample mean is a finite-sample efficient estimator for the mean of the normal distribution.

Asymptotic efficiency

Asymptotic efficiency requires Consistency (statistics), asymptotic normally distribution of estimator, and asymptotic variance-covariance matrix no worse than any other estimator.

Example: Median

Consider a sample of size $N$ drawn from a normal distribution of mean $\mu$ and unit variance, i.e., $X_n \sim \mathcal(\mu, 1).$

The sample mean, $\overline$, of the sample $X_1, X_2, \ldots, X_N$, defined as

:$\overline = \frac \sum_^ X_n \sim \mathcal\left(\mu, \frac\right).$

The variance of the mean, 1/''N'' (the square of the standard error) is equal to the reciprocal of the Fisher information from the sample and thus, by the Cramér–Rao inequality, the sample mean is efficient in the sense that its efficiency is unity (100%).

Now consider the sample median, $\widetilde$. This is an unbiased and consistent estimator for $\mu$. For large $N$ the sample median is approximately normally distributed with mean $\mu$ and variance $/,$

:$\widetilde \sim \mathcal \left(\mu, \frac \pi \right).$

The efficiency of the median for large $N$ is thus

: $e\left(\widetilde\right) = \left(\frac 1 N\right) \left(\frac \pi \right)^ = 2/\pi \approx 0.64.$

In other words, the relative variance of the median will be $\pi/2 \approx 1.57$, or 57% greater than the variance of the mean – the standard error of the median will be 25% greater than that of the mean.

Note that this is the asymptotic efficiency — that is, the efficiency in the limit as sample size $N$ tends to infinity. For finite values of $N,$ the efficiency is higher than this (for example, a sample size of 3 gives an efficiency of about 74%).

The sample mean is thus more efficient than the sample median in this example. However, there may be measures by which the median performs better. For example, the median is far more robust to outliers, so that if the Gaussian model is questionable or approximate, there may advantages to using the median (see Robust statistics).

Dominant estimators

If $T_1$ and $T_2$ are estimators for the parameter $\theta$, then $T_1$ is said to dominate $T_2$ if:
# its mean squared error (MSE) is smaller for at least some value of $\theta$
# the MSE does not exceed that of $T_2$ for any value of θ.

Formally, $T_1$ dominates $T_2$ if
:
\operatorname
(T_1 - \theta)^2 \leq
\operatorname