Estimation theory is a branch of
statistics
Statistics (from German: '' Statistik'', "description of a state, a country") is the discipline that concerns the collection, organization, analysis, interpretation, and presentation of data. In applying statistics to a scientific, indust ...
that deals with estimating the values of
parameters based on measured empirical data that has a random component. The parameters describe an underlying physical setting in such a way that their value affects the distribution of the measured data. An ''
estimator'' attempts to approximate the unknown parameters using the measurements.
In estimation theory, two approaches are generally considered:
[
]
* The probabilistic approach (described in this article) assumes that the measured data is random with
probability distribution
In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomenon ...
dependent on the parameters of interest
* The
set-membership approach assumes that the measured data vector belongs to a set which depends on the parameter vector.
Examples
For example, it is desired to estimate the proportion of a population of voters who will vote for a particular candidate. That proportion is the parameter sought; the estimate is based on a small random sample of voters. Alternatively, it is desired to estimate the probability of a voter voting for a particular candidate, based on some demographic features, such as age.
Or, for example, in
radar
Radar is a detection system that uses radio waves to determine the distance (''ranging''), angle, and radial velocity of objects relative to the site. It can be used to detect aircraft, Marine radar, ships, spacecraft, guided missiles, motor v ...
the aim is to find the range of objects (airplanes, boats, etc.) by analyzing the two-way transit timing of received echoes of transmitted pulses. Since the reflected pulses are unavoidably embedded in electrical noise, their measured values are randomly distributed, so that the transit time must be estimated.
As another example, in electrical communication theory, the measurements which contain information regarding the parameters of interest are often associated with a
noisy signal.
Basics
For a given model, several statistical "ingredients" are needed so the estimator can be implemented. The first is a
statistical sample – a set of data points taken from a
random vector (RV) of size ''N''. Put into a
vector,
:
Secondly, there are ''M'' parameters
:
whose values are to be estimated. Third, the continuous
probability density function
In probability theory, a probability density function (PDF), or density of a continuous random variable, is a function whose value at any given sample (or point) in the sample space (the set of possible values taken by the random variable) ca ...
(pdf) or its discrete counterpart, the
probability mass function
In probability and statistics, a probability mass function is a function that gives the probability that a discrete random variable is exactly equal to some value. Sometimes it is also known as the discrete density function. The probability mass ...
(pmf), of the underlying distribution that generated the data must be stated conditional on the values of the parameters:
:
It is also possible for the parameters themselves to have a probability distribution (e.g.,
Bayesian statistics
Bayesian statistics is a theory in the field of statistics based on the Bayesian interpretation of probability where probability expresses a ''degree of belief'' in an event. The degree of belief may be based on prior knowledge about the event, ...
). It is then necessary to define the
Bayesian probability
Bayesian probability is an interpretation of the concept of probability, in which, instead of frequency or propensity of some phenomenon, probability is interpreted as reasonable expectation representing a state of knowledge or as quantification ...
:
After the model is formed, the goal is to estimate the parameters, with the estimates commonly denoted
, where the "hat" indicates the estimate.
One common estimator is the
minimum mean squared error
In statistics and signal processing, a minimum mean square error (MMSE) estimator is an estimation method which minimizes the mean square error (MSE), which is a common measure of estimator quality, of the fitted values of a dependent variable. I ...
(MMSE) estimator, which utilizes the error between the estimated parameters and the actual value of the parameters
:
as the basis for optimality. This error term is then squared and the
expected value
In probability theory, the expected value (also called expectation, expectancy, mathematical expectation, mean, average, or first moment) is a generalization of the weighted average. Informally, the expected value is the arithmetic mean of a ...
of this squared value is minimized for the MMSE estimator.
Estimators
Commonly used estimators (estimation methods) and topics related to them include:
*
Maximum likelihood
In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed stat ...
estimators
*
Bayes estimator
In estimation theory and decision theory, a Bayes estimator or a Bayes action is an estimator or decision rule that minimizes the posterior expected value of a loss function (i.e., the posterior expected loss). Equivalently, it maximizes the ...
s
*
Method of moments estimators
*
Cramér–Rao bound
In estimation theory and statistics, the Cramér–Rao bound (CRB) expresses a lower bound on the variance of unbiased estimators of a deterministic (fixed, though unknown) parameter, the variance of any such estimator is at least as high as the ...
*
Least squares
The method of least squares is a standard approach in regression analysis to approximate the solution of overdetermined systems (sets of equations in which there are more equations than unknowns) by minimizing the sum of the squares of the re ...
*
Minimum mean squared error
In statistics and signal processing, a minimum mean square error (MMSE) estimator is an estimation method which minimizes the mean square error (MSE), which is a common measure of estimator quality, of the fitted values of a dependent variable. I ...
(MMSE), also known as Bayes least squared error (BLSE)
*
Maximum a posteriori (MAP)
*
Minimum variance unbiased estimator (MVUE)
*
Nonlinear system identification
*
Best linear unbiased estimator (BLUE)
*Unbiased estimators — see
estimator bias.
*
Particle filter
Particle filters, or sequential Monte Carlo methods, are a set of Monte Carlo algorithms used to solve filtering problems arising in signal processing and Bayesian statistical inference. The filtering problem consists of estimating the inte ...
*
Markov chain Monte Carlo
In statistics, Markov chain Monte Carlo (MCMC) methods comprise a class of algorithms for sampling from a probability distribution. By constructing a Markov chain that has the desired distribution as its equilibrium distribution, one can obtain ...
(MCMC)
*
Kalman filter, and its various derivatives
*
Wiener filter
Examples
Unknown constant in additive white Gaussian noise
Consider a received
discrete signal,