HOME

TheInfoList



OR:

In
estimation theory Estimation theory is a branch of statistics that deals with estimating the values of parameters based on measured empirical data that has a random component. The parameters describe an underlying physical setting in such a way that their value ...
and
decision theory Decision theory (or the theory of choice; not to be confused with choice theory) is a branch of applied probability theory concerned with the theory of making decisions based on assigning probabilities to various factors and assigning numerical ...
, a Bayes estimator or a Bayes action is an
estimator In statistics, an estimator is a rule for calculating an estimate of a given quantity based on observed data: thus the rule (the estimator), the quantity of interest (the estimand) and its result (the estimate) are distinguished. For example, the ...
or
decision rule In decision theory, a decision rule is a function which maps an observation to an appropriate action. Decision rules play an important role in the theory of statistics and economics, and are closely related to the concept of a strategy in game the ...
that minimizes the posterior
expected value In probability theory, the expected value (also called expectation, expectancy, mathematical expectation, mean, average, or first moment) is a generalization of the weighted average. Informally, the expected value is the arithmetic mean of a ...
of a loss function (i.e., the posterior expected loss). Equivalently, it maximizes the posterior expectation of a
utility As a topic of economics, utility is used to model worth or value. Its usage has evolved significantly over time. The term was introduced initially as a measure of pleasure or happiness as part of the theory of utilitarianism by moral philosophe ...
function. An alternative way of formulating an estimator within
Bayesian statistics Bayesian statistics is a theory in the field of statistics based on the Bayesian interpretation of probability where probability expresses a ''degree of belief'' in an event. The degree of belief may be based on prior knowledge about the event, ...
is
maximum a posteriori estimation In Bayesian statistics, a maximum a posteriori probability (MAP) estimate is an estimate of an unknown quantity, that equals the mode of the posterior distribution. The MAP can be used to obtain a point estimate of an unobserved quantity on the ...
.


Definition

Suppose an unknown parameter \theta is known to have a
prior distribution In Bayesian statistical inference, a prior probability distribution, often simply called the prior, of an uncertain quantity is the probability distribution that would express one's beliefs about this quantity before some evidence is taken into ...
\pi. Let \widehat = \widehat(x) be an estimator of \theta (based on some measurements ''x''), and let L(\theta,\widehat) be a loss function, such as squared error. The Bayes risk of \widehat is defined as E_\pi(L(\theta, \widehat)), where the expectation is taken over the probability distribution of \theta: this defines the risk function as a function of \widehat. An estimator \widehat is said to be a ''Bayes estimator'' if it minimizes the Bayes risk among all estimators. Equivalently, the estimator which minimizes the posterior expected loss E(L(\theta,\widehat) , x) ''for each x '' also minimizes the Bayes risk and therefore is a Bayes estimator. If the prior is improper then an estimator which minimizes the posterior expected loss ''for each x'' is called a generalized Bayes estimator.Lehmann and Casella, Definition 4.2.9


Examples


Minimum mean square error estimation

The most common risk function used for Bayesian estimation is the mean square error (MSE), also called ''squared error risk''. The MSE is defined by :\mathrm = E\left (\widehat(x) - \theta)^2 \right where the expectation is taken over the joint distribution of \theta and x.


Posterior mean

Using the MSE as risk, the Bayes estimate of the unknown parameter is simply the mean of the posterior distribution, :\widehat(x) = E x\int \theta\, p(\theta , x)\,d\theta. This is known as the ''minimum mean square error'' (MMSE) estimator.


Bayes estimators for conjugate priors

If there is no inherent reason to prefer one prior probability distribution over another, a
conjugate prior In Bayesian probability theory, if the posterior distribution p(\theta \mid x) is in the same probability distribution family as the prior probability distribution p(\theta), the prior and posterior are then called conjugate distributions, and ...
is sometimes chosen for simplicity. A conjugate prior is defined as a prior distribution belonging to some
parametric family In mathematics and its applications, a parametric family or a parameterized family is a family of objects (a set of related objects) whose differences depend only on the chosen values for a set of parameters. Common examples are parametrized (fa ...
, for which the resulting posterior distribution also belongs to the same family. This is an important property, since the Bayes estimator, as well as its statistical properties (variance, confidence interval, etc.), can all be derived from the posterior distribution. Conjugate priors are especially useful for sequential estimation, where the posterior of the current measurement is used as the prior in the next measurement. In sequential estimation, unless a conjugate prior is used, the posterior distribution typically becomes more complex with each added measurement, and the Bayes estimator cannot usually be calculated without resorting to numerical methods. Following are some examples of conjugate priors. * If x, \theta is Normal, x, \theta \sim N(\theta,\sigma^2), and the prior is normal, \theta \sim N(\mu,\tau^2), then the posterior is also Normal and the Bayes estimator under MSE is given by :\widehat(x)=\frac\mu+\fracx. * If x_1, ..., x_n are iid Poisson random variables x_i, \theta \sim P(\theta), and if the prior is Gamma distributed \theta \sim G(a,b), then the posterior is also Gamma distributed, and the Bayes estimator under MSE is given by :\widehat(X)=\frac. * If x_1, ..., x_n are iid uniformly distributed x_i, \theta \sim U(0,\theta), and if the prior is Pareto distributed \theta \sim Pa(\theta_0,a), then the posterior is also Pareto distributed, and the Bayes estimator under MSE is given by :\widehat(X)=\frac.


Alternative risk functions

Risk functions are chosen depending on how one measures the distance between the estimate and the unknown parameter. The MSE is the most common risk function in use, primarily due to its simplicity. However, alternative risk functions are also occasionally used. The following are several examples of such alternatives. We denote the posterior generalized distribution function by F.


Posterior median and other quantiles

* A "linear" loss function, with a>0 , which yields the posterior median as the Bayes' estimate: : L(\theta,\widehat) = a, \theta-\widehat, : F(\widehat(x), X) = \tfrac. * Another "linear" loss function, which assigns different "weights" a,b>0 to over or sub estimation. It yields a
quantile In statistics and probability, quantiles are cut points dividing the range of a probability distribution into continuous intervals with equal probabilities, or dividing the observations in a sample in the same way. There is one fewer quantile th ...
from the posterior distribution, and is a generalization of the previous loss function: : L(\theta,\widehat) = \begin a, \theta-\widehat, , & \mbox\theta-\widehat \ge 0 \\ b, \theta-\widehat, , & \mbox\theta-\widehat < 0 \end : F(\widehat(x), X) = \frac.


Posterior mode

* The following loss function is trickier: it yields either the posterior mode, or a point close to it depending on the curvature and properties of the posterior distribution. Small values of the parameter K>0 are recommended, in order to use the mode as an approximation ( L>0 ): : L(\theta,\widehat) = \begin 0, & \mbox, \theta-\widehat, < K \\ L, & \mbox, \theta-\widehat, \ge K. \end Other loss functions can be conceived, although the
mean squared error In statistics, the mean squared error (MSE) or mean squared deviation (MSD) of an estimator (of a procedure for estimating an unobserved quantity) measures the average of the squares of the errors—that is, the average squared difference between ...
is the most widely used and validated. Other loss functions are used in statistics, particularly in
robust statistics Robust statistics are statistics with good performance for data drawn from a wide range of probability distributions, especially for distributions that are not normal. Robust statistical methods have been developed for many common problems, such ...
.


Generalized Bayes estimators

The prior distribution p has thus far been assumed to be a true probability distribution, in that :\int p(\theta) d\theta = 1. However, occasionally this can be a restrictive requirement. For example, there is no distribution (covering the set, R, of all real numbers) for which every real number is equally likely. Yet, in some sense, such a "distribution" seems like a natural choice for a non-informative prior, i.e., a prior distribution which does not imply a preference for any particular value of the unknown parameter. One can still define a function p(\theta) = 1, but this would not be a proper probability distribution since it has infinite mass, :\int=\infty. Such measures p(\theta), which are not probability distributions, are referred to as
improper prior In Bayesian statistical inference, a prior probability distribution, often simply called the prior, of an uncertain quantity is the probability distribution that would express one's beliefs about this quantity before some evidence is taken into ...
s. The use of an improper prior means that the Bayes risk is undefined (since the prior is not a probability distribution and we cannot take an expectation under it). As a consequence, it is no longer meaningful to speak of a Bayes estimator that minimizes the Bayes risk. Nevertheless, in many cases, one can define the posterior distribution :p(\theta, x) = \frac. This is a definition, and not an application of
Bayes' theorem In probability theory and statistics, Bayes' theorem (alternatively Bayes' law or Bayes' rule), named after Thomas Bayes, describes the probability of an event, based on prior knowledge of conditions that might be related to the event. For examp ...
, since Bayes' theorem can only be applied when all distributions are proper. However, it is not uncommon for the resulting "posterior" to be a valid probability distribution. In this case, the posterior expected loss : \int is typically well-defined and finite. Recall that, for a proper prior, the Bayes estimator minimizes the posterior expected loss. When the prior is improper, an estimator which minimizes the posterior expected loss is referred to as a generalized Bayes estimator.


Example

A typical example is estimation of a location parameter with a loss function of the type L(a-\theta). Here \theta is a location parameter, i.e., p(x, \theta) = f(x-\theta). It is common to use the improper prior p(\theta)=1 in this case, especially when no other more subjective information is available. This yields :p(\theta, x) = \frac = \frac so the posterior expected loss :E x= \int = \frac \int L(a-\theta) f(x-\theta) d\theta. The generalized Bayes estimator is the value a(x) that minimizes this expression for a given x. This is equivalent to minimizing :\int L(a-\theta) f(x-\theta) d\theta for a given x.        (1) In this case it can be shown that the generalized Bayes estimator has the form x+a_0, for some constant a_0. To see this, let a_0 be the value minimizing (1) when x=0. Then, given a different value x_1, we must minimize :\int L(a-\theta) f(x_1-\theta) d\theta = \int L(a-x_1-\theta') f(-\theta') d\theta'.        (2) This is identical to (1), except that a has been replaced by a-x_1. Thus, the expression minimizing is given by a-x_1 = a_0, so that the optimal estimator has the form :a(x) = a_0 + x.\,\!


Empirical Bayes estimators

A Bayes estimator derived through the
empirical Bayes method Empirical Bayes methods are procedures for statistical inference in which the prior probability distribution is estimated from the data. This approach stands in contrast to standard Bayesian methods, for which the prior distribution is fixed b ...
is called an empirical Bayes estimator. Empirical Bayes methods enable the use of auxiliary empirical data, from observations of related parameters, in the development of a Bayes estimator. This is done under the assumption that the estimated parameters are obtained from a common prior. For example, if independent observations of different parameters are performed, then the estimation performance of a particular parameter can sometimes be improved by using data from other observations. There are parametric and
non-parametric Nonparametric statistics is the branch of statistics that is not based solely on parametrized families of probability distributions (common examples of parameters are the mean and variance). Nonparametric statistics is based on either being distri ...
approaches to empirical Bayes estimation. Parametric empirical Bayes is usually preferable since it is more applicable and more accurate on small amounts of data.


Example

The following is a simple example of parametric empirical Bayes estimation. Given past observations x_1,\ldots,x_n having conditional distribution f(x_i, \theta_i), one is interested in estimating \theta_ based on x_. Assume that the \theta_i's have a common prior \pi which depends on unknown parameters. For example, suppose that \pi is normal with unknown mean \mu_\pi\,\! and variance \sigma_\pi\,\!. We can then use the past observations to determine the mean and variance of \pi in the following way. First, we estimate the mean \mu_m\,\! and variance \sigma_m\,\! of the marginal distribution of x_1, \ldots, x_n using the
maximum likelihood In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statis ...
approach: :\widehat_m=\frac\sum, :\widehat_m^=\frac\sum. Next, we use the law of total expectation to compute \mu_m and the law of total variance to compute \sigma_m^ such that : \mu_m=E_\pi mu_f(\theta)\,\!, : \sigma_m^=E_\pi sigma_f^(\theta)E_\pi \mu_f(\theta)-\mu_m)^ where \mu_f(\theta) and \sigma_f(\theta) are the moments of the conditional distribution f(x_i, \theta_i), which are assumed to be known. In particular, suppose that \mu_f(\theta) = \theta and that \sigma_f^(\theta) = K; we then have : \mu_\pi=\mu_m \,\!, : \sigma_\pi^=\sigma_m^-\sigma_f^=\sigma_m^-K . Finally, we obtain the estimated moments of the prior, : \widehat_\pi=\widehat_m, : \widehat_\pi^=\widehat_m^-K. For example, if x_i, \theta_i \sim N(\theta_i,1), and if we assume a normal prior (which is a conjugate prior in this case), we conclude that \theta_\sim N(\widehat_\pi,\widehat_\pi^) , from which the Bayes estimator of \theta_ based on x_ can be calculated.


Properties


Admissibility

Bayes rules having finite Bayes risk are typically admissible. The following are some specific examples of admissibility theorems. * If a Bayes rule is unique then it is admissible. For example, as stated above, under mean squared error (MSE) the Bayes rule is unique and therefore admissible. * If θ belongs to a
discrete set ] In mathematics, a point ''x'' is called an isolated point of a subset ''S'' (in a topological space ''X'') if ''x'' is an element of ''S'' and there exists a neighborhood of ''x'' which does not contain any other points of ''S''. This is equiva ...
, then all Bayes rules are admissible. * If θ belongs to a continuous (non-discrete) set, and if the risk function R(θ,δ) is continuous in θ for every δ, then all Bayes rules are admissible. By contrast, generalized Bayes rules often have undefined Bayes risk in the case of improper priors. These rules are often inadmissible and the verification of their admissibility can be difficult. For example, the generalized Bayes estimator of a location parameter θ based on Gaussian samples (described in the "Generalized Bayes estimator" section above) is inadmissible for p>2; this is known as Stein's phenomenon.


Asymptotic efficiency

Let θ be an unknown random variable, and suppose that x_1,x_2,\ldots are iid samples with density f(x_i, \theta). Let \delta_n = \delta_n(x_1,\ldots,x_n) be a sequence of Bayes estimators of θ based on an increasing number of measurements. We are interested in analyzing the asymptotic performance of this sequence of estimators, i.e., the performance of \delta_n for large ''n''. To this end, it is customary to regard θ as a deterministic parameter whose true value is \theta_0. Under specific conditions, for large samples (large values of ''n''), the posterior density of θ is approximately normal. In other words, for large ''n'', the effect of the prior probability on the posterior is negligible. Moreover, if δ is the Bayes estimator under MSE risk, then it is
asymptotically unbiased In analytic geometry, an asymptote () of a curve is a line such that the distance between the curve and the line approaches zero as one or both of the ''x'' or ''y'' coordinates tends to infinity. In projective geometry and related contexts, ...
and it
converges in distribution In probability theory, there exist several different notions of convergence of random variables. The convergence of sequences of random variables to some limit random variable is an important concept in probability theory, and its applications to ...
to the
normal distribution In statistics, a normal distribution or Gaussian distribution is a type of continuous probability distribution for a real-valued random variable. The general form of its probability density function is : f(x) = \frac e^ The parameter \mu i ...
: : \sqrt(\delta_n - \theta_0) \to N\left(0 , \frac\right), where ''I''(θ0) is the
fisher information In mathematical statistics, the Fisher information (sometimes simply called information) is a way of measuring the amount of information that an observable random variable ''X'' carries about an unknown parameter ''θ'' of a distribution that mode ...
of θ0. It follows that the Bayes estimator δ''n'' under MSE is asymptotically efficient. Another estimator which is asymptotically normal and efficient is the maximum likelihood estimator (MLE). The relations between the maximum likelihood and Bayes estimators can be shown in the following simple example.


Example: estimating ''p'' in a binomial distribution

Consider the estimator of θ based on binomial sample ''x''~b(θ,''n'') where θ denotes the probability for success. Assuming θ is distributed according to the conjugate prior, which in this case is the Beta distribution B(''a'',''b''), the posterior distribution is known to be B(a+x,b+n-x). Thus, the Bayes estimator under MSE is : \delta_n(x)=E x\frac. The MLE in this case is x/n and so we get, : \delta_n(x)=\fracE
theta Theta (, ; uppercase: Θ or ; lowercase: θ or ; grc, ''thē̂ta'' ; Modern: ''thī́ta'' ) is the eighth letter of the Greek alphabet, derived from the Phoenician letter Teth . In the system of Greek numerals, it has a value of 9. G ...
\frac\delta_. The last equation implies that, for ''n'' → ∞, the Bayes estimator (in the described problem) is close to the MLE. On the other hand, when ''n'' is small, the prior information is still relevant to the decision problem and affects the estimate. To see the relative weight of the prior information, assume that ''a''=''b''; in this case each measurement brings in 1 new bit of information; the formula above shows that the prior information has the same weight as ''a+b'' bits of the new information. In applications, one often knows very little about fine details of the prior distribution; in particular, there is no reason to assume that it coincides with B(''a'',''b'') exactly. In such a case, one possible interpretation of this calculation is: "there is a non-pathological prior distribution with the mean value 0.5 and the standard deviation ''d'' which gives the weight of prior information equal to 1/(4''d''2)-1 bits of new information." Another example of the same phenomena is the case when the prior estimate and a measurement are normally distributed. If the prior is centered at ''B'' with deviation Σ, and the measurement is centered at ''b'' with deviation σ,
then Then may refer to: * Then language The Then language (also known as Yánghuáng 佯僙语 in Chinese; alternate spellings: Tʻen and Ten) is a Kam–Sui language spoken in Pingtang County, southern Guizhou. It is spoken by the Yanghuang 佯 ...
the posterior is centered at \fracB+\fracb, with weights in this weighted average being α=σ², β=Σ². Moreover, the squared posterior deviation is Σ²+σ². In other words, the prior is combined with the measurement in ''exactly'' the same way as if it were an extra measurement to take into account. For example, if Σ=σ/2, then the deviation of 4 measurements combined matches the deviation of the prior (assuming that errors of measurements are independent). And the weights α,β in the formula for posterior match this: the weight of the prior is 4 times the weight of the measurement. Combining this prior with ''n'' measurements with average ''v'' results in the posterior centered at \fracV+\fracv; in particular, the prior plays the same role as 4 measurements made in advance. In general, the prior has the weight of (σ/Σ)² measurements. Compare to the example of binomial distribution: there the prior has the weight of (σ/Σ)²−1 measurements. One can see that the exact weight does depend on the details of the distribution, but when σ≫Σ, the difference becomes small.


Practical example of Bayes estimators

The Internet Movie Database uses a formula for calculating and comparing the ratings of films by its users, including their Top Rated 250 Titles which is claimed to give "a true Bayesian estimate".IMDb Top 250
/ref> The following Bayesian formula was initially used to calculate a weighted average score for the Top 250, though the formula has since changed: :W = \ where: :W\ = weighted rating :R\ = average rating for the movie as a number from 1 to 10 (mean) = (Rating) :v\ = number of votes/ratings for the movie = (votes) :m\ = weight given to the prior estimate (in this case, the number of votes IMDB deemed necessary for average rating to approach statistical validity) :C\ = the mean vote across the whole pool (currently 7.0) Note that ''W'' is just the weighted arithmetic mean of ''R'' and ''C'' with weight vector ''(v, m)''. As the number of ratings surpasses ''m'', the confidence of the average rating surpasses the confidence of the mean vote for all films (C), and the weighted bayesian rating (W) approaches a straight average (R). The closer ''v'' (the number of ratings for the film) is to zero, the closer ''W'' is to ''C'', where W is the weighted rating and C is the average rating of all films. So, in simpler terms, the fewer ratings/votes cast for a film, the more that film's Weighted Rating will skew towards the average across all films, while films with many ratings/votes will have a rating approaching its pure arithmetic average rating. IMDb's approach ensures that a film with only a few ratings, all at 10, would not rank above "the Godfather", for example, with a 9.2 average from over 500,000 ratings.


See also

*
Recursive Bayesian estimation In probability theory, statistics, and machine learning, recursive Bayesian estimation, also known as a Bayes filter, is a general probabilistic approach for estimating an unknown probability density function (PDF) recursively over time using inco ...
* Generalized expected utility


Notes


References

* *


External links


Bayesian estimation on cnx.org
* {{DEFAULTSORT:Bayes Estimator Estimator