Law Of Total Variance

The law of total variance is a fundamental result in probability theory that expresses the variance of a random variable in terms of its conditional variances and conditional means given another random variable . Informally, it states that the overall variability of can be split into an “unexplained” component (the average of within-group variances) and an “explained” component (the variance of group means).

Formally, if and are random variables on the same probability space, and has finite variance, then:

\operatorname(Y) \;=\; \operatorname\bigl operatorname(Y \mid X)\bigr\;+\; \operatorname\!\bigl(\operatorname \mid Xbigr).\!

This identity is also known as the variance decomposition formula, the conditional variance formula, the law of iterated variances, or colloquially as Eve’s law, in parallel to the “Adam’s law” naming for the law of total expectation.

In actuarial science (particularly in credibility theory), the two terms \operatorname operatorname(Y \mid X)/math> and \operatorname(\operatorname \mid X are called the expected value of the process variance (EVPV) and the variance of the hypothetical means (VHM) respectively.

Explanation

Let be a random variable and another random variable on the same probability space. The law of total variance can be understood by noting:

# $\operatorname(Y \mid X)$ measures how much varies around its conditional mean \operatorname \mid X
# Taking the expectation of this conditional variance across all values of gives \operatorname operatorname(Y \mid X)/math>, often termed the “unexplained” or within-group part.
# The variance of the conditional mean, \operatorname(\operatorname \mid X, measures how much these conditional means differ (i.e. the “explained” or between-group part).

Adding these components yields the total variance $\operatorname(Y)$, mirroring how analysis of variance partitions variation.

Examples

Example 1 (Exam Scores)

Suppose five students take an exam scored 0–100. Let = student’s score and indicate whether the student is *international* or *domestic*:

* Mean and variance for international: \operatorname \mid X=\text= 50,\; \operatorname(Y\mid X=\text) \approx 1266.7.
* Mean and variance for domestic: \operatorname \mid X=\text= 50,\; \operatorname(Y\mid X=\text) = 100.

Both groups share the same mean (50), so the explained variance \operatorname(\operatorname \mid X is 0, and the total variance equals the average of the within-group variances (weighted by group size), i.e. 800.

Example 2 (Mixture of Two Gaussians)

Let be a coin flip taking values with probability and with probability . Given Heads, ~ Normal($\mu_h,\sigma_h^2$); given Tails, ~ Normal($\mu_t,\sigma_t^2$). Then
\operatorname operatorname(Y\mid X)= h\,\sigma_h^2 + (1 - h)\,\sigma_t^2,
\operatorname(\operatorname \mid X = h\,(1 - h)\,(\mu_h - \mu_t)^2,
so
$\operatorname(Y) = h\,\sigma_h^2 + (1 - h)\,\sigma_t^2 \;+\; h\,(1 - h)\,(\mu_h-\mu_t)^2.$

Example 3 (Dice and Coins)

Consider a two-stage experiment:
# Roll a fair die (values 1–6) to choose one of six biased coins.
# Flip that chosen coin; let =1 if Heads, 0 if Tails.

Then \operatorname \mid X=i= p_i, \; \operatorname(Y\mid X=i)=p_i(1-p_i). The overall variance of becomes
\operatorname(Y) = \operatorname\bigl _X(1 - p_X)\bigr+ \operatorname\bigl(p_X\bigr),
with $p_X$ uniform on $\.$

Proof

Discrete/Finite Proof

Let $(X_i,Y_i)$, $i=1,\ldots,n$, be observed pairs. Define \overline = \operatorname Then
\operatorname(Y)
= \frac\sum_^n \bigl(Y_i - \overline\bigr)^2
= \frac\sum_^n \Bigl Y_i - \overline_) + (\overline_ - \overline)\Bigr2,
where \overline_=\operatorname \mid X=X_i Expanding the square and noting the cross term cancels in summation yields:
\operatorname(Y)
= \operatorname\bigl operatorname(Y\mid X)\bigr\;+\; \operatorname\!\bigl(\operatorname \mid Xbigr).\!

General Case

Using \operatorname(Y) = \operatorname ^2- \operatorname 2 and the law of total expectation:
\operatorname ^2= \operatorname\bigl operatorname(Y^2 \mid X)\bigr
= \operatorname\bigl operatorname(Y\mid X) + \operatorname[Y\mid X2\bigr">\mid_X.html" ;"title="operatorname(Y\mid X) + \operatorname[Y\mid X">operatorname(Y\mid X) + \operatorname[Y\mid X2\bigr
Subtract \operatorname 2 = \bigl(\operatorname[\operatorname(Y\mid X)]\bigr)^2 and regroup to arrive at
\operatorname(Y) = \operatorname\bigl operatorname(Y\mid X)\bigr+ \operatorname\!\bigl(\operatorname \mid Xbigr).\!

Applications

Analysis of Variance (ANOVA)

In a one-way analysis of variance, the total sum of squares (proportional to $\operatorname(Y)$) is split into a “between-group” sum of squares (\operatorname(\operatorname \mid X) plus a “within-group” sum of squares (\operatorname operatorname(Y\mid X)/math>). The F-test examines whether the explained component is sufficiently large to indicate has a significant effect on .

Regression and R²

In linear regression and related models, if \hat=\operatorname \mid X the fraction of variance explained is
$R^2 = \frac = \frac = 1 - \frac.$
In the simple linear case (one predictor), $R^2$ also equals the square of the Pearson correlation coefficient between and .

Machine Learning and Bayesian Inference

In many Bayesian and ensemble methods, one decomposes prediction uncertainty via the law of total variance. For a Bayesian neural network with random parameters $\theta$:
\operatorname(Y)=\operatorname\bigl operatorname(Y\mid \theta)\bigr+ \operatorname\bigl(\operatorname \mid \thetabigr),
often referred to as “aleatoric” (within-model) vs. “epistemic” (between-model) uncertainty.

Actuarial Science

Credibility theory uses the same partitioning: the expected value of process variance (EVPV), \operatorname operatorname(Y\mid X) and the variance of hypothetical means (VHM), \operatorname(\operatorname \mid X. The ratio of explained to total variance determines how much “credibility” to give to individual risk classifications.

Information Theory

For jointly Gaussian $(X,Y)$, the fraction \operatorname(\operatorname \mid X/\operatorname(Y) relates directly to the mutual information $I(Y;X).$C. G. Bowsher & P. S. Swain (2012). "Identifying sources of variation and the flow of information in biochemical networks," ''PNAS'' 109 (20): E1320–E1328. In non-Gaussian settings, a high explained-variance ratio still indicates significant information about contained in .

Generalizations

The law of total variance generalizes to multiple or nested conditionings. For example, with two conditioning variables $X_1$ and $X_2$:
\operatorname(Y)
= \operatorname\bigl operatorname(Y\mid X_1,X_2)\bigr+ \operatorname\bigl operatorname(\operatorname[Y\mid X_1,X_2mid X_1)\bigr">\mid_X_1,X_2.html" ;"title="operatorname(\operatorname[Y\mid X_1,X_2">operatorname(\operatorname[Y\mid X_1,X_2mid X_1)\bigr+ \operatorname(\operatorname[Y\mid X_1]).
More generally, the law of total cumulance extends this approach to higher moments.

See also

* Law of total expectation (Adam’s law)
* Law of total covariance
* Law of total cumulance
* Analysis of variance
* Conditional expectation
* R-squared
* Fraction of variance unexplained
* Variance decomposition

References

*
*
*
*
* {{cite journal , last1= Bowsher, first1= C.G. , last2= Swain, first2= P.S. , title=Identifying sources of variation and the flow of information in biochemical networks , journal=PNAS , volume=109 , issue=20 , pages=E1320–E1328 , year=2012 , doi=10.1073/pnas.1118365109

Algebra of random variables
Statistical deviation and dispersion
Articles containing proofs
Theory of probability distributions
Theorems in statistics
Statistical laws