Differential entropy
   HOME

TheInfoList



OR:

Differential entropy (also referred to as continuous entropy) is a concept in information theory that began as an attempt by
Claude Shannon Claude Elwood Shannon (April 30, 1916 – February 24, 2001) was an American mathematician, electrical engineer, and cryptographer known as a "father of information theory". As a 21-year-old master's degree student at the Massachusetts Inst ...
to extend the idea of (Shannon)
entropy Entropy is a scientific concept, as well as a measurable physical property, that is most commonly associated with a state of disorder, randomness, or uncertainty. The term and the concept are used in diverse fields, from classical thermodynam ...
, a measure of average
surprisal In information theory, the information content, self-information, surprisal, or Shannon information is a basic quantity derived from the probability of a particular Event (probability theory), event occurring from a random variable. It can be tho ...
of a random variable, to continuous probability distributions. Unfortunately, Shannon did not derive this formula, and rather just assumed it was the correct continuous analogue of discrete entropy, but it is not. The actual continuous version of discrete entropy is the
limiting density of discrete points In information theory, the limiting density of discrete points is an adjustment to the formula of Claude Shannon for differential entropy. It was formulated by Edwin Thompson Jaynes to address defects in the initial definition of differential e ...
(LDDP). Differential entropy (described here) is commonly encountered in the literature, but it is a limiting case of the LDDP, and one that loses its fundamental association with discrete
entropy Entropy is a scientific concept, as well as a measurable physical property, that is most commonly associated with a state of disorder, randomness, or uncertainty. The term and the concept are used in diverse fields, from classical thermodynam ...
. In terms of measure theory, the differential entropy of a probability measure is the negative
relative entropy Relative may refer to: General use *Kinship and family, the principle binding the most basic social units society. If two people are connected by circumstances of birth, they are said to be ''relatives'' Philosophy *Relativism, the concept that ...
from that measure to the Lebesgue measure, where the latter is treated as if it were a probability measure, despite being unnormalized.


Definition

Let X be a random variable with a
probability density function In probability theory, a probability density function (PDF), or density of a continuous random variable, is a function whose value at any given sample (or point) in the sample space (the set of possible values taken by the random variable) ca ...
f whose support is a set \mathcal X. The ''differential entropy'' h(X) or h(f) is defined as For probability distributions which don't have an explicit density function expression, but have an explicit quantile function expression, Q(p), then h(Q) can be defined in terms of the derivative of Q(p) i.e. the quantile density function Q'(p) as :h(Q) = \int_0^1 \log Q'(p)\,dp. As with its discrete analog, the units of differential entropy depend on the base of the
logarithm In mathematics, the logarithm is the inverse function to exponentiation. That means the logarithm of a number  to the base  is the exponent to which must be raised, to produce . For example, since , the ''logarithm base'' 10 of ...
, which is usually 2 (i.e., the units are
bit The bit is the most basic unit of information in computing and digital communications. The name is a portmanteau of binary digit. The bit represents a logical state with one of two possible values. These values are most commonly represente ...
s). See logarithmic units for logarithms taken in different bases. Related concepts such as
joint A joint or articulation (or articular surface) is the connection made between bones, ossicles, or other hard structures in the body which link an animal's skeletal system into a functional whole.Saladin, Ken. Anatomy & Physiology. 7th ed. McGraw- ...
, conditional differential entropy, and
relative entropy Relative may refer to: General use *Kinship and family, the principle binding the most basic social units society. If two people are connected by circumstances of birth, they are said to be ''relatives'' Philosophy *Relativism, the concept that ...
are defined in a similar fashion. Unlike the discrete analog, the differential entropy has an offset that depends on the units used to measure X. For example, the differential entropy of a quantity measured in millimeters will be more than the same quantity measured in meters; a dimensionless quantity will have differential entropy of more than the same quantity divided by 1000. One must take care in trying to apply properties of discrete entropy to differential entropy, since probability density functions can be greater than 1. For example, the uniform distribution \mathcal(0,1/2) has ''negative'' differential entropy :\int_0^\frac -2\log(2)\,dx=-\log(2)\, being less than that of \mathcal(0,1) which has ''zero'' differential entropy. Thus, differential entropy does not share all properties of discrete entropy. Note that the continuous
mutual information In probability theory and information theory, the mutual information (MI) of two random variables is a measure of the mutual dependence between the two variables. More specifically, it quantifies the " amount of information" (in units such ...
I(X;Y) has the distinction of retaining its fundamental significance as a measure of discrete information since it is actually the limit of the discrete mutual information of ''partitions'' of X and Y as these partitions become finer and finer. Thus it is invariant under non-linear
homeomorphisms In the mathematical field of topology, a homeomorphism, topological isomorphism, or bicontinuous function is a bijective and continuous function between topological spaces that has a continuous inverse function. Homeomorphisms are the isomorp ...
(continuous and uniquely invertible maps), including linear transformations of X and Y, and still represents the amount of discrete information that can be transmitted over a channel that admits a continuous space of values. For the direct analogue of discrete entropy extended to the continuous space, see
limiting density of discrete points In information theory, the limiting density of discrete points is an adjustment to the formula of Claude Shannon for differential entropy. It was formulated by Edwin Thompson Jaynes to address defects in the initial definition of differential e ...
.


Properties of differential entropy

* For probability densities f and g, the
Kullback–Leibler divergence In mathematical statistics, the Kullback–Leibler divergence (also called relative entropy and I-divergence), denoted D_\text(P \parallel Q), is a type of statistical distance: a measure of how one probability distribution ''P'' is different fr ...
D_(f , , g) is greater than or equal to 0 with equality only if f=g
almost everywhere In measure theory (a branch of mathematical analysis), a property holds almost everywhere if, in a technical sense, the set for which the property holds takes up nearly all possibilities. The notion of "almost everywhere" is a companion notion to ...
. Similarly, for two random variables X and Y, I(X;Y) \ge 0 and h(X, Y) \le h(X) with equality
if and only if In logic and related fields such as mathematics and philosophy, "if and only if" (shortened as "iff") is a biconditional logical connective between statements, where either both statements are true or both are false. The connective is b ...
X and Y are
independent Independent or Independents may refer to: Arts, entertainment, and media Artist groups * Independents (artist group), a group of modernist painters based in the New Hope, Pennsylvania, area of the United States during the early 1930s * Independ ...
. * The chain rule for differential entropy holds as in the discrete case ::h(X_1, \ldots, X_n) = \sum_^ h(X_i, X_1, \ldots, X_) \leq \sum_^ h(X_i). * Differential entropy is translation invariant, i.e. for a constant c. ::h(X+c) = h(X) * Differential entropy is in general not invariant under arbitrary invertible maps. :: In particular, for a constant a :::h(aX) = h(X)+ \log , a, :: For a vector valued random variable \mathbf and an invertible (square)
matrix Matrix most commonly refers to: * ''The Matrix'' (franchise), an American media franchise ** ''The Matrix'', a 1999 science-fiction action film ** "The Matrix", a fictional setting, a virtual reality environment, within ''The Matrix'' (franchis ...
\mathbf :::h(\mathbf\mathbf)=h(\mathbf)+\log \left( , \det \mathbf, \right) * In general, for a transformation from a random vector to another random vector with same dimension \mathbf=m \left(\mathbf\right), the corresponding entropies are related via ::h(\mathbf) \leq h(\mathbf) + \int f(x) \log \left\vert \frac \right\vert dx :where \left\vert \frac \right\vert is the Jacobian of the transformation m. The above inequality becomes an equality if the transform is a bijection. Furthermore, when m is a rigid rotation, translation, or combination thereof, the Jacobian determinant is always 1, and h(Y)=h(X). * If a random vector X \in \mathbb^n has mean zero and
covariance In probability theory and statistics, covariance is a measure of the joint variability of two random variables. If the greater values of one variable mainly correspond with the greater values of the other variable, and the same holds for the ...
matrix K, h(\mathbf) \leq \frac \log(\det) = \frac \log 2\pi e)^n \det/math> with equality if and only if X is
jointly gaussian In probability theory and statistics, the multivariate normal distribution, multivariate Gaussian distribution, or joint normal distribution is a generalization of the one-dimensional (univariate) normal distribution to higher dimensions. One ...
(see below). However, differential entropy does not have other desirable properties: * It is not invariant under
change of variables Change or Changing may refer to: Alteration * Impermanence, a difference in a state of affairs at different points in time * Menopause, also referred to as "the change", the permanent cessation of the menstrual period * Metamorphosis, or change, ...
, and is therefore most useful with dimensionless variables. * It can be negative. A modification of differential entropy that addresses these drawbacks is the relative information entropy, also known as the Kullback–Leibler divergence, which includes an
invariant measure In mathematics, an invariant measure is a measure that is preserved by some function. The function may be a geometric transformation. For examples, circular angle is invariant under rotation, hyperbolic angle is invariant under squeeze mapping, ...
factor (see
limiting density of discrete points In information theory, the limiting density of discrete points is an adjustment to the formula of Claude Shannon for differential entropy. It was formulated by Edwin Thompson Jaynes to address defects in the initial definition of differential e ...
).


Maximization in the normal distribution


Theorem

With a
normal distribution In statistics, a normal distribution or Gaussian distribution is a type of continuous probability distribution for a real-valued random variable. The general form of its probability density function is : f(x) = \frac e^ The parameter \mu ...
, differential entropy is maximized for a given variance. A Gaussian random variable has the largest entropy amongst all random variables of equal variance, or, alternatively, the maximum entropy distribution under constraints of mean and variance is the Gaussian.


Proof

Let g(x) be a
Gaussian Carl Friedrich Gauss (1777–1855) is the eponym of all of the topics listed below. There are over 100 topics all named after this German mathematician and scientist, all in the fields of mathematics, physics, and astronomy. The English eponymo ...
PDF with mean μ and variance \sigma^2 and f(x) an arbitrary PDF with the same variance. Since differential entropy is translation invariant we can assume that f(x) has the same mean of \mu as g(x). Consider the
Kullback–Leibler divergence In mathematical statistics, the Kullback–Leibler divergence (also called relative entropy and I-divergence), denoted D_\text(P \parallel Q), is a type of statistical distance: a measure of how one probability distribution ''P'' is different fr ...
between the two distributions : 0 \leq D_(f , , g) = \int_^\infty f(x) \log \left( \frac \right) dx = -h(f) - \int_^\infty f(x)\log(g(x)) dx. Now note that :\begin \int_^\infty f(x)\log(g(x)) dx &= \int_^\infty f(x)\log\left( \frace^\right) dx \\ &= \int_^\infty f(x) \log\frac dx \,+\, \log(e)\int_^\infty f(x)\left( -\frac\right) dx \\ &= -\tfrac\log(2\pi\sigma^2) - \log(e)\frac \\ &= -\tfrac\left(\log(2\pi\sigma^2) + \log(e)\right) \\ &= -\tfrac\log(2\pi e \sigma^2) \\ &= -h(g) \end because the result does not depend on f(x) other than through the variance. Combining the two results yields : h(g) - h(f) \geq 0 \! with equality when f(x)=g(x) following from the properties of Kullback–Leibler divergence.


Alternative proof

This result may also be demonstrated using the
variational calculus The calculus of variations (or Variational Calculus) is a field of mathematical analysis that uses variations, which are small changes in functions and functionals, to find maxima and minima of functionals: mappings from a set of functions t ...
. A Lagrangian function with two Lagrangian multipliers may be defined as: :L=\int_^\infty g(x)\ln(g(x))\,dx-\lambda_0\left(1-\int_^\infty g(x)\,dx\right)-\lambda\left(\sigma^2-\int_^\infty g(x)(x-\mu)^2\,dx\right) where ''g(x)'' is some function with mean μ. When the entropy of ''g(x)'' is at a maximum and the constraint equations, which consist of the normalization condition \left(1=\int_^\infty g(x)\,dx\right) and the requirement of fixed variance \left(\sigma^2=\int_^\infty g(x)(x-\mu)^2\,dx\right), are both satisfied, then a small variation δ''g''(''x'') about ''g(x)'' will produce a variation δ''L'' about ''L'' which is equal to zero: :0=\delta L=\int_^\infty \delta g(x)\left (\ln(g(x))+1+\lambda_0+\lambda(x-\mu)^2\right )\,dx Since this must hold for any small δ''g''(''x''), the term in brackets must be zero, and solving for ''g(x)'' yields: :g(x)=e^ Using the constraint equations to solve for λ0 and λ yields the normal distribution: :g(x)=\frace^


Example: Exponential distribution

Let X be an
exponentially distributed In probability theory and statistics, the exponential distribution is the probability distribution of the time between events in a Poisson point process, i.e., a process in which events occur continuously and independently at a constant averag ...
random variable with parameter \lambda, that is, with probability density function :f(x) = \lambda e^ \mbox x \geq 0. Its differential entropy is then Here, h_e(X) was used rather than h(X) to make it explicit that the logarithm was taken to base ''e'', to simplify the calculation.


Relation to estimator error

The differential entropy yields a lower bound on the expected squared error of an
estimator In statistics, an estimator is a rule for calculating an estimate of a given quantity based on observed data: thus the rule (the estimator), the quantity of interest (the estimand) and its result (the estimate) are distinguished. For example, the ...
. For any random variable X and estimator \widehat the following holds: :\operatorname X - \widehat)^2\ge \frace^ with equality if and only if X is a Gaussian random variable and \widehat is the mean of X.


Differential entropies for various distributions

In the table below \Gamma(x) = \int_0^ e^ t^ dt is the
gamma function In mathematics, the gamma function (represented by , the capital letter gamma from the Greek alphabet) is one commonly used extension of the factorial function to complex numbers. The gamma function is defined for all complex numbers except ...
, \psi(x) = \frac \ln\Gamma(x)=\frac is the
digamma function In mathematics, the digamma function is defined as the logarithmic derivative of the gamma function: :\psi(x)=\frac\ln\big(\Gamma(x)\big)=\frac\sim\ln-\frac. It is the first of the polygamma functions. It is strictly increasing and strict ...
, B(p,q) = \frac is the
beta function In mathematics, the beta function, also called the Euler integral of the first kind, is a special function that is closely related to the gamma function and to binomial coefficients. It is defined by the integral : \Beta(z_1,z_2) = \int_0^1 t^( ...
, and γ''E'' is
Euler's constant Euler's constant (sometimes also called the Euler–Mascheroni constant) is a mathematical constant usually denoted by the lowercase Greek letter gamma (). It is defined as the limiting difference between the harmonic series and the natural ...
. {, class="wikitable" style="background:white" , + Table of differential entropies , - ! Distribution Name !! Probability density function (pdf) !! Differential entropy in
nat Nat or NAT may refer to: Computing * Network address translation (NAT), in computer networking Organizations * National Actors Theatre, New York City, U.S. * National AIDS trust, a British charity * National Archives of Thailand * National As ...
s , , Support , - ,
Uniform A uniform is a variety of clothing worn by members of an organization while participating in that organization's activity. Modern uniforms are most often worn by armed forces and paramilitary organizations such as police, emergency services, ...
, , f(x) = \frac{1}{b-a} , , \ln(b - a) \, , , ,b, , - ,
Normal Normal(s) or The Normal(s) may refer to: Film and television * ''Normal'' (2003 film), starring Jessica Lange and Tom Wilkinson * ''Normal'' (2007 film), starring Carrie-Anne Moss, Kevin Zegers, Callum Keith Rennie, and Andrew Airlie * ''Norma ...
, , f(x) = \frac{1}{\sqrt{2\pi\sigma^2 \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right) , , \ln\left(\sigma\sqrt{2\,\pi\,e}\right) , , (-\infty,\infty)\, , - ,
Exponential Exponential may refer to any of several mathematical topics related to exponentiation, including: *Exponential function, also: **Matrix exponential, the matrix analogue to the above *Exponential decay, decrease at a rate proportional to value *Expo ...
, , f(x) = \lambda \exp\left(-\lambda x\right) , , 1 - \ln \lambda \, , , - , Rayleigh distribution, Rayleigh , , f(x) = \frac{x}{\sigma^2} \exp\left(-\frac{x^2}{2\sigma^2}\right) , , 1 + \ln \frac{\sigma}{\sqrt{2 + \frac{\gamma_E}{2}, , [0,\infty)\, , - , Beta distribution, Beta , , f(x) = \frac{x^{\alpha-1}(1-x)^{\beta-1{B(\alpha,\beta)} for 0 \leq x \leq 1 , , \ln B(\alpha,\beta) - (\alpha-1)[\psi(\alpha) - \psi(\alpha +\beta)]\,
- (\beta-1) psi(\beta) - \psi(\alpha + \beta)\, , , ,1, , - ,
Cauchy Baron Augustin-Louis Cauchy (, ; ; 21 August 178923 May 1857) was a French mathematician, engineer, and physicist who made pioneering contributions to several branches of mathematics, including mathematical analysis and continuum mechanics. He w ...
, , f(x) = \frac{\gamma}{\pi} \frac{1}{\gamma^2 + x^2} , , \ln(4\pi\gamma) \, , , (-\infty,\infty)\, , - ,
Chi Chi or CHI may refer to: Greek *Chi (letter), the Greek letter (uppercase Χ, lowercase χ); Chinese *Chi (length), ''Chi'' (length) (尺), a traditional unit of length, about ⅓ meter *Chi (mythology) (螭), a dragon *Chi (surname) (池, pin ...
, , f(x) = \frac{2}{2^{k/2} \Gamma(k/2)} x^{k-1} \exp\left(-\frac{x^2}{2}\right) , , \ln{\frac{\Gamma(k/2)}{\sqrt{2} - \frac{k-1}{2} \psi\left(\frac{k}{2}\right) + \frac{k}{2}, , - , Chi-squared distribution, Chi-squared , , f(x) = \frac{1}{2^{k/2} \Gamma(k/2)} x^{\frac{k}{2}\!-\!1} \exp\left(-\frac{x}{2}\right) , , \ln 2\Gamma\left(\frac{k}{2}\right) - \left(1 - \frac{k}{2}\right)\psi\left(\frac{k}{2}\right) + \frac{k}{2}, , [0,\infty)\, , - , Erlang distribution, Erlang , , f(x) = \frac{\lambda^k}{(k-1)!} x^{k-1} \exp(-\lambda x) , , (1-k)\psi(k) + \ln \frac{\Gamma(k)}{\lambda} + k, , [0,\infty)\, , - , F distribution, F , , f(x) = \frac{n_1^{\frac{n_1}{2 n_2^{\frac{n_2}{2}{B(\frac{n_1}{2},\frac{n_2}{2})} \frac{x^{\frac{n_1}{2} - 1{(n_2 + n_1 x)^{\frac{n_1 + n2}{2} , , \ln \frac{n_1}{n_2} B\left(\frac{n_1}{2},\frac{n_2}{2}\right) + \left(1 - \frac{n_1}{2}\right) \psi\left(\frac{n_1}{2}\right) -
\left(1 + \frac{n_2}{2}\right)\psi\left(\frac{n_2}{2}\right) + \frac{n_1 + n_2}{2} \psi\left(\frac{n_1\!+\!n_2}{2}\right), , - , Gamma distribution, Gamma , , f(x) = \frac{x^{k - 1} \exp(-\frac{x}{\theta})}{\theta^k \Gamma(k)} , , \ln(\theta \Gamma(k)) + (1 - k)\psi(k) + k \, , , [0,\infty)\, , - , Laplace distribution, Laplace , , f(x) = \frac{1}{2b} \exp\left(-\frac{, x - \mu{b}\right) , , 1 + \ln(2b) \, , , (-\infty,\infty)\, , - , Logistic distribution, Logistic , , f(x) = \frac{e^{-x/s{s(1 + e^{-x/s})^2} , , \ln s + 2 \, , , (-\infty,\infty)\, , - ,
Lognormal In probability theory, a log-normal (or lognormal) distribution is a continuous probability distribution of a random variable whose logarithm is normally distributed. Thus, if the random variable is log-normally distributed, then has a norma ...
, , f(x) = \frac{1}{\sigma x \sqrt{2\pi \exp\left(-\frac{(\ln x - \mu)^2}{2\sigma^2}\right) , , \mu + \frac{1}{2} \ln(2\pi e \sigma^2), , - , Maxwell–Boltzmann distribution, Maxwell–Boltzmann , , f(x) = \frac{1}{a^3}\sqrt{\frac{2}{\pi\,x^{2}\exp\left(-\frac{x^2}{2a^2}\right) , , \ln(a\sqrt{2\pi})+\gamma_E-\frac{1}{2}, , [0,\infty)\, , - , Generalized Gaussian distribution, Generalized normal , , f(x) = \frac{2 \beta^{\frac{\alpha}{2}{\Gamma(\frac{\alpha}{2})} x^{\alpha - 1} \exp(-\beta x^2) , , \ln{\frac{\Gamma(\alpha/2)}{2\beta^{\frac{1}{2 - \frac{\alpha - 1}{2} \psi\left(\frac{\alpha}{2}\right) + \frac{\alpha}{2}, , (-\infty,\infty)\, , - , Pareto , , f(x) = \frac{\alpha x_m^\alpha}{x^{\alpha+1 , , \ln \frac{x_m}{\alpha} + 1 + \frac{1}{\alpha}, , - , Student's t-distribution, Student's t , , f(x) = \frac{(1 + x^2/\nu)^{-\frac{\nu+1}{2}{\sqrt{\nu}B(\frac{1}{2},\frac{\nu}{2})} , , \frac{\nu\!+\!1}{2}\left(\psi\left(\frac{\nu\!+\!1}{2}\right)\!-\!\psi\left(\frac{\nu}{2}\right)\right)\!+\!\ln \sqrt{\nu} B\left(\frac{1}{2},\frac{\nu}{2}\right), , (-\infty,\infty)\, , - , Triangular distribution, Triangular , , f(x) = \begin{cases} \frac{2(x-a)}{(b-a)(c-a)} & \mathrm{for\ } a \le x \leq c, \\[4pt] \frac{2(b-x)}{(b-a)(b-c)} & \mathrm{for\ } c < x \le b, \\[4pt] \end{cases} , , \frac{1}{2} + \ln \frac{b-a}{2}, , ,b, , - ,
Weibull Weibull is a Swedish locational surname. The Weibull family share the same roots as the Danish / Norwegian noble family of Falsenbr>They originated from and were named after the village of Weiböl in Widstedts parish, Jutland, but settled in Skà ...
, , f(x) = \frac{k}{\lambda^k} x^{k-1} \exp\left(-\frac{x^k}{\lambda^k}\right) , , \frac{(k-1)\gamma_E}{k} + \ln \frac{\lambda}{k} + 1, , - , Multivariate normal distribution, Multivariate normal , , f_X(\vec{x}) =
\frac{\exp \left( -\frac{1}{2} ( \vec{x} - \vec{\mu})^\top \Sigma^{-1}\cdot(\vec{x} - \vec{\mu}) \right)} {(2\pi)^{N/2} \left, \Sigma\^{1/2 , , \frac{1}{2}\ln\{(2\pi e)^{N} \det(\Sigma)\}, , \mathbb{R}^N , - Many of the differential entropies are from.


Variants

As described above, differential entropy does not share all properties of discrete entropy. For example, the differential entropy can be negative; also it is not invariant under continuous coordinate transformations. Edwin Thompson Jaynes showed in fact that the expression above is not the correct limit of the expression for a finite set of probabilities. A modification of differential entropy adds an
invariant measure In mathematics, an invariant measure is a measure that is preserved by some function. The function may be a geometric transformation. For examples, circular angle is invariant under rotation, hyperbolic angle is invariant under squeeze mapping, ...
factor to correct this, (see
limiting density of discrete points In information theory, the limiting density of discrete points is an adjustment to the formula of Claude Shannon for differential entropy. It was formulated by Edwin Thompson Jaynes to address defects in the initial definition of differential e ...
). If m(x) is further constrained to be a probability density, the resulting notion is called
relative entropy Relative may refer to: General use *Kinship and family, the principle binding the most basic social units society. If two people are connected by circumstances of birth, they are said to be ''relatives'' Philosophy *Relativism, the concept that ...
in information theory: :D(p, , m) = \int p(x)\log\frac{p(x)}{m(x)}\,dx. The definition of differential entropy above can be obtained by partitioning the range of X into bins of length h with associated sample points ih within the bins, for X Riemann integrable. This gives a quantized version of X, defined by X_h = ih if ih \le X \le (i+1)h. Then the entropy of X_h = ih is :H_h=-\sum_i hf(ih)\log (f(ih)) - \sum hf(ih)\log(h). The first term on the right approximates the differential entropy, while the second term is approximately -\log(h). Note that this procedure suggests that the entropy in the discrete sense of a
continuous random variable In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomenon ...
should be \infty.


See also

*
Information entropy In information theory, the entropy of a random variable is the average level of "information", "surprise", or "uncertainty" inherent to the variable's possible outcomes. Given a discrete random variable X, which takes values in the alphabet \ ...
*
Self-information In information theory, the information content, self-information, surprisal, or Shannon information is a basic quantity derived from the probability of a particular event occurring from a random variable. It can be thought of as an alternative ...
*
Entropy estimation In various science/engineering applications, such as independent component analysis, image analysis, genetic analysis, speech recognition, manifold learning, and time delay estimationBenesty, J.; Yiteng Huang; Jingdong Chen (2007) Time Delay Estima ...


References


External links

* * {{planetmath reference, urlname=DifferentialEntropy, title=Differential entropy Entropy and information Information theory Statistical randomness