Bernoulli Entropy

In information theory, the binary entropy function, denoted $\operatorname H(p)$ or $\operatorname H_\text(p)$, is defined as the entropy of a Bernoulli process with probability $p$ of one of two values. It is a special case of $\Eta(X)$, the entropy function. Mathematically, the Bernoulli trial is modelled as a random variable $X$ that can take on only two values: 0 and 1, which are mutually exclusive and exhaustive.

If $\operatorname(X=1) = p$, then $\operatorname(X=0) = 1-p$ and the entropy of $X$ (in shannons) is given by

:$\operatorname H(X) = \operatorname H_\text(p) = -p \log_2 p - (1 - p) \log_2 (1 - p)$,

where $0 \log_2 0$ is taken to be 0. The logarithms in this formula are usually taken (as shown in the graph) to the base 2. See ''binary logarithm''.

When $p=\tfrac 1 2$, the binary entropy function attains its maximum value. This is the case of an unbiased coin flip.

$\operatorname H(p)$ is distinguished from the entropy function $\Eta(X)$ in that the former takes a single real number as a parameter whereas the latter takes a distribution or random variable as a parameter.
Sometimes the binary entropy function is also written as $\operatorname H_2(p)$.
However, it is different from and should not be confused with the Rényi entropy, which is denoted as $\Eta_2(X)$.

Explanation

In terms of information theory, ''entropy'' is considered to be a measure of the uncertainty in a message. To put it intuitively, suppose $p=0$. At this probability, the event is certain never to occur, and so there is no uncertainty at all, leading to an entropy of 0. If $p=1$, the result is again certain, so the entropy is 0 here as well. When $p=1/2$, the uncertainty is at a maximum; if one were to place a fair bet on the outcome in this case, there is no advantage to be gained with prior knowledge of the probabilities. In this case, the entropy is maximum at a value of 1 bit. Intermediate values fall between these cases; for instance, if $p=1/4$, there is still a measure of uncertainty on the outcome, but one can still predict the outcome correctly more often than not, so the uncertainty measure, or entropy, is less than 1 full bit.

Derivative

The derivative of the binary entropy function may be expressed as the negative of the logit function:
:$\operatorname H_\text(p) = - \operatorname_2(p) = -\log_2\left( \frac \right)$.

Taylor series

The Taylor series of the binary entropy function in a neighborhood of 1/2 is
:$\operatorname H_\text(p) = 1 - \frac \sum^_ \frac$
for $0\le p\le 1$.

Bounds

The following bounds hold for $0 < p < 1$:
:$\ln(2) \cdot \log_2(p) \cdot \log_2(1-p) \leq H_\text(p) \leq \log_2(p) \cdot \log_2(1-p)$
and
:$4p(1-p) \leq H_\text(p) \leq (4p(1-p))^$
where $\ln$ denotes natural logarithm.

See also

* Metric entropy
* Information theory
* Information entropy
*Quantities of information

References

Further reading

* MacKay, David J. C.
Information Theory, Inference, and Learning Algorithms
' Cambridge: Cambridge University Press, 2003. {{ISBN, 0-521-64298-1

Entropy and information

zh-yue:二元熵函數