In
probability theory
Probability theory is the branch of mathematics concerned with probability. Although there are several different probability interpretations, probability theory treats the concept in a rigorous mathematical manner by expressing it through a set o ...
and
computer science
Computer science is the study of computation, automation, and information. Computer science spans theoretical disciplines (such as algorithms, theory of computation, information theory, and automation) to Applied science, practical discipli ...
, a log probability is simply a
logarithm
In mathematics, the logarithm is the inverse function to exponentiation. That means the logarithm of a number to the base is the exponent to which must be raised, to produce . For example, since , the ''logarithm base'' 10 o ...
of a
probability
Probability is the branch of mathematics concerning numerical descriptions of how likely an Event (probability theory), event is to occur, or how likely it is that a proposition is true. The probability of an event is a number between 0 and ...
. The use of log probabilities means representing probabilities on a
logarithmic scale
A logarithmic scale (or log scale) is a way of displaying numerical data over a very wide range of values in a compact way—typically the largest numbers in the data are hundreds or even thousands of times larger than the smallest numbers. Such a ...
, instead of the standard
, 1
The comma is a punctuation mark that appears in several variants in different languages. It has the same shape as an apostrophe or single closing quotation mark () in many typefaces, but it differs from them in being placed on the baseline (t ...
/math>
unit interval
In mathematics, the unit interval is the closed interval , that is, the set of all real numbers that are greater than or equal to 0 and less than or equal to 1. It is often denoted ' (capital letter ). In addition to its role in real analysis, ...
.
Since the probabilities of
independent
Independent or Independents may refer to:
Arts, entertainment, and media Artist groups
* Independents (artist group), a group of modernist painters based in the New Hope, Pennsylvania, area of the United States during the early 1930s
* Independ ...
events multiply, and logarithms convert multiplication to addition, log probabilities of independent events add. Log probabilities are thus practical for computations, and have an intuitive interpretation in terms of
information theory
Information theory is the scientific study of the quantification (science), quantification, computer data storage, storage, and telecommunication, communication of information. The field was originally established by the works of Harry Nyquist a ...
: the negative of the average log probability is the
information entropy of an event. Similarly,
likelihoods are often transformed to the log scale, and the corresponding
log-likelihood can be interpreted as the degree to which an event supports a
statistical model
A statistical model is a mathematical model that embodies a set of statistical assumptions concerning the generation of Sample (statistics), sample data (and similar data from a larger Statistical population, population). A statistical model repres ...
. The log probability is widely used in implementations of computations with probability, and is studied as a concept in its own right in some applications of information theory, such as
natural language processing
Natural language processing (NLP) is an interdisciplinary subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to pro ...
.
Motivation
Representing probabilities in this way has several practical advantages:
# Speed. Since multiplication is more
expensive
In production, research, retail, and accounting, a cost is the value of money that has been used up to produce something or deliver a service, and hence is not available for use anymore. In business, the cost may be one of acquisition, in which ...
than addition, taking the product of a high number of probabilities is often faster if they are represented in log form. (The conversion to log form is expensive, but is only incurred once.) Multiplication arises from calculating the probability that multiple independent events occur: the probability that all independent events of interest occur is the product of all these events' probabilities.
# Accuracy. The use of log probabilities improves
numerical stability
In the mathematical subfield of numerical analysis, numerical stability is a generally desirable property of numerical algorithms. The precise definition of stability depends on the context. One is numerical linear algebra and the other is algorit ...
, when the probabilities are very small, because of the way in which computers
approximate real numbers.
# Simplicity. Many probability distributions have an exponential form. Taking the log of these distributions eliminates the exponential function, unwrapping the exponent. For example, the log probability of the normal distribution's
probability density function
In probability theory, a probability density function (PDF), or density of a continuous random variable, is a function whose value at any given sample (or point) in the sample space (the set of possible values taken by the random variable) can ...
is
instead of
. Log probabilities make some mathematical manipulations easier to perform.
Representation issues
The logarithm function is not defined for zero, so log probabilities can only represent non-zero probabilities. Since the logarithm of a number in
interval is negative, often the negative log probabilities are used. In that case the log probabilities in the following formulas would be
inverted.
Any base can be selected for the logarithm.
:
:
Basic manipulations
The product of probabilities
corresponds to addition in logarithmic space.
:
The
sum of probabilities is a bit more involved to compute in logarithmic space, requiring the computation of one exponent and one logarithm.
However, in many applications a multiplication of probabilities (giving the probability of all independent events occurring) is used more often than their addition (giving the probability of at least one of them occurring). Additionally, the cost of computing the addition can be avoided in some situations by simply using the highest probability as an approximation. Since probabilities are non-negative this gives a lower bound. This approximation is used in reverse to get a
continuous approximation of the max function.
Addition in log space
:
The formula above is more accurate than
, provided one takes advantage of the asymmetry in the addition formula.
should be the larger (least negative) of the two operands. This also produces the correct behavior if one of the operands is
floating-point
In computing, floating-point arithmetic (FP) is arithmetic that represents real numbers approximately, using an integer with a fixed precision, called the significand, scaled by an integer exponent of a fixed base. For example, 12.345 can b ...
negative infinity
In mathematics, the affinely extended real number system is obtained from the real number system \R by adding two infinity elements: +\infty and -\infty, where the infinities are treated as actual numbers. It is useful in describing the algebra on ...
, which corresponds to a probability of zero.
:
This quantity is
indeterminate
Indeterminate may refer to:
In mathematics
* Indeterminate (variable), a symbol that is treated as a variable
* Indeterminate system, a system of simultaneous equations that has more than one solution
* Indeterminate equation, an equation that ha ...
, and will result in
NaN.
:
This is the desired answer.
The above formula alone will incorrectly produce an indeterminate result in the case where both arguments are
. This should be checked for separately to return
.
For numerical reasons, one should use a function that computes
(
log1p) directly.
See also
*
Information content
*
Log-likelihood
Probability
Probability is the branch of mathematics concerning numerical descriptions of how likely an Event (probability theory), event is to occur, or how likely it is that a proposition is true. The probability of an event is a number between 0 and ...
Mathematics of computing