The principle of maximum entropy states that the
probability distribution
In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomenon ...
which best represents the current state of knowledge about a system is the one with largest
entropy
Entropy is a scientific concept, as well as a measurable physical property, that is most commonly associated with a state of disorder, randomness, or uncertainty. The term and the concept are used in diverse fields, from classical thermodyna ...
, in the context of precisely stated prior data (such as a
proposition
In logic and linguistics, a proposition is the meaning of a declarative sentence. In philosophy, " meaning" is understood to be a non-linguistic entity which is shared by all sentences with the same meaning. Equivalently, a proposition is the no ...
that expresses
testable information).
Another way of stating this: Take precisely stated prior data or testable information about a probability distribution function. Consider the set of all trial probability distributions that would encode the prior data. According to this principle, the distribution with maximal
information entropy is the best choice.
History
The principle was first expounded by
E. T. Jaynes in two papers in 1957 where he emphasized a natural correspondence between
statistical mechanics and
information theory
Information theory is the scientific study of the quantification, storage, and communication of information. The field was originally established by the works of Harry Nyquist and Ralph Hartley, in the 1920s, and Claude Shannon in the 1940s. ...
. In particular, Jaynes offered a new and very general rationale why the Gibbsian method of statistical mechanics works. He argued that the
entropy
Entropy is a scientific concept, as well as a measurable physical property, that is most commonly associated with a state of disorder, randomness, or uncertainty. The term and the concept are used in diverse fields, from classical thermodyna ...
of statistical mechanics and the
information entropy of
information theory
Information theory is the scientific study of the quantification, storage, and communication of information. The field was originally established by the works of Harry Nyquist and Ralph Hartley, in the 1920s, and Claude Shannon in the 1940s. ...
are basically the same thing. Consequently,
statistical mechanics should be seen just as a particular application of a general tool of logical
inference
Inferences are steps in reasoning, moving from premises to logical consequences; etymologically, the word ''wikt:infer, infer'' means to "carry forward". Inference is theoretically traditionally divided into deductive reasoning, deduction and in ...
and information theory.
Overview
In most practical cases, the stated prior data or testable information is given by a set of
conserved quantities (average values of some moment functions), associated with the
probability distribution
In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomenon ...
in question. This is the way the maximum entropy principle is most often used in
statistical thermodynamics. Another possibility is to prescribe some
symmetries of the probability distribution. The equivalence between
conserved quantities and corresponding
symmetry group
In group theory, the symmetry group of a geometric object is the group of all transformations under which the object is invariant, endowed with the group operation of composition. Such a transformation is an invertible mapping of the amb ...
s implies a similar equivalence for these two ways of specifying the testable information in the maximum entropy method.
The maximum entropy principle is also needed to guarantee the uniqueness and consistency of probability assignments obtained by different methods,
statistical mechanics and
logical inference
Inferences are steps in reasoning, moving from premises to logical consequences; etymologically, the word ''infer'' means to "carry forward". Inference is theoretically traditionally divided into deduction and induction, a distinction that in ...
in particular.
The maximum entropy principle makes explicit our freedom in using different forms of
prior data. As a special case, a uniform
prior probability density (Laplace's
principle of indifference, sometimes called the principle of insufficient reason), may be adopted. Thus, the maximum entropy principle is not merely an alternative way to view the usual methods of inference of classical statistics, but represents a significant conceptual generalization of those methods.
However these statements do not imply that thermodynamical systems need not be shown to be
ergodic
In mathematics, ergodicity expresses the idea that a point of a moving system, either a dynamical system or a stochastic process, will eventually visit all parts of the space that the system moves in, in a uniform and random sense. This implies tha ...
to justify treatment as a
statistical ensemble.
In ordinary language, the principle of maximum entropy can be said to express a claim of epistemic modesty, or of maximum ignorance. The selected distribution is the one that makes the least claim to being informed beyond the stated prior data, that is to say the one that admits the most ignorance beyond the stated prior data.
Testable information
The principle of maximum entropy is useful explicitly only when applied to ''testable information''. Testable information is a statement about a probability distribution whose truth or falsity is well-defined. For example, the statements
:the
expectation of the variable
is 2.87
and
:
(where
and
are probabilities of events) are statements of testable information.
Given testable information, the maximum entropy procedure consists of seeking the
probability distribution
In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomenon ...
which maximizes
information entropy, subject to the constraints of the information. This constrained optimization problem is typically solved using the method of
Lagrange multipliers.
Entropy maximization with no testable information respects the universal "constraint" that the sum of the probabilities is one. Under this constraint, the maximum entropy discrete probability distribution is the
uniform distribution
Uniform distribution may refer to:
* Continuous uniform distribution
* Discrete uniform distribution
* Uniform distribution (ecology)
* Equidistributed sequence
See also
*
* Homogeneous distribution
In mathematics, a homogeneous distribution ...
,
:
Applications
The principle of maximum entropy is commonly applied in two ways to inferential problems:
Prior probabilities
The principle of maximum entropy is often used to obtain
prior probability distributions for
Bayesian inference
Bayesian inference is a method of statistical inference in which Bayes' theorem is used to update the probability for a hypothesis as more evidence or information becomes available. Bayesian inference is an important technique in statistics, and ...
. Jaynes was a strong advocate of this approach, claiming the maximum entropy distribution represented the least informative distribution.
A large amount of literature is now dedicated to the elicitation of maximum entropy priors and links with
channel coding
In computing, telecommunication, information theory, and coding theory, an error correction code, sometimes error correcting code, (ECC) is used for controlling errors in data over unreliable or noisy communication channels. The central idea ...
.
Posterior probabilities
Maximum entropy is a sufficient updating rule for
radical probabilism.
Richard Jeffrey's
probability kinematics is a special case of maximum entropy inference. However, maximum entropy is not a generalisation of all such sufficient updating rules.
Maximum entropy models
Alternatively, the principle is often invoked for model specification: in this case the observed data itself is assumed to be the testable information. Such models are widely used in
natural language processing
Natural language processing (NLP) is an interdisciplinary subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to proc ...
. An example of such a model is
logistic regression
In statistics, the logistic model (or logit model) is a statistical model that models the probability of an event taking place by having the log-odds for the event be a linear combination of one or more independent variables. In regression an ...
, which corresponds to the maximum entropy classifier for independent observations.
Probability density estimation
One of the main applications of the maximum entropy principle is in discrete and continuous
density estimation.
Similar to
support vector machine
In machine learning, support vector machines (SVMs, also support vector networks) are supervised learning models with associated learning algorithms that analyze data for classification and regression analysis. Developed at AT&T Bell Laborat ...
estimators,
the maximum entropy principle may require the solution to a
quadratic programming problem, and thus provide
a sparse mixture model as the optimal density estimator. One important advantage of the method is its ability to incorporate prior information in the density estimation.
General solution for the maximum entropy distribution with linear constraints
Discrete case
We have some testable information ''I'' about a quantity ''x'' taking values in . We assume this information has the form of ''m'' constraints on the expectations of the functions ''f
k''; that is, we require our probability distribution to satisfy the moment inequality/equality constraints:
:
where the
are observables. We also require the probability density to sum to one, which may be viewed as a primitive constraint on the identity function and an observable equal to 1 giving the constraint
:
The probability distribution with maximum information entropy subject to these inequality/equality constraints is of the form:
:
for some
. It is sometimes called the
Gibbs distribution
In statistical mechanics and mathematics, a Boltzmann distribution (also called Gibbs distribution Translated by J.B. Sykes and M.J. Kearsley. See section 28) is a probability distribution or probability measure that gives the probability t ...
. The normalization constant is determined by:
:
and is conventionally called the
partition function. (The
Pitman–Koopman theorem states that the necessary and sufficient condition for a sampling distribution to admit
sufficient statistics
In statistics, a statistic is ''sufficient'' with respect to a statistical model and its associated unknown parameter if "no other statistic that can be calculated from the same sample provides any additional information as to the value of the pa ...
of bounded dimension is that it have the general form of a maximum entropy distribution.)
The λ
k parameters are Lagrange multipliers. In the case of equality constraints their values are determined from the solution of the nonlinear equations
:
In the case of inequality constraints, the Lagrange multipliers are determined from the solution of a
convex optimization
Convex optimization is a subfield of mathematical optimization that studies the problem of minimizing convex functions over convex sets (or, equivalently, maximizing concave functions over convex sets). Many classes of convex optimization pr ...
program with linear constraints.
In both cases, there is no
closed form solution
In mathematics, a closed-form expression is a mathematical expression that uses a finite number of standard operations. It may contain constants, variables, certain well-known operations (e.g., + − × ÷), and functions (e.g., ''n''th r ...
, and the computation of the Lagrange multipliers usually requires
numerical methods.
Continuous case
For
continuous distributions, the Shannon entropy cannot be used, as it is only defined for discrete probability spaces. Instead
Edwin Jaynes (1963, 1968, 2003) gave the following formula, which is closely related to the
relative entropy (see also
differential entropy).
:
where ''q''(''x''), which Jaynes called the "invariant measure", is proportional to the
limiting density of discrete points. For now, we shall assume that ''q'' is known; we will discuss it further after the solution equations are given.
A closely related quantity, the relative entropy, is usually defined as the
Kullback–Leibler divergence of ''p'' from ''q'' (although it is sometimes, confusingly, defined as the negative of this). The inference principle of minimizing this, due to Kullback, is known as the
Principle of Minimum Discrimination Information.
We have some testable information ''I'' about a quantity ''x'' which takes values in some
interval of the
real numbers
In mathematics, a real number is a number that can be used to measure a ''continuous'' one-dimensional quantity such as a distance, duration or temperature. Here, ''continuous'' means that values can have arbitrarily small variations. Every re ...
(all integrals below are over this interval). We assume this information has the form of ''m'' constraints on the expectations of the functions ''f
k'', i.e. we require our probability density function to satisfy the inequality (or purely equality) moment constraints:
:
where the
are observables. We also require the probability density to integrate to one, which may be viewed as a primitive constraint on the identity function and an observable equal to 1 giving the constraint
:
The probability density function with maximum ''H
c'' subject to these constraints is:
: