Smooth Maximum

In mathematics, a smooth maximum of an indexed family ''x''₁, ..., ''x''_''n'' of numbers is a smooth approximation to the maximum function $\max(x_1,\ldots,x_n),$ meaning a parametric family of functions $m_\alpha(x_1,\ldots,x_n)$ such that for every , the function is smooth, and the family converges to the maximum function as . The concept of smooth minimum is similarly defined. In many cases, a single family approximates both: maximum as the parameter goes to positive infinity, minimum as the parameter goes to negative infinity; in symbols, as and as . The term can also be used loosely for a specific smooth function that behaves similarly to a maximum, without necessarily being part of a parametrized family.

Examples

Boltzmann operator

For large positive values of the parameter $\alpha > 0$, the following formulation is a smooth, differentiable approximation of the maximum function. For negative values of the parameter that are large in absolute value, it approximates the minimum.

:$\mathcal_\alpha (x_1,\ldots,x_n) = \frac$

$\mathcal_\alpha$ has the following properties:
#$\mathcal_\alpha\to \max$ as $\alpha\to\infty$
#$\mathcal_0$ is the arithmetic mean of its inputs
#$\mathcal_\alpha\to \min$ as $\alpha\to -\infty$

The gradient of $\mathcal_$ is closely related to softmax and is given by

:
\nabla_\mathcal_\alpha (x_1,\ldots,x_n) = \frac + \alpha(x_i - \mathcal_\alpha (x_1,\ldots,x_n))

This makes the softmax function useful for optimization techniques that use gradient descent.

This operator is sometimes called the Boltzmann operator, after the Boltzmann distribution.

LogSumExp

Another smooth maximum is LogSumExp:

:$\mathrm_\alpha(x_1, \ldots, x_n) = \frac \log \sum_^n \exp \alpha x_i$

This can also be normalized if the $x_i$ are all non-negative, yielding a function with domain is defined as follows:
:$\mathrm_\alpha(x) = \frac \log \frac \sum_^n \exp \alpha x_i$
It is a Non-expansive function">non-expansive operator. As $\alpha \to \infty$, it acts like a maximum. As $\alpha \to 0$, it acts like an arithmetic mean. As $\alpha \to -\infty$, it acts like a minimum. This operator can be viewed as a particular instantiation of the quasi-arithmetic mean. It can also be derived from information theoretical principles as a way of regularizing policies with a cost function defined by KL divergence. The operator has previously been utilized in other areas, such as power engineering.

p-Norm

Another smooth maximum is the p-norm:

:$\, (x_1, \ldots, x_n) \, _p = \left( \sum_^n , x_i, ^p \right)^\frac$

which converges to $\, (x_1, \ldots, x_n) \, _\infty = \max_ , x_i,$ as $p \to \infty$.

An advantage of the p-norm is that it is a norm. As such it is scale invariant (homogeneous): $\, (\lambda x_1, \ldots, \lambda x_n) \, _p = , \lambda, \cdot \, (x_1, \ldots, x_n) \, _p$, and it satisfies the triangle inequality.

Smooth maximum unit

The following binary operator is called the Smooth Maximum Unit (SMU):
:$\begin \textstyle\max_\varepsilon(a, b) &= \frac \\ &= \frac \end$
where $\varepsilon \geq 0$ is a parameter. As $\varepsilon \to 0$, $, \cdot, _\varepsilon \to , \cdot,$ and thus $\textstyle\max_\varepsilon \to \max$.

See also

* LogSumExp
* Softmax function
* Generalized mean

References

{{Reflist

Mathematical notation
Basic concepts in set theory

https://www.johndcook.com/soft_maximum.pdf

M. Lange, D. Zühlke, O. Holz, and T. Villmann, "Applications of lp-norms and their smooth approximations for gradient based learning vector quantization," ''in Proc. ESANN'', Apr. 2014, pp. 271-276.
(https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2014-153.pdf)