Smooth Maximum

In mathematics, a smooth maximum of an indexed family ''x''₁, ..., ''x''_''n'' of numbers is a smooth approximation to the maximum function $\max(x_1,\ldots,x_n),$ meaning a parametric family of functions $m_\alpha(x_1,\ldots,x_n)$ such that for every , the function is smooth, and the family converges to the maximum function as . The concept of smooth minimum is similarly defined. In many cases, a single family approximates both: maximum as the parameter goes to positive infinity, minimum as the parameter goes to negative infinity; in symbols, as and as . The term can also be used loosely for a specific smooth function that behaves similarly to a maximum, without necessarily being part of a parametrized family.

Examples

For large positive values of the parameter $\alpha > 0$, the following formulation is a smooth, differentiable approximation of the maximum function. For negative values of the parameter that are large in absolute value, it approximates the minimum.

:$\mathcal_\alpha (x_1,\ldots,x_n) = \frac$

$\mathcal_\alpha$ has the following properties:
#$\mathcal_\alpha\to \max$ as $\alpha\to\infty$
#$\mathcal_0$ is the arithmetic mean of its inputs
#$\mathcal_\alpha\to \min$ as $\alpha\to -\infty$

The gradient of $\mathcal_$ is closely related to softmax and is given by

:
\nabla_\mathcal_\alpha (x_1,\ldots,x_n) = \frac + \alpha(x_i - \mathcal_\alpha (x_1,\ldots,x_n))

This makes the softmax function useful for optimization techniques that use gradient descent.

LogSumExp

Another smooth maximum is LogSumExp:

:$\mathrm_\alpha(x_1, \ldots, x_n) = (1/\alpha)\log( \exp(\alpha x_1) + \ldots + \exp( \alpha x_n))$

This can also be normalized if the $x_i$ are all non-negative, yielding a function with domain $[0,\infty)^n$ and range $[0, \infty)$:
:$g(x_1, \ldots, x_n) = \log( \exp(x_1) + \ldots + \exp(x_n) - (n - 1) )$

The $(n - 1)$ term corrects for the fact that $\exp(0) = 1$ by canceling out all but one zero exponential, and $\log 1 = 0$ if all $x_i$ are zero.

p-Norm

Another smooth maximum is the p-norm:

:$, , (x_1, \ldots, x_n) , , _p = \left( , x_1, ^p + \cdots + , x_n, ^p \right)^$

which converges to $, , (x_1, \ldots, x_n) , , _\infty = \max_ , x_i,$ as $p \to \infty$.

An advantage of the p-norm is that it is a norm. As such it is "scale invariant" (homogeneous): $, , (\lambda x_1, \ldots, \lambda x_n) , , _p = , \lambda, \times , , (x_1, \ldots, x_n) , , _p$, and it satisfies the triangular inequality.

Other choices of smoothing function

:$\begin \textstyle\max_\varepsilon(a, b) &= \frac \\ &= \frac \end$
where $\varepsilon \geq 0$ is a parameter. As $\varepsilon \to 0$, $, \cdot, _\varepsilon \to , \cdot,$ and thus $\textstyle\max_\varepsilon \to \max$.

See also

* LogSumExp
* Softmax function
* Generalized mean

References

{{Reflist

Mathematical notation
Basic concepts in set theory

https://www.johndcook.com/soft_maximum.pdf

M. Lange, D. Zühlke, O. Holz, and T. Villmann, "Applications of lp-norms and their smooth approximations for gradient based learning vector quantization," ''in Proc. ESANN'', Apr. 2014, pp. 271-276.
(https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2014-153.pdf)