HOME

TheInfoList



OR:

In the context of
artificial neural network Artificial neural networks (ANNs), usually simply called neural networks (NNs) or neural nets, are computing systems inspired by the biological neural networks that constitute animal brains. An ANN is based on a collection of connected units ...
s, the rectifier or ReLU (rectified linear unit) activation function is an activation function defined as the positive part of its argument: : f(x) = x^+ = \max(0, x), where ''x'' is the input to a neuron. This is also known as a
ramp function The ramp function is a unary real function, whose graph is shaped like a ramp. It can be expressed by numerous definitions, for example "0 for negative inputs, output equals input for non-negative inputs". The term "ramp" can also be used for o ...
and is analogous to half-wave rectification in
electrical engineering Electrical engineering is an engineering discipline concerned with the study, design, and application of equipment, devices, and systems which use electricity, electronics, and electromagnetism. It emerged as an identifiable occupation in the l ...
. This activation function started showing up in the context of visual feature extraction in hierarchical neural networks starting in the late 1960s. It was later argued that it has strong
biological Biology is the scientific study of life. It is a natural science with a broad scope but has several unifying themes that tie it together as a single, coherent field. For instance, all organisms are made up of cells that process hereditary i ...
motivations and mathematical justifications. In 2011 it was found to enable better training of deeper networks, compared to the widely used activation functions prior to 2011, e.g., the logistic sigmoid (which is inspired by
probability theory Probability theory is the branch of mathematics concerned with probability. Although there are several different probability interpretations, probability theory treats the concept in a rigorous mathematical manner by expressing it through a set ...
; see
logistic regression In statistics, the logistic model (or logit model) is a statistical model that models the probability of an event taking place by having the log-odds for the event be a linear combination of one or more independent variables. In regression analy ...
) and its more practical counterpart, the
hyperbolic tangent In mathematics, hyperbolic functions are analogues of the ordinary trigonometric functions, but defined using the hyperbola rather than the circle. Just as the points form a circle with a unit radius, the points form the right half of the u ...
. The rectifier is, , the most popular activation function for deep neural networks. Rectified linear units find applications in
computer vision Computer vision is an interdisciplinary scientific field that deals with how computers can gain high-level understanding from digital images or videos. From the perspective of engineering, it seeks to understand and automate tasks that the human ...
and
speech recognition Speech recognition is an interdisciplinary subfield of computer science and computational linguistics that develops methodologies and technologies that enable the recognition and translation of spoken language into text by computers with the mai ...
Andrew L. Maas, Awni Y. Hannun, Andrew Y. Ng (2014)
Rectifier Nonlinearities Improve Neural Network Acoustic Models
using deep neural nets and
computational neuroscience Computational neuroscience (also known as theoretical neuroscience or mathematical neuroscience) is a branch of neuroscience which employs mathematical models, computer simulations, theoretical analysis and abstractions of the brain to u ...
.


Advantages

* Sparse activation: For example, in a randomly initialized network, only about 50% of hidden units are activated (have a non-zero output). * Better gradient propagation: Fewer vanishing gradient problems compared to sigmoidal activation functions that saturate in both directions. * Efficient computation: Only comparison, addition and multiplication. * Scale-invariant: \max(0, ax) = a \max(0, x) \text a \geq 0. Rectifying activation functions were used to separate specific excitation and unspecific inhibition in the neural abstraction pyramid, which was trained in a supervised way to learn several computer vision tasks. In 2011, the use of the rectifier as a non-linearity has been shown to enable training deep supervised neural networks without requiring unsupervised pre-training. Rectified linear units, compared to
sigmoid function A sigmoid function is a mathematical function having a characteristic "S"-shaped curve or sigmoid curve. A common example of a sigmoid function is the logistic function shown in the first figure and defined by the formula: :S(x) = \frac = \f ...
or similar activation functions, allow faster and effective training of deep neural architectures on large and complex datasets.


Potential problems

* Non-differentiable at zero; however, it is differentiable anywhere else, and the value of the
derivative In mathematics, the derivative of a function of a real variable measures the sensitivity to change of the function value (output value) with respect to a change in its argument (input value). Derivatives are a fundamental tool of calculus. ...
at zero can be arbitrarily chosen to be 0 or 1. * Not zero-centered. * Unbounded. * Dying ReLU problem: ReLU (rectified linear unit) neurons can sometimes be pushed into states in which they become inactive for essentially all inputs. In this state, no gradients flow backward through the neuron, and so the neuron becomes stuck in a perpetually inactive state and "dies". This is a form of the vanishing gradient problem. In some cases, large numbers of neurons in a network can become stuck in dead states, effectively decreasing the model capacity. This problem typically arises when the learning rate is set too high. It may be mitigated by using leaky ReLUs instead, which assign a small positive slope for ''x'' < 0; however, the performance is reduced.


Variants


Piecewise-linear variants


Leaky ReLU

Leaky ReLUs allow a small, positive gradient when the unit is not active. :f(x) = \begin x & \text x > 0, \\ 0.01x & \text. \end


Parametric ReLU

Parametric ReLUs (PReLUs) take this idea further by making the coefficient of leakage into a parameter that is learned along with the other neural-network parameters. :f(x) = \begin x & \text x > 0, \\ a x & \text. \end Note that for ''a'' ≤ 1, this is equivalent to : f(x) = \max(x, ax) and thus has a relation to "maxout" networks.


Other non-linear variants


Gaussian-error linear unit (GELU)

GELU is a smooth approximation to the rectifier: : f(x) = x \cdot \Phi(x), where Φ(''x'') is the
cumulative distribution function In probability theory and statistics, the cumulative distribution function (CDF) of a real-valued random variable X, or just distribution function of X, evaluated at x, is the probability that X will take a value less than or equal to x. Ever ...
of the standard
normal distribution In statistics, a normal distribution or Gaussian distribution is a type of continuous probability distribution for a real-valued random variable. The general form of its probability density function is : f(x) = \frac e^ The parameter \mu i ...
. This activation function is illustrated in the figure at the start of this article. It has a non-monotonic “bump” when ''x'' < 0 and serves as the default activation for models such as BERT.


SiLU

The SiLU (sigmoid linear unit) or swish function is another smooth approximation, first coined in the GELU paper: : f(x) = x \cdot \operatorname(x), where \operatorname(x) is the
sigmoid function A sigmoid function is a mathematical function having a characteristic "S"-shaped curve or sigmoid curve. A common example of a sigmoid function is the logistic function shown in the first figure and defined by the formula: :S(x) = \frac = \f ...
.


Softplus

A smooth approximation to the rectifier is the
analytic function In mathematics, an analytic function is a function that is locally given by a convergent power series. There exist both real analytic functions and complex analytic functions. Functions of each type are infinitely differentiable, but complex ...
:f(x) = \ln(1 + e^x), which is called the ''softplus'' or ''SmoothReLU'' function. For large negative x it is roughly \ln 1, so just above 0, while for large positive x it is roughly \ln(e^x), so just above x. A sharpness parameter k may be included: :f(x) = \frac. The derivative of softplus is the
logistic function A logistic function or logistic curve is a common S-shaped curve ( sigmoid curve) with equation f(x) = \frac, where For values of x in the domain of real numbers from -\infty to +\infty, the S-curve shown on the right is obtained, with th ...
. Starting from the parametric version, :f'(x) = \frac = \frac. The logistic
sigmoid function A sigmoid function is a mathematical function having a characteristic "S"-shaped curve or sigmoid curve. A common example of a sigmoid function is the logistic function shown in the first figure and defined by the formula: :S(x) = \frac = \f ...
is a smooth approximation of the derivative of the rectifier, the
Heaviside step function The Heaviside step function, or the unit step function, usually denoted by or (but sometimes , or ), is a step function, named after Oliver Heaviside (1850–1925), the value of which is zero for negative arguments and one for positive argumen ...
. The multivariable generalization of single-variable softplus is the
LogSumExp The LogSumExp (LSE) (also called RealSoftMax or multivariable softplus) function is a smooth maximum – a smooth approximation to the maximum function, mainly used by machine learning algorithms. It is defined as the logarithm of the sum of ...
with the first argument set to zero: : \operatorname^+(x_1, \dots, x_n) := \operatorname(0, x_1, \dots, x_n) = \ln\left(1 + e^ + \cdots + e^ \right). The LogSumExp function is : \operatorname(x_1, \dots, x_n) = \ln\left(e^ + \cdots + e^\right), and its gradient is the softmax; the softmax with the first argument set to zero is the multivariable generalization of the logistic function. Both LogSumExp and softmax are used in machine learning.


ELU

Exponential linear units try to make the mean activations closer to zero, which speeds up learning. It has been shown that ELUs can obtain higher classification accuracy than ReLUs. : f(x) = \begin x & \text x > 0, \\ a \left(e^x - 1\right) & \text, \end where a is a hyper-parameter to be tuned, and a \geq 0 is a constraint. The ELU can be viewed as a smoothed version of a shifted ReLU (SReLU), which has the form f(x) = \max(-a, x), given the same interpretation of a.


Mish

The mish function could also be used as a smooth approximation of the rectifier.. It is defined as : f(x) = x \tanh\big(\operatorname(x)\big), where \tanh(x) is the
hyperbolic tangent In mathematics, hyperbolic functions are analogues of the ordinary trigonometric functions, but defined using the hyperbola rather than the circle. Just as the points form a circle with a unit radius, the points form the right half of the u ...
, and \operatorname(x) is the
softplus In the context of artificial neural networks, the rectifier or ReLU (rectified linear unit) activation function is an activation function defined as the positive part of its argument: : f(x) = x^+ = \max(0, x), where ''x'' is the input to a neu ...
function. Mish is non-
monotonic In mathematics, a monotonic function (or monotone function) is a function between ordered sets that preserves or reverses the given order. This concept first arose in calculus, and was later generalized to the more abstract setting of ord ...
and self-gated. It was inspired by Swish, itself a variant of ReLU.


Metallic mean function

The function describing the
metallic mean The metallic means (also ratios or constants) of the successive natural numbers are the continued fractions: n + \cfrac = ;n,n,n,n,\dots= \frac. The golden ratio (1.618...) is the metallic mean between 1 and 2, while the silver ratio (2.414. ...
s is: :f(x) = \frac It satisfies the following properties: * f is
monotonic In mathematics, a monotonic function (or monotone function) is a function between ordered sets that preserves or reverses the given order. This concept first arose in calculus, and was later generalized to the more abstract setting of ord ...
* f is strictly
positive Positive is a property of positivity and may refer to: Mathematics and science * Positive formula, a logical formula not containing negation * Positive number, a number that is greater than 0 * Plus sign, the sign "+" used to indicate a posi ...
* f(0) = 1 * f'(x) = \frac \left( \frac + 1 \right) : f is C^\infty smooth : f is algebraic It also satisfies the following
asymptotic In analytic geometry, an asymptote () of a curve is a line such that the distance between the curve and the line approaches zero as one or both of the ''x'' or ''y'' coordinates tends to infinity. In projective geometry and related contexts, ...
properties: * f(x) \sim x as x \to \infty * f(x) \sim -\frac as x \to \infty


See also

*
Softmax function The softmax function, also known as softargmax or normalized exponential function, converts a vector of real numbers into a probability distribution of possible outcomes. It is a generalization of the logistic function to multiple dimensions, a ...
*
Sigmoid function A sigmoid function is a mathematical function having a characteristic "S"-shaped curve or sigmoid curve. A common example of a sigmoid function is the logistic function shown in the first figure and defined by the formula: :S(x) = \frac = \f ...
*
Tobit model In statistics, a tobit model is any of a class of regression models in which the observed range of the dependent variable is censored in some way. The term was coined by Arthur Goldberger in reference to James Tobin, who developed the model in 195 ...
* Layer (deep learning)


References

{{Differentiable computing Artificial neural networks