A sigmoid function is a
mathematical function
In mathematics, a function from a set to a set assigns to each element of exactly one element of .; the words map, mapping, transformation, correspondence, and operator are often used synonymously. The set is called the domain of the functi ...
having a characteristic "S"-shaped curve or sigmoid curve.
A common example of a sigmoid function is the
logistic function shown in the first figure and defined by the formula:
:
Other standard sigmoid functions are given in the
Examples section. In some fields, most notably in the context of
artificial neural network
Artificial neural networks (ANNs), usually simply called neural networks (NNs) or neural nets, are computing systems inspired by the biological neural networks that constitute animal brains.
An ANN is based on a collection of connected unit ...
s, the term "sigmoid function" is used as an alias for the logistic function.
Special cases of the sigmoid function include the
Gompertz curve
The Gompertz curve or Gompertz function is a type of mathematical model for a time series, named after Benjamin Gompertz (1779–1865). It is a sigmoid function which describes growth as being slowest at the start and end of a given time period. Th ...
(used in modeling systems that saturate at large values of x) and the
ogee curve (used in the
spillway
A spillway is a structure used to provide the controlled release of water downstream from a dam or levee, typically into the riverbed of the dammed river itself. In the United Kingdom, they may be known as overflow channels. Spillways ensure th ...
of some
dam
A dam is a barrier that stops or restricts the flow of surface water or underground streams. Reservoirs created by dams not only suppress floods but also provide water for activities such as irrigation, human consumption, industrial use ...
s). Sigmoid functions have domain of all
real number
In mathematics, a real number is a number that can be used to measure a ''continuous'' one-dimensional quantity such as a distance, duration or temperature. Here, ''continuous'' means that values can have arbitrarily small variations. Every ...
s, with return (response) value commonly
monotonically increasing but could be decreasing. Sigmoid functions most often show a return value (y axis) in the range 0 to 1. Another commonly used range is from −1 to 1.
A wide variety of sigmoid functions including the logistic and
hyperbolic tangent functions have been used as the
activation function
In artificial neural networks, the activation function of a node defines the output of that node given an input or set of inputs.
A standard integrated circuit can be seen as a digital network of activation functions that can be "ON" (1) or " ...
of
artificial neuron
An artificial neuron is a mathematical function conceived as a model of biological neurons, a neural network. Artificial neurons are elementary units in an artificial neural network. The artificial neuron receives one or more inputs (representing ...
s. Sigmoid curves are also common in statistics as
cumulative distribution functions (which go from 0 to 1), such as the integrals of the
logistic density, the
normal density, and
Student's ''t'' probability density functions. The logistic sigmoid function is invertible, and its inverse is the
logit function.
Definition
A sigmoid function is a
bounded,
differentiable
In mathematics, a differentiable function of one real variable is a function whose derivative exists at each point in its domain. In other words, the graph of a differentiable function has a non-vertical tangent line at each interior point in its ...
, real function that is defined for all real input values and has a non-negative derivative at each point
and exactly one
inflection point. A sigmoid "function" and a sigmoid "curve" refer to the same object.
Properties
In general, a sigmoid function is
monotonic
In mathematics, a monotonic function (or monotone function) is a function between ordered sets that preserves or reverses the given order. This concept first arose in calculus, and was later generalized to the more abstract setting of ord ...
, and has a first
derivative
In mathematics, the derivative of a function of a real variable measures the sensitivity to change of the function value (output value) with respect to a change in its argument (input value). Derivatives are a fundamental tool of calculus. ...
which is
bell shaped. Conversely, the
integral
In mathematics, an integral assigns numbers to functions in a way that describes displacement, area, volume, and other concepts that arise by combining infinitesimal data. The process of finding integrals is called integration. Along wit ...
of any continuous, non-negative, bell-shaped function (with one local maximum and no local minimum, unless degenerate) will be sigmoidal. Thus the
cumulative distribution functions for many common
probability distributions are sigmoidal. One such example is the
error function, which is related to the cumulative distribution function of a
normal distribution; another is the
arctan function, which is related to the cumulative distribution function of a
Cauchy distribution
The Cauchy distribution, named after Augustin Cauchy, is a continuous probability distribution. It is also known, especially among physicists, as the Lorentz distribution (after Hendrik Lorentz), Cauchy–Lorentz distribution, Lorentz(ian) fun ...
.
A sigmoid function is constrained by a pair of
horizontal asymptotes as
.
A sigmoid function is
convex
Convex or convexity may refer to:
Science and technology
* Convex lens, in optics
Mathematics
* Convex set, containing the whole line segment that joins points
** Convex polygon, a polygon which encloses a convex set of points
** Convex polytop ...
for values less than a particular point, and it is
concave
Concave or concavity may refer to:
Science and technology
* Concave lens
* Concave mirror
Mathematics
* Concave function, the negative of a convex function
* Concave polygon, a polygon which is not convex
* Concave set
* The concavity of a ...
for values greater than that point: in many of the examples here, that point is 0.
Examples
*
Logistic function
*
Hyperbolic tangent (shifted and scaled version of the logistic function, above)
*
Arctangent function
*
Gudermannian function
*
Error function
*
Generalised logistic function
*
Smoothstep function
* Some
algebraic function In mathematics, an algebraic function is a function that can be defined
as the root of a polynomial equation. Quite often algebraic functions are algebraic expressions using a finite number of terms, involving only the algebraic operations additi ...
s, for example
* and in a more general form
* Up to shifts and scaling, many sigmoids are special cases of
where
is the inverse of the negative
Box–Cox transformation, and
and
are shape parameters.
*
Smooth Interpolation normalized to (-1,1) and
is the slope at zero:
using the hyperbolic tangent mentioned above.
Applications
Many natural processes, such as those of complex system
learning curves, exhibit a progression from small beginnings that accelerates and approaches a climax over time. When a specific mathematical model is lacking, a sigmoid function is often used.
The
van Genuchten–Gupta model
The Van Genuchten–Gupta model is an inverted Sigmoid function, S-curve applicable to crop yield and soil salinity relations.M. Th. van Genuchten and S.K. Gupta, 1993. USDA-ARS, U.S. Salinity Laboratory 4500 Glenwood Drive, Riverside, California, ...
is based on an inverted S-curve and applied to the response of crop yield to
soil salinity.
Examples of the application of the logistic S-curve to the response of crop yield (wheat) to both the soil salinity and depth to
water table
The water table is the upper surface of the zone of saturation. The zone of saturation is where the pores and fractures of the ground are saturated with water. It can also be simply explained as the depth below which the ground is saturated.
T ...
in the soil are shown in
modeling crop response in agriculture.
In
artificial neural network
Artificial neural networks (ANNs), usually simply called neural networks (NNs) or neural nets, are computing systems inspired by the biological neural networks that constitute animal brains.
An ANN is based on a collection of connected unit ...
s, sometimes non-smooth functions are used instead for efficiency; these are known as
hard sigmoid In artificial intelligence, especially computer vision and artificial neural networks, a hard sigmoid is non-smooth function used in place of a sigmoid function. These retain the basic shape of a sigmoid, rising from 0 to 1, but using simpler functi ...
s.
In
audio signal processing
Audio signal processing is a subfield of signal processing that is concerned with the electronic manipulation of audio signals. Audio signals are electronic representations of sound waves— longitudinal waves which travel through air, consist ...
, sigmoid functions are used as
waveshaper transfer function
In engineering, a transfer function (also known as system function or network function) of a system, sub-system, or component is a mathematical function that theoretically models the system's output for each possible input. They are widely used ...
s to emulate the sound of
analog circuitry clipping
Clipping may refer to:
Words
* Clipping (morphology), the formation of a new word by shortening it, e.g. "ad" from "advertisement"
* Clipping (phonetics), shortening the articulation of a speech sound, usually a vowel
* Clipping (publications) ...
.
In
biochemistry
Biochemistry or biological chemistry is the study of chemical processes within and relating to living organisms. A sub-discipline of both chemistry and biology, biochemistry may be divided into three fields: structural biology, enzymology and ...
and
pharmacology, the
Hill
A hill is a landform that extends above the surrounding terrain. It often has a distinct summit.
Terminology
The distinction between a hill and a mountain is unclear and largely subjective, but a hill is universally considered to be not a ...
and
Hill–Langmuir equation
In biochemistry and pharmacology, the Hill equation refers to two closely related equations that reflect the binding of ligands to macromolecules, as a function of the ligand concentration. A ligand is "a substance that forms a complex with a bio ...
s are sigmoid functions.
In computer graphics and real-time rendering, some of the sigmoid functions are used to blend colors or geometry between two values, smoothly and without visible seams or discontinuities.
Titration curve
Titrations are often recorded on graphs called titration curves, which generally contain the volume of the titrant as the independent variable and the pH of the solution as the dependent variable (because it changes depending on the composition ...
s between strong acids and strong bases have a sigmoid shape due to the logarithmic nature of the
pH scale.
The logistic function can be calculated efficiently by utilizing
type III Unums.
See also
*
Heaviside step function
The Heaviside step function, or the unit step function, usually denoted by or (but sometimes , or ), is a step function, named after Oliver Heaviside (1850–1925), the value of which is zero for negative arguments and one for positive argum ...
*
Logistic regression
*
Logit
*
Softplus function
*
Soboleva modified hyperbolic tangent
*
Softmax function
The softmax function, also known as softargmax or normalized exponential function, converts a vector of real numbers into a probability distribution of possible outcomes. It is a generalization of the logistic function to multiple dimensions, a ...
*
Swish function
The swish function is a mathematical function defined as follows:
: \operatorname(x) = x \operatorname(\beta x) = \frac.
where β is either constant or a trainable parameter depending on the model. For β = 1, the function becomes e ...
*
Weibull distribution
In probability theory and statistics, the Weibull distribution is a continuous probability distribution. It is named after Swedish mathematician Waloddi Weibull, who described it in detail in 1951, although it was first identified by Maurice Re ...
*
Fermi–Dirac statistics
References
Further reading
* . (NB. In particular see "Chapter 4: Artificial Neural Networks" (in particular pp. 96–97) where Mitchell uses the word "logistic function" and the "sigmoid function" synonymously – this function he also calls the "squashing function" – and the sigmoid (aka logistic) function is used to compress the outputs of the "neurons" in multi-layer neural nets.)
* (NB. Properties of the sigmoid, including how it can shift along axes and how its domain may be transformed.)
External links
*
{{Differentiable computing
Elementary special functions
Artificial neural networks