An autoencoder is a type of

Sparse autoencoder

''CS294A Lecture notes'', ''72''(2011), 1-19. ::$s(\backslash rho,\; \backslash hat\backslash rho)\; =\; KL(\backslash rho\; ,\; ,\; \backslash hat)\; =\; \backslash rho\; \backslash log\; \backslash frac+(1-\; \backslash rho)\backslash log\; \backslash frac$ or the L1 loss, as $s(\backslash rho,\; \backslash hat\backslash rho)\; =\; ,\; \backslash rho-\; \backslash hat\backslash rho,$, or the L2 loss, as $s(\backslash rho,\; \backslash hat\backslash rho)\; =\; ,\; \backslash rho-\; \backslash hat\backslash rho,\; ^2$. Alternatively, the sparsity regularization loss may be defined without reference to any "desired sparsity", but simply force as much sparsity as possible. In this case, one can sparsity regularization loss as $$L\_(\backslash theta,\; \backslash phi)\; =\; \backslash mathbb\; \backslash mathbb\; E\_\backslash left;\; href="/html/ALL/s/\backslash sum\_\_w\_k\_\backslash .html"\; ;"title="\backslash sum\_\; w\_k\; \backslash ">h\_k\backslash ,\; \backslash right$$

Variational Autoencoder based Anomaly Detection using Reconstruction Probability

''Special Lecture on IE'', ''2'', 1-18. By learning to replicate the most salient features in the training data under some of the constraints described previously, the model is encouraged to learn to precisely reproduce the most frequently observed characteristics. When facing anomalies, the model should worsen its reconstruction performance. In most cases, only data with normal instances are used to train the autoencoder; in others, the frequency of anomalies is small compared to the observation set so that its contribution to the learned representation could be ignored. After training, the autoencoder will accurately reconstruct "normal" data, while failing to do so with unfamiliar anomalous data. Reconstruction error (the error between the original data and its low dimensional reconstruction) is used as an anomaly score to detect anomalies. Recent literature has however shown that certain autoencoding models can, counterintuitively, be very good at reconstructing anomalous examples and consequently not able to reliably perform anomaly detection.

artificial neural network
Artificial neural networks (ANNs), usually simply called neural networks (NNs) or neural nets, are computing systems inspired by the biological neural networks that constitute animal brains.
An ANN is based on a collection of connected units ...

used to learn efficient codings of unlabeled data ( unsupervised learning). The encoding is validated and refined by attempting to regenerate the input from the encoding. The autoencoder learns a representation (encoding) for a set of data, typically for dimensionality reduction, by training the network to ignore insignificant data (“noise”).
Variants exist, aiming to force the learned representations to assume useful properties. Examples are regularized autoencoders (''Sparse'', ''Denoising'' and ''Contractive''), which are effective in learning representations for subsequent classification tasks, and ''Variational'' autoencoders, with applications as generative models. Autoencoders are applied to many problems, including facial recognition, feature detection, anomaly detection and acquiring the meaning of words. Autoencoders are also generative models which can randomly generate new data that is similar to the input data (training data).
Mathematical principles

Definition

An autoencoder is defined by the following components:Two sets: the space of decoded messages $\backslash mathcal\; X$; the space of encoded messages $\backslash mathcal\; Z$. Almost always, both $\backslash mathcal\; X$ and $\backslash mathcal\; Z$ are Euclidean spaces, that is, $\backslash mathcal\; X\; =\; \backslash R^m,\; \backslash mathcal\; Z\; =\; \backslash R^n$ for some $m,\; n$.

Two parametrized families of functions: the encoder family $E\_\backslash phi:\backslash mathcal\; \backslash rightarrow\; \backslash mathcal$, parametrized by $\backslash phi$; the decoder family $D\_\backslash theta:\backslash mathcal\; \backslash rightarrow\; \backslash mathcal$, parametrized by $\backslash theta$.For any $x\backslash in\; \backslash mathcal\; X$, we usually write $z\; =\; E\_\backslash phi(x)$, and refer to it as the code, the latent variable, latent representation, latent vector, etc. Conversely, for any $z\backslash in\; \backslash mathcal\; Z$, we usually write $x\text{'}\; =\; D\_\backslash theta(z)$, and refer to it as the (decoded) message. Usually, both the encoder and the decoder are defined as multilayer perceptrons. For example, a one-layer-MLP encoder $E\_\backslash phi$ is: :$E\_\backslash phi(\backslash mathbf\; x)\; =\; \backslash sigma(Wx+b)$ where $\backslash sigma$ is an element-wise activation function such as a sigmoid function or a rectified linear unit, $W$ is a matrix called "weight", and $b$ is a vector called "bias".

Training an autoencoder

An autoencoder, by itself, is simply a tuple of two functions. To judge its ''quality'', we need a ''task''. A task is defined by a reference probability distribution $\backslash mu\_$ over $\backslash mathcal\; X$, and a "reconstruction quality" function $d:\; \backslash mathcal\; X\; \backslash times\; \backslash mathcal\; X\; \backslash to;\; href="/html/ALL/s/,\_\backslash infty.html"\; ;"title=",\; \backslash infty">,\; \backslash infty$least-squares
The method of least squares is a standard approach in regression analysis to approximate the solution of overdetermined systems (sets of equations in which there are more equations than unknowns) by minimizing the sum of the squares of the res ...

optimization:$$\backslash min\_\; L(\backslash theta,\; \backslash phi),\; \backslash text\; L(\backslash theta,\; \backslash phi)\; =\; \backslash frac\backslash sum\_^N\; \backslash ,\; x\_i\; -\; D\_\backslash theta(E\_\backslash phi(x\_i))\backslash ,\; \_2^2$$
Interpretation

An autoencoder has two main parts: an encoder that maps the message to a code, and a decoder that reconstructs the message from the code. An optimal autoencoder would perform as close to perfect reconstruction as possible, with "close to perfect" defined by the reconstruction quality function $d$. The simplest way to perform the copying task perfectly would be to duplicate the signal. To suppress this behavior, the code space $\backslash mathcal\; Z$ usually has fewer dimensions than the message space $\backslash mathcal$. Such an autoencoder is called ''undercomplete''. It can be interpreted as compressing the message, or reducing its dimensionality. At the limit of an ideal undercomplete autoencoder, every possible code $z$ in the code space is used to encode a message $x$ that really appears in the distribution $\backslash mu\_$, and the decoder is also perfect: $D\_\backslash theta(E\_\backslash phi(x))\; =\; x$. This ideal autoencoder can then be used to generate messages indistinguishable from real messages, by feeding its decoder arbitrary code $z$ and obtaining $D\_\backslash theta(z)$, which is a message that really appears in the distribution $\backslash mu\_$. If the code space $\backslash mathcal\; Z$ has dimension larger than (''overcomplete''), or equal to, the message space $\backslash mathcal$, or the hidden units are given enough capacity, an autoencoder can learn the identity function and become useless. However, experimental results found that overcomplete autoencoders might still learn useful features. In the ideal setting, the code dimension and the model capacity could be set on the basis of the complexity of the data distribution to be modeled. A standard way to do so is to add modifications to the basic autoencoder, to be detailed below.History

The autoencoder has also been called the autoassociator, or Diabolo network. Its first applications date to the 1980s. Their most traditional application was dimensionality reduction or feature learning, but the concept became widely used for learning generative models of data.Generating Faces with Torch, Boesen A., Larsen L. and Sonderby S.K., 2015 Some of the most powerful AIs in the 2010s involved autoencoders stacked inside deep neural networks.Variations

Regularized autoencoders

Various techniques exist to prevent autoencoders from learning the identity function and to improve their ability to capture important information and learn richer representations.Sparse autoencoder (SAE)

Inspired by thesparse coding
Neural coding (or Neural representation) is a neuroscience field concerned with characterising the hypothetical relationship between the stimulus and the individual or ensemble neuronal responses and the relationship among the electrical activity ...

hypothesis in neuroscience, sparse autoencoders are variants of autoencoders, such that the codes $D\_\backslash phi(x)$ for messages tend to be ''sparse codes'', that is, $D\_\backslash phi(x)$ is close to zero in most entries. Sparse autoencoders may include more (rather than fewer) hidden units than inputs, but only a small number of the hidden units are allowed to be active at the same time. Encouraging sparsity improves performance on classification tasks.
There are two main ways to enforce sparsity. One way is to simply clamp all but the highest-k activations of the latent code to zero. This is the k-sparse autoencoder.
The k-sparse autoencoder inserts the following "k-sparse function" in the latent layer of a standard autoencoder:$$f\_k(x\_1,\; ...,\; x\_n)\; =\; (x\_1\; b\_1,\; ...,\; x\_n\; b\_n)$$where $b\_i\; =\; 1$ if $,\; x\_i,$ ranks in the top k, and 0 otherwise.
Backpropagating through $f\_k$ is simple: set gradient to 0 for $b\_i\; =\; 0$ entries, and keep gradient for $b\_i=1$ entries. This is essentially a generalized ReLU function.
The other way is a relaxed version of the k-sparse autoencoder. Instead of forcing sparsity, we add a sparsity regularization loss, then optimize for$$\backslash min\_L(\backslash theta,\; \backslash phi)\; +\; \backslash lambda\; L\_\; (\backslash theta,\; \backslash phi)$$where $\backslash lambda\; >\; 0$ measures how much sparsity we want to enforce.
Let the autoencoder architecture have $K$ layers. To define a sparsity regularization loss, we need a "desired" sparsity $\backslash hat\; \backslash rho\_k$ for each layer, a weight $w\_k$ for how much to enforce each sparsity, and a function $s:;\; href="/html/ALL/s/,\_1.html"\; ;"title=",\; 1">,\; 1$Sparse autoencoder

''CS294A Lecture notes'', ''72''(2011), 1-19. ::$s(\backslash rho,\; \backslash hat\backslash rho)\; =\; KL(\backslash rho\; ,\; ,\; \backslash hat)\; =\; \backslash rho\; \backslash log\; \backslash frac+(1-\; \backslash rho)\backslash log\; \backslash frac$ or the L1 loss, as $s(\backslash rho,\; \backslash hat\backslash rho)\; =\; ,\; \backslash rho-\; \backslash hat\backslash rho,$, or the L2 loss, as $s(\backslash rho,\; \backslash hat\backslash rho)\; =\; ,\; \backslash rho-\; \backslash hat\backslash rho,\; ^2$. Alternatively, the sparsity regularization loss may be defined without reference to any "desired sparsity", but simply force as much sparsity as possible. In this case, one can sparsity regularization loss as $$L\_(\backslash theta,\; \backslash phi)\; =\; \backslash mathbb\; \backslash mathbb\; E\_\backslash left;\; href="/html/ALL/s/\backslash sum\_\_w\_k\_\backslash .html"\; ;"title="\backslash sum\_\; w\_k\; \backslash ">h\_k\backslash ,\; \backslash right$$

Denoising autoencoder (DAE)

Denoising autoencoders (DAE) try to achieve a ''good'' representation by changing the ''reconstruction criterion''. A DAE is defined by adding a noise process to the standard autoencoder. A noise process is defined by a probability distribution $\backslash mu\_T$ over functions $T:\backslash mathcal\; X\; \backslash to\; \backslash mathcal\; X$. That is, the function $T$ takes a message $x\backslash in\; \backslash mathcal\; X$, and corrupts it to a noisy version $T(x)$. The function $T$ is selected randomly, with a probability distribution $\backslash mu\_T$. Given a task $(\backslash mu\_,\; d)$, the problem of training a DAE is the optimization problem:$$\backslash min\_L(\backslash theta,\; \backslash phi)\; =\; \backslash mathbb\; \backslash mathbb\; E\_;\; href="/html/ALL/s/(x,\_(D\_\backslash theta\backslash circ\_E\_\backslash phi\_\backslash circ\_T)(x)).html"\; ;"title="(x,\; (D\_\backslash theta\backslash circ\; E\_\backslash phi\; \backslash circ\; T)(x))">(x,\; (D\_\backslash theta\backslash circ\; E\_\backslash phi\; \backslash circ\; T)(x))$$Gaussian noise
Gaussian noise, named after Carl Friedrich Gauss, is a term from signal processing theory denoting a kind of signal noise that has a probability density function (pdf) equal to that of the normal distribution (which is also known as the Gaussia ...

,
* masking noise (a fraction of the input is randomly chosen and set to 0)
* salt-and-pepper noise (a fraction of the input is randomly chosen and randomly set to its minimum or maximum value).
Contractive autoencoder (CAE)

A contractive autoencoder adds the contractive regularization loss to the standard autoencoder loss:$$\backslash min\_L(\backslash theta,\; \backslash phi)\; +\; \backslash lambda\; L\_\; (\backslash theta,\; \backslash phi)$$where $\backslash lambda\; >\; 0$ measures how much contractive-ness we want to enforce. The contractive regularization loss itself is defined as the expected Frobenius norm of the Jacobian matrix of the encoder activations with respect to the input:$$L\_(\backslash theta,\; \backslash phi)\; =\; \backslash mathbb\; E\_\; \backslash ,\; \backslash nabla\_x\; E\_\backslash phi(x)\; \backslash ,\; \_F^2$$To understand what $L\_$ measures, note the fact$$\backslash ,\; E\_\backslash phi(x\; +\; \backslash delta\; x)\; -\; E\_\backslash phi(x)\backslash ,\; \_2\; \backslash leq\; \backslash ,\; \backslash nabla\_x\; E\_\backslash phi(x)\; \backslash ,\; \_F\; \backslash ,\; \backslash delta\; x\backslash ,\; \_2$$for any message $x\backslash in\; \backslash mathcal\; X$, and small variation $\backslash delta\; x$ in it. Thus, if $\backslash ,\; \backslash nabla\_x\; E\_\backslash phi(x)\; \backslash ,\; \_F^2$ is small, it means that a small neighborhood of the message maps to a small neighborhood of its code. This is a desired property, as it means small variation in the message leads to small, perhaps even zero, variation in its code, like how two pictures may look the same even if they are not exactly the same. The DAE can be understood as an infinitesimal limit of CAE: in the limit of small Gaussian input noise, DAEs make the reconstruction function resist small but finite-sized input perturbations, while CAEs make the extracted features resist infinitesimal input perturbations.Minimal description length autoencoder

Concrete autoencoder

The concrete autoencoder is designed for discrete feature selection. A concrete autoencoder forces the latent space to consist only of a user-specified number of features. The concrete autoencoder uses a continuous relaxation of the categorical distribution to allow gradients to pass through the feature selector layer, which makes it possible to use standard backpropagation to learn an optimal subset of input features that minimize reconstruction loss.Variational autoencoder (VAE)

Variational autoencoders (VAEs) belong to the families ofvariational Bayesian methods
Variational Bayesian methods are a family of techniques for approximating intractable integrals arising in Bayesian inference and machine learning. They are typically used in complex statistical models consisting of observed variables (usually ...

. Despite the architectural similarities with basic autoencoders, VAEs are architecture with different goals and with a completely different mathematical formulation. The latent space is in this case composed by a mixture of distributions instead of a fixed vector.
Given an input dataset $x$ characterized by an unknown probability function $P(x)$ and a multivariate latent encoding vector $z$, the objective is to model the data as a distribution $p\_\backslash theta(x)$, with $\backslash theta$ defined as the set of the network parameters so that $p\_\backslash theta(x)\; =\; \backslash int\_p\_\backslash theta(x,z)dz$.
Advantages of depth

Autoencoders are often trained with a single layer encoder and a single layer decoder, but using many-layered (deep) encoders and decoders offers many advantages. * Depth can exponentially reduce the computational cost of representing some functions. * Depth can exponentially decrease the amount of training data needed to learn some functions. * Experimentally, deep autoencoders yield better compression compared to shallow or linear autoencoders.Training

Geoffrey Hinton
Geoffrey Everest Hinton One or more of the preceding sentences incorporates text from the royalsociety.org website where: (born 6 December 1947) is a British-Canadian cognitive psychologist and computer scientist, most noted for his work on ...

developed the deep belief network technique for training many-layered deep autoencoders. His method involves treating each neighbouring set of two layers as a restricted Boltzmann machine so that pretraining approximates a good solution, then using backpropagation to fine-tune the results.
Researchers have debated whether joint training (i.e. training the whole architecture together with a single global reconstruction objective to optimize) would be better for deep auto-encoders. A 2015 study showed that joint training learns better data models along with more representative features for classification as compared to the layerwise method. However, their experiments showed that the success of joint training depends heavily on the regularization strategies adopted.
Applications

The two main applications of autoencoders are dimensionality reduction and information retrieval, but modern variations have been applied to other tasks.Dimensionality reduction

Dimensionality reduction was one of the first deep learning applications. For Hinton's 2006 study, he pretrained a multi-layer autoencoder with a stack of RBMs and then used their weights to initialize a deep autoencoder with gradually smaller hidden layers until hitting a bottleneck of 30 neurons. The resulting 30 dimensions of the code yielded a smaller reconstruction error compared to the first 30 components of a principal component analysis (PCA), and learned a representation that was qualitatively easier to interpret, clearly separating data clusters. Representing dimensions can improve performance on tasks such as classification. Indeed, the hallmark of dimensionality reduction is to place semantically related examples near each other.Principal component analysis

If linear activations are used, or only a single sigmoid hidden layer, then the optimal solution to an autoencoder is strongly related to principal component analysis (PCA). The weights of an autoencoder with a single hidden layer of size $p$ (where $p$ is less than the size of the input) span the same vector subspace as the one spanned by the first $p$ principal components, and the output of the autoencoder is an orthogonal projection onto this subspace. The autoencoder weights are not equal to the principal components, and are generally not orthogonal, yet the principal components may be recovered from them using the singular value decomposition. However, the potential of autoencoders resides in their non-linearity, allowing the model to learn more powerful generalizations compared to PCA, and to reconstruct the input with significantly lower information loss.Information retrieval

Information retrieval
Information retrieval (IR) in computing and information science is the process of obtaining information system resources that are relevant to an information need from a collection of those resources. Searches can be based on full-text or other ...

benefits particularly from dimensionality reduction in that search can become more efficient in certain kinds of low dimensional spaces. Autoencoders were indeed applied to semantic hashing, proposed by Salakhutdinov and Hinton in 2007. By training the algorithm to produce a low-dimensional binary code, all database entries could be stored in a hash table
In computing, a hash table, also known as hash map, is a data structure that implements an associative array or dictionary. It is an abstract data type that maps keys to values. A hash table uses a hash function to compute an ''index'', al ...

mapping binary code vectors to entries. This table would then support information retrieval by returning all entries with the same binary code as the query, or slightly less similar entries by flipping some bits from the query encoding.
Anomaly detection

Another application for autoencoders is anomaly detection.An, J., & Cho, S. (2015)Variational Autoencoder based Anomaly Detection using Reconstruction Probability

''Special Lecture on IE'', ''2'', 1-18. By learning to replicate the most salient features in the training data under some of the constraints described previously, the model is encouraged to learn to precisely reproduce the most frequently observed characteristics. When facing anomalies, the model should worsen its reconstruction performance. In most cases, only data with normal instances are used to train the autoencoder; in others, the frequency of anomalies is small compared to the observation set so that its contribution to the learned representation could be ignored. After training, the autoencoder will accurately reconstruct "normal" data, while failing to do so with unfamiliar anomalous data. Reconstruction error (the error between the original data and its low dimensional reconstruction) is used as an anomaly score to detect anomalies. Recent literature has however shown that certain autoencoding models can, counterintuitively, be very good at reconstructing anomalous examples and consequently not able to reliably perform anomaly detection.

Image processing

The characteristics of autoencoders are useful in image processing. One example can be found in lossy image compression, where autoencoders outperformed other approaches and proved competitive against JPEG 2000. Another useful application of autoencoders in image preprocessing is image denoising. Autoencoders found use in more demanding contexts such asmedical imaging
Medical imaging is the technique and process of imaging the interior of a body for clinical analysis and medical intervention, as well as visual representation of the function of some organs or tissues (physiology). Medical imaging seeks to rev ...

where they have been used for image denoising as well as super-resolution. In image-assisted diagnosis, experiments have applied autoencoders for breast cancer
Breast cancer is cancer that develops from breast tissue. Signs of breast cancer may include a lump in the breast, a change in breast shape, dimpling of the skin, milk rejection, fluid coming from the nipple, a newly inverted nipple, or a ...

detection and for modelling the relation between the cognitive decline of Alzheimer's disease
Alzheimer's disease (AD) is a neurodegenerative disease that usually starts slowly and progressively worsens. It is the cause of 60–70% of cases of dementia. The most common early symptom is difficulty in remembering recent events. As ...

and the latent features of an autoencoder trained with MRI.
Drug discovery

In 2019 molecules generated with variational autoencoders were validated experimentally in mice.Popularity prediction

Recently, a stacked autoencoder framework produced promising results in predicting popularity of social media posts, which is helpful for online advertising strategies.Machine translation

Autoencoders have been applied tomachine translation
Machine translation, sometimes referred to by the abbreviation MT (not to be confused with computer-aided translation, machine-aided human translation or interactive translation), is a sub-field of computational linguistics that investigates ...

, which is usually referred to as neural machine translation (NMT). Unlike traditional autoencoders, the output does not match the input - it is in another language. In NMT, texts are treated as sequences to be encoded into the learning procedure, while on the decoder side sequences in the target language(s) are generated. Language
Language is a structured system of communication. The structure of a language is its grammar and the free components are its vocabulary. Languages are the primary means by which humans communicate, and may be conveyed through a variety of ...

-specific autoencoders incorporate further linguistic
Linguistics is the scientific study of human language. It is called a scientific study because it entails a comprehensive, systematic, objective, and precise analysis of all aspects of language, particularly its nature and structure. Linguis ...

features into the learning procedure, such as Chinese decomposition features. Machine translation is rarely still done with autoencoders, but rather transformer
A transformer is a passive component that transfers electrical energy from one electrical circuit to another circuit, or multiple circuits. A varying current in any coil of the transformer produces a varying magnetic flux in the transformer's c ...

networks.
See also

* Representation learning * Sparse dictionary learning * Deep learningReferences

{{Noise Neural network architectures Unsupervised learning Dimension reduction