The wake-sleep algorithm is an

unsupervised learning Unsupervised learning is a type of algorithm that learns patterns from untagged data. The hope is that through mimicry, which is an important mode of learning in people, the machine is forced to build a concise representation of its world and t ...

algorithm for a

stochastic Stochastic (, ) refers to the property of being well described by a random probability distribution. Although stochasticity and randomness are distinct in that the former refers to a modeling approach and the latter refers to phenomena themselve ...

multilayer

neural network A neural network is a network or neural circuit, circuit of biological neurons, or, in a modern sense, an artificial neural network, composed of artificial neurons or nodes. Thus, a neural network is either a biological neural network, made up ...

. The algorithm adjusts the parameters so as to produce a good density estimator. There are two learning phases, the “wake” phase and the “sleep” phase, which are performed alternately. It was first designed as a model for brain functioning using variational Bayesian learning. After that, the algorithm was adapted to

machine learning Machine learning (ML) is a field of inquiry devoted to understanding and building methods that 'learn', that is, methods that leverage data to improve performance on some set of tasks. It is seen as a part of artificial intelligence. Machine ...

. It can be viewed as a way to train a

Helmholtz Machine Hermann Ludwig Ferdinand von Helmholtz (31 August 1821 – 8 September 1894) was a German physicist and physician who made significant contributions in several scientific fields, particularly hydrodynamic stability. The Helmholtz Association, ...

. It can also be used in Deep Belief Networks (DBN).

Description

The wake-sleep algorithm is visualized as a stack of layers containing representations of data. Layers above represent data from the layer below it. Actual data is placed below the bottom layer, causing layers on top of it to become gradually more abstract. Between each pair of layers there is a recognition weight and generative weight, which are trained to improve reliability during the algorithm runtime. The wake-sleep algorithm is

convergent Convergent is an adjective for things that converge. It is commonly used in mathematics and may refer to: *Convergent boundary, a type of plate tectonic boundary * Convergent (continued fraction) * Convergent evolution * Convergent series Converg ...

and can be stochastic if alternated appropriately.

Training

Training consists of two phases – the “wake” phase and the “sleep” phase.

The "wake" phase

Neurons are fired by recognition connections (from what would be input to what would be output). Generative connections (leading from outputs to inputs) are then modified to increase probability that they would recreate the correct activity in the layer below – closer to actual data from sensory input.

The "sleep" phase

The process is reversed in the “sleep” phase – neurons are fired by generative connections while recognition connections are being modified to increase probability that they would recreate the correct activity in the layer above – further to actual data from sensory input.

Potential risks

Variational Bayesian learning is based on

probabilities Probability is the branch of mathematics concerning numerical descriptions of how likely an event is to occur, or how likely it is that a proposition is true. The probability of an event is a number between 0 and 1, where, roughly speakin ...

. There is a chance that an approximation is performed with mistakes, damaging further data representations. Another downside pertains to complicated or corrupted data samples, making it difficult to infer a representational pattern. The wake-sleep algorithm has been suggested not to be powerful enough for the layers of the inference network in order to recover a good estimator of the posterior distribution of latent variables.{{cite arXiv, title = Reweighted Wake-Sleep, eprint= 1406.2751, date = 2014-06-10, first1 = Jörg, last1 = Bornschein, first2 = Yoshua, last2 = Bengio, class= cs.LG

References