|
Long Short-term Memory
Long short-term memory (LSTM) is an artificial neural network used in the fields of artificial intelligence and deep learning. Unlike standard feedforward neural networks, LSTM has feedback connections. Such a recurrent neural network (RNN) can process not only single data points (such as images), but also entire sequences of data (such as speech or video). For example, LSTM is applicable to tasks such as unsegmented, connected handwriting recognition, speech recognition, machine translation, robot control, video games, and healthcare. The name of LSTM refers to the analogy that a standard RNN has both "long-term memory" and "short-term memory". The connection weights and biases in the network change once per episode of training, analogous to how physiological changes in synaptic strengths store long-term memories; the activation patterns in the network change once per time-step, analogous to how the moment-to-moment change in electric firing patterns in the brain store shor ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
LSTM Cell
Long short-term memory (LSTM) is an artificial neural network used in the fields of artificial intelligence and deep learning. Unlike standard feedforward neural networks, LSTM has feedback connections. Such a recurrent neural network (RNN) can process not only single data points (such as images), but also entire sequences of data (such as speech or video). For example, LSTM is applicable to tasks such as unsegmented, connected handwriting recognition, speech recognition, machine translation, robot control, video games, and healthcare. The name of LSTM refers to the analogy that a standard RNN has both "long-term memory" and "short-term memory". The connection weights and biases in the network change once per episode of training, analogous to how physiological changes in synaptic strengths store long-term memories; the activation patterns in the network change once per time-step, analogous to how the moment-to-moment change in electric firing patterns in the brain store short- ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Vanishing Gradient Problem
In machine learning, the vanishing gradient problem is encountered when training artificial neural networks with gradient-based learning methods and backpropagation. In such methods, during each iteration of training each of the neural network's weights receives an update proportional to the partial derivative of the error function with respect to the current weight. The problem is that in some cases, the gradient will be vanishingly small, effectively preventing the weight from changing its value. In the worst case, this may completely stop the neural network from further training. As one example of the problem cause, traditional activation functions such as the hyperbolic tangent function have gradients in the range , and backpropagation computes gradients by the chain rule. This has the effect of multiplying of these small numbers to compute gradients of the early layers in an -layer network, meaning that the gradient (error signal) decreases exponentially with while the early ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Connectionist Temporal Classification (CTC)
Connectionist temporal classification (CTC) is a type of neural network output and associated scoring function, for training recurrent neural networks (RNNs) such as LSTM networks to tackle sequence problems where the timing is variable. It can be used for tasks like on-line handwriting recognition or recognizing phones in speech audio. CTC refers to the outputs and scoring, and is independent of the underlying neural network structure. It was introduced in 2006. The input is a sequence of observations, and the outputs are a sequence of labels, which can include blank outputs. The difficulty of training comes from there being many more observations than there are labels. For example in speech audio there can be multiple time slices which correspond to a single phoneme. Since we don't know the alignment of the observed sequence with the target labels we predict a probability distribution at each time step. A CTC network has a continuous output (e.g. softmax), which is fitted thro ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Spectral Radius
In mathematics, the spectral radius of a square matrix is the maximum of the absolute values of its eigenvalues. More generally, the spectral radius of a bounded linear operator is the supremum of the absolute values of the elements of its spectrum. The spectral radius is often denoted by . Definition Matrices Let be the eigenvalues of a matrix . The spectral radius of is defined as :\rho(A) = \max \left \. The spectral radius can be thought of as an infimum of all norms of a matrix. Indeed, on the one hand, \rho(A) \leqslant \, A\, for every natural matrix norm \, \cdot\, ; and on the other hand, Gelfand's formula states that \rho(A) = \lim_ \, A^k\, ^ . Both of these results are shown below. However, the spectral radius does not necessarily satisfy \, A\mathbf\, \leqslant \rho(A) \, \mathbf\, for arbitrary vectors \mathbf \in \mathbb^n . To see why, let r > 1 be arbitrary and consider the matrix : C_r = \begin 0 & r^ \\ r & 0 \end . The characteristic polynomial ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Backpropagation Through Time
Backpropagation through time (BPTT) is a gradient-based technique for training certain types of recurrent neural networks. It can be used to train Elman networks. The algorithm was independently derived by numerous researchers. Algorithm The training data for a recurrent neural network is an ordered sequence of k input-output pairs, \langle \mathbf_0,\mathbf_0 \rangle, \langle\mathbf_1,\mathbf_1 \rangle,\langle\mathbf_2,\mathbf_2\rangle,...,\langle\mathbf_,\mathbf_\rangle. An initial value must be specified for the hidden state \mathbf_0. Typically, a vector of all zeros is used for this purpose. BPTT begins by unfolding a recurrent neural network in time. The unfolded network contains k inputs and outputs, but every copy of the network shares the same parameters. Then the backpropagation algorithm is used to find the gradient of the cost with respect to all the network parameters. Consider an example of a neural network that contains a recurrent layer f and a feedforward ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Gradient Descent
In mathematics, gradient descent (also often called steepest descent) is a first-order iterative optimization algorithm for finding a local minimum of a differentiable function. The idea is to take repeated steps in the opposite direction of the gradient (or approximate gradient) of the function at the current point, because this is the direction of steepest descent. Conversely, stepping in the direction of the gradient will lead to a local maximum of that function; the procedure is then known as gradient ascent. Gradient descent is generally attributed to Augustin-Louis Cauchy, who first suggested it in 1847. Jacques Hadamard independently proposed a similar method in 1907. Its convergence properties for non-linear optimization problems were first studied by Haskell Curry in 1944, with the method becoming increasingly well-studied and used in the following decades. Description Gradient descent is based on the observation that if the multi-variable function F(\mathbf) is de ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Convolution
In mathematics (in particular, functional analysis), convolution is a mathematical operation on two functions ( and ) that produces a third function (f*g) that expresses how the shape of one is modified by the other. The term ''convolution'' refers to both the result function and to the process of computing it. It is defined as the integral of the product of the two functions after one is reflected about the y-axis and shifted. The choice of which function is reflected and shifted before the integral does not change the integral result (see commutativity). The integral is evaluated for all values of shift, producing the convolution function. Some features of convolution are similar to cross-correlation: for real-valued functions, of a continuous or discrete variable, convolution (f*g) differs from cross-correlation (f \star g) only in that either or is reflected about the y-axis in convolution; thus it is a cross-correlation of and , or and . For complex-valued fun ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Convolutional Neural Network
In deep learning, a convolutional neural network (CNN, or ConvNet) is a class of artificial neural network (ANN), most commonly applied to analyze visual imagery. CNNs are also known as Shift Invariant or Space Invariant Artificial Neural Networks (SIANN), based on the shared-weight architecture of the convolution kernels or filters that slide along input features and provide translation- equivariant responses known as feature maps. Counter-intuitively, most convolutional neural networks are not invariant to translation, due to the downsampling operation they apply to the input. They have applications in image and video recognition, recommender systems, image classification, image segmentation, medical image analysis, natural language processing, brain–computer interfaces, and financial time series. CNNs are regularized versions of multilayer perceptrons. Multilayer perceptrons usually mean fully connected networks, that is, each neuron in one layer is connected to a ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Peephole Long Short-Term Memory
A peephole, peekhole, spyhole, doorhole, magic eye, magic mirror or door viewer, is a small, round opening through a door from which a viewer on the inside of a dwelling may "peek" to see directly outside the door. The lenses are made and arranged in such a way that viewing is only possible in one direction. The opening is typically no larger than the diameter of a dime (). In a door, usually for apartments or hotel rooms, a peephole enables to see outside without opening the door nor revealing one's presence. Glass peepholes are often fitted with a fisheye lens to allow a wider field of view from the inside. ''Popular Science'', July 1950, p. 153, right-side. Preventing inside viewability Simple peepholes may allow people outside to see inside. A fisheye lens o ...[...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Hyperbolic Tangent
In mathematics, hyperbolic functions are analogues of the ordinary trigonometric functions, but defined using the hyperbola rather than the circle. Just as the points form a circle with a unit radius, the points form the right half of the unit hyperbola. Also, similarly to how the derivatives of and are and respectively, the derivatives of and are and respectively. Hyperbolic functions occur in the calculations of angles and distances in hyperbolic geometry. They also occur in the solutions of many linear differential equations (such as the equation defining a catenary), cubic equations, and Laplace's equation in Cartesian coordinates. Laplace's equations are important in many areas of physics, including electromagnetic theory, heat transfer, fluid dynamics, and special relativity. The basic hyperbolic functions are: * hyperbolic sine "" (), * hyperbolic cosine "" (),''Collins Concise Dictionary'', p. 328 from which are derived: * hyperbolic tangent "" (), * hy ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Sigmoid Function
A sigmoid function is a mathematical function having a characteristic "S"-shaped curve or sigmoid curve. A common example of a sigmoid function is the logistic function shown in the first figure and defined by the formula: :S(x) = \frac = \frac=1-S(-x). Other standard sigmoid functions are given in the Examples section. In some fields, most notably in the context of artificial neural networks, the term "sigmoid function" is used as an alias for the logistic function. Special cases of the sigmoid function include the Gompertz curve (used in modeling systems that saturate at large values of x) and the ogee curve (used in the spillway of some dams). Sigmoid functions have domain of all real numbers, with return (response) value commonly monotonically increasing but could be decreasing. Sigmoid functions most often show a return value (y axis) in the range 0 to 1. Another commonly used range is from −1 to 1. A wide variety of sigmoid functions including the logistic and ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Activation Function
In artificial neural networks, the activation function of a node defines the output of that node given an input or set of inputs. A standard integrated circuit can be seen as a digital network of activation functions that can be "ON" (1) or "OFF" (0), depending on input. This is similar to the linear perceptron in neural networks. However, only ''nonlinear'' activation functions allow such networks to compute nontrivial problems using only a small number of nodes, and such activation functions are called nonlinearities. Classification of activation functions The most common activation functions can be divided in three categories: ridge functions, radial functions and fold functions. An activation function f is saturating if \lim_ , \nabla f(v), = 0. It is nonsaturating if it is not saturating. Non-saturating activation functions, such as ReLU, may be better than saturating activation functions, as they don't suffer from vanishing gradient. Ridge activation functions ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
