A transformer is a
deep learning
Deep learning (also known as deep structured learning) is part of a broader family of machine learning methods based on artificial neural networks with representation learning. Learning can be supervised, semi-supervised or unsupervised.
...
model that adopts the mechanism of
self-attention, differentially weighting the significance of each part of the input data. It is used primarily in the fields of
natural language processing
Natural language processing (NLP) is an interdisciplinary subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to proc ...
(NLP)
and
computer vision
Computer vision is an interdisciplinary scientific field that deals with how computers can gain high-level understanding from digital images or videos. From the perspective of engineering, it seeks to understand and automate tasks that the human ...
(CV).
Like
recurrent neural networks (RNNs), transformers are designed to process sequential input data, such as natural language, with applications towards tasks such as
translation
Translation is the communication of the meaning of a source-language text by means of an equivalent target-language text. The English language draws a terminological distinction (which does not exist in every language) between ''transla ...
and
text summarization. However, unlike RNNs, transformers process the entire input all at once. The
attention mechanism provides context for any position in the input sequence. For example, if the input data is a natural language sentence, the transformer does not have to process one word at a time. This allows for more
parallelization
Parallel computing is a type of computation
Computation is any type of arithmetic or non-arithmetic calculation that follows a well-defined model (e.g., an algorithm).
Mechanical or electronic devices (or, historically, people) that perform ...
than RNNs and therefore reduces training times.
Transformers were introduced in 2017 by a team at
Google Brain
Google Brain is a deep learning artificial intelligence research team under the umbrella of Google AI, a research division at Google dedicated to artificial intelligence. Formed in 2011, Google Brain combines open-ended machine learning research ...
and are increasingly the model of choice for NLP problems,
replacing RNN models such as
long short-term memory
Long short-term memory (LSTM) is an artificial neural network used in the fields of artificial intelligence and deep learning. Unlike standard feedforward neural networks, LSTM has feedback connections. Such a recurrent neural network (RNN) c ...
(LSTM). The additional training parallelization allows training on larger datasets. This led to the development of
pretrained systems such as
BERT (Bidirectional Encoder Representations from Transformers) and
GPT (Generative Pre-trained Transformer), which were trained with large language datasets, such as the
Wikipedia
Wikipedia is a multilingual free online encyclopedia written and maintained by a community of volunteers, known as Wikipedians, through open collaboration and using a wiki-based editing system. Wikipedia is the largest and most-read refer ...
Corpus and
Common Crawl
Common Crawl is a nonprofit 501(c)(3) organization that crawls the web and freely provides its archives and datasets to the public. Common Crawl's web archive consists of petabytes of data collected since 2011. It completes crawls generally ever ...
, and can be fine-tuned for specific tasks.
Background
Before transformers, most state-of-the-art NLP systems relied on gated RNNs, such as
LSTMs and
gated recurrent units (GRUs), with added
attention mechanisms. Transformers also make use of attention mechanisms but, unlike RNNs, do not have a recurrent structure. This means that provided with enough training data, attention mechanisms alone can match the performance of RNNs with attention.
Sequential processing
Gated RNNs process tokens sequentially, maintaining a state vector that contains a representation of the data seen prior to the current token. To process the
th token, the model combines the state representing the sentence up to token
with the information of the new token to create a new state, representing the sentence up to token
. Theoretically, the information from one token can propagate arbitrarily far down the sequence, if at every point the state continues to encode contextual information about the token. In practice this mechanism is flawed: the
vanishing gradient problem leaves the model's state at the end of a long sentence without precise, extractable information about preceding tokens.
The dependency of token computations on results of previous token computations also makes it hard to parallelize computation on modern deep learning hardware. This can make the training of RNNs inefficient.
Self-Attention
These problems were addressed by attention mechanisms. Attention mechanisms let a model draw from the state at any preceding point along the sequence. The attention layer can access all previous states and weight them according to a learned measure of relevance, providing relevant information about far-away tokens.
A clear example of the value of attention is in
language translation
Translation is the communication of the meaning of a source-language text by means of an equivalent target-language text. The English language draws a terminological distinction (which does not exist in every language) between ''transl ...
, where context is essential to assign the meaning of a word in a sentence. In an English-to-French translation system, the first word of the French output most probably depends heavily on the first few words of the English input. However, in a classic LSTM model, in order to produce the first word of the French output, the model is given only the state vector after processing the ''last'' English word. Theoretically, this vector can encode information about the whole English sentence, giving the model all necessary knowledge. In practice, this information is often poorly preserved by the LSTM. An attention mechanism can be added to address this problem: the decoder is given access to the state vectors of every English input word, not just the last, and can learn attention weights that dictate how much to attend to each English input state vector.
When added to RNNs, attention mechanisms increase performance. The development of the Transformer architecture revealed that attention mechanisms were powerful in themselves and that sequential recurrent processing of data was not necessary to achieve the quality gains of RNNs with attention. Transformers use an attention mechanism without an RNN, processing all tokens at the same time and calculating attention weights between them in successive layers. Since the attention mechanism only uses information about other tokens from lower layers, it can be computed for all tokens in parallel, which leads to improved training speed.
Architecture
Input
The input text is parsed into tokens by a
byte pair encoding tokenizer, and each token is converted via a
word embedding into a vector. Then, positional information of the token is added to the word embedding.
Encoder–decoder architecture
Like earlier seq2seq models, the original Transformer model used an encoder–decoder architecture. The encoder consists of encoding layers that process the input iteratively one layer after another, while the decoder consists of decoding layers that do the same thing to the encoder's output.
The function of each encoder layer is to generate encodings that contain information about which parts of the inputs are relevant to each other. It passes its encodings to the next encoder layer as inputs. Each decoder layer does the opposite, taking all the encodings and using their incorporated contextual information to generate an output sequence. To achieve this, each encoder and decoder layer makes use of an attention mechanism.
For each part of the input, attention weighs the relevance of every other part and draws from them to produce the output.
Each decoder layer has an additional attention mechanism that draws information from the outputs of previous decoders, before the decoder layer draws information from the encodings.
Both the encoder and decoder layers have a
feed-forward neural network for additional processing of the outputs and contain residual connections and layer normalization steps.
Scaled dot-product attention
The transformer building blocks are scaled dot-product
attention
Attention is the behavioral and cognitive process of selectively concentrating on a discrete aspect of information, whether considered subjective or objective, while ignoring other perceivable information. William James (1890) wrote that "Att ...
units. When a sentence is passed into a transformer model, attention weights are calculated between every token simultaneously. The attention unit produces embeddings for every token in context that contain information about the token itself along with a weighted combination of other relevant tokens each weighted by its attention weight.
For each attention unit the transformer model learns three weight matrices; the query weights
, the key weights
, and the value weights
. For each token
, the input
word embedding is multiplied with each of the three weight matrices to produce a query vector
, a key vector
, and a value vector
. Attention weights are calculated using the query and key vectors: the attention weight
from token
to token
is the dot product between
and
. The attention weights are divided by the square root of the dimension of the key vectors,
, which stabilizes gradients during training, and passed through a
softmax which normalizes the weights. The fact that
and
are different matrices allows attention to be non-symmetric: if token
attends to token
(i.e.
is large), this does not necessarily mean that token
will attend to token
(i.e.
could be small). The output of the attention unit for token
is the weighted sum of the value vectors of all tokens, weighted by
, the attention from token
to each token.
The attention calculation for all tokens can be expressed as one large matrix calculation using the
softmax function
The softmax function, also known as softargmax or normalized exponential function, converts a vector of real numbers into a probability distribution of possible outcomes. It is a generalization of the logistic function to multiple dimensions, a ...
, which is useful for training due to computational matrix operation optimizations that quickly compute matrix operations. The matrices
,
and
are defined as the matrices where the
th rows are vectors
,
, and
respectively.
Multi-head attention
One set of
matrices is called an ''attention head'', and each layer in a transformer model has multiple attention heads. While each attention head attends to the tokens that are relevant to each token, with multiple attention heads the model can do this for different definitions of "relevance". In addition the influence field representing relevance can become progressively dilated in successive layers. Many transformer attention heads encode relevance relations that are meaningful to humans. For example, some attention heads can attend mostly to the next word, while others mainly attend from verbs to their direct objects. The computations for each attention head can be performed in parallel, which allows for fast processing. The outputs for the attention layer are concatenated to pass into the feed-forward neural network layers.
Encoder
Each encoder consists of two major components: a self-attention mechanism and a feed-forward neural network. The self-attention mechanism accepts input encodings from the previous encoder and weighs their relevance to each other to generate output encodings. The feed-forward neural network further processes each output encoding individually. These output encodings are then passed to the next encoder as its input, as well as to the decoders.
The first encoder takes positional information and
embeddings of the input sequence as its input, rather than encodings. The positional information is necessary for the transformer to make use of the order of the sequence, because no other part of the transformer makes use of this.
The encoder is bidirectional. Attention can be placed on tokens before and after the current token. Tokens are used instead of words to account for
polysemy
Polysemy ( or ; ) is the capacity for a sign (e.g. a symbol, a morpheme, a word, or a phrase) to have multiple related meanings. For example, a word can have several word senses. Polysemy is distinct from ''monosemy'', where a word has a singl ...
.
Positional encoding
The positional encoding is defined as a function of type
, where
is a positive even integer, by
where
. Here,
is a free parameter that should be significantly larger than the biggest
that would be input into the positional encoding function. In the original paper,
the authors chose
.
The function is in a simpler form when written as a complex function of type
where
.
The main reason the authors chose this as the positional encoding function is that it allows one to perform shifts as linear transformations:
where
is the distance one wishes to shift. This allows the transformer to take any encoded position, and find the encoding of the position 1-step-ahead, or 1-step-behind, etc, by a matrix multiplication.
By taking a linear sum, any convolution can also be implemented as linear transformations:
for any constants
. This allows the transformer to take any encoded position and find a linear sum of the encoded locations of its neighbors. This sum of encoded positions, when fed into the attention mechanism, would create attention weights on its neighbors, much like what happens in a
convolutional neural network
In deep learning, a convolutional neural network (CNN, or ConvNet) is a class of artificial neural network (ANN), most commonly applied to analyze visual imagery. CNNs are also known as Shift Invariant or Space Invariant Artificial Neural Netwo ...
language model. In the author's words, "we hypothesized it would allow the model to easily learn to attend by relative position".
In typical implementations, all operations are done over the real numbers, not the complex numbers, but since
complex multiplication can be implemented as real 2-by-2 matrix multiplication, this is a mere notational difference.
Other positional encoding schemes exist.
Decoder
Each decoder consists of three major components: a self-attention mechanism, an attention mechanism over the encodings, and a feed-forward neural network. The decoder functions in a similar fashion to the encoder, but an additional attention mechanism is inserted which instead draws relevant information from the encodings generated by the encoders. This mechanism can also be called the ''encoder-decoder attention''.
Like the first encoder, the first decoder takes positional information and embeddings of the output sequence as its input, rather than encodings. The transformer must not use the current or future output to predict an output, so the output sequence must be partially masked to prevent this reverse information flow.
This allows for
autoregressive text generation. For all attention heads, attention can't be placed on following tokens. The last decoder is followed by a final
linear transformation
In mathematics, and more specifically in linear algebra, a linear map (also called a linear mapping, linear transformation, vector space homomorphism, or in some contexts linear function) is a mapping V \to W between two vector spaces that pre ...
and
softmax layer, to produce the output probabilities over the vocabulary.
GPT has a decoder-only architecture.
Alternatives
Training transformer-based architectures can be expensive, especially for long inputs. Alternative architectures include the Reformer (which reduces the computational load from
to
), or models like ETC/BigBird (which can reduce it to
) where
is the length of the sequence. This is done using
locality-sensitive hashing and reversible layers.
Ordinary transformers require a memory size which is quadratic in the size of the context window. Attention Free Transformers reduce this to a linear dependence while still retaining the advantages of a transformer by linking the key to the value.
A benchmark for comparing transformer architectures was introduced in late 2020.
Training
Transformers typically undergo
self-supervised learning
Self-supervised learning (SSL) refers to a machine learning paradigm, and corresponding methods, for processing unlabelled data to obtain useful representations that can help with downstream learning tasks. The most salient thing about SSL metho ...
involving
unsupervised pretraining followed by
supervised fine-tuning. Pretraining is typically done on a larger dataset than fine-tuning, due to the limited availability of labeled training data. Tasks for pretraining and fine-tuning commonly include:
*
language modeling
* next-sentence prediction
*
question answering
Question answering (QA) is a computer science discipline within the fields of information retrieval and natural language processing (NLP), which is concerned with building systems that automatically answer questions posed by humans in a natural ...
*
reading comprehension
Reading comprehension is the ability to process text, understand its meaning, and to integrate with what the reader already knows. Fundamental skills required in efficient reading comprehension are knowing meaning of words, ability to understand ...
*
sentiment analysis
Sentiment analysis (also known as opinion mining or emotion AI) is the use of natural language processing, text analysis, computational linguistics, and biometrics to systematically identify, extract, quantify, and study affective states and subjec ...
*
paraphrasing
A paraphrase () is a restatement of the meaning of a text or passage using other words. The term itself is derived via Latin ', . The act of paraphrasing is also called ''paraphrasis''.
History
Although paraphrases likely abounded in oral tra ...
Applications
The transformer has had great success in
natural language processing
Natural language processing (NLP) is an interdisciplinary subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to proc ...
(NLP), for example the tasks of
machine translation
Machine translation, sometimes referred to by the abbreviation MT (not to be confused with computer-aided translation, machine-aided human translation or interactive translation), is a sub-field of computational linguistics that investigates ...
and
time series
In mathematics, a time series is a series of data points indexed (or listed or graphed) in time order. Most commonly, a time series is a sequence taken at successive equally spaced points in time. Thus it is a sequence of discrete-time data. Ex ...
prediction. Many pretrained models such as
GPT-2
Generative Pre-trained Transformer 2 (GPT-2) is an open-source artificial intelligence created by OpenAI in February 2019. GPT-2 translates text, answers questions, summarizes passages, and generates text output on a level that, while someti ...
,
GPT-3
Generative Pre-trained Transformer 3 (GPT-3) is an autoregressive language model that uses deep learning to produce human-like text. Given an initial text as prompt, it will produce text that continues the prompt.
The architecture is a standar ...
,
BERT, XLNet, and RoBERTa demonstrate the ability of transformers to perform a wide variety of such NLP-related tasks, and have the potential to find real-world applications.
These may include:
*
machine translation
Machine translation, sometimes referred to by the abbreviation MT (not to be confused with computer-aided translation, machine-aided human translation or interactive translation), is a sub-field of computational linguistics that investigates ...
*
document summarization
Automatic summarization is the process of shortening a set of data computationally, to create a subset (a Abstract (summary), summary) that represents the most important or relevant information within the original content. Artificial intelligence ...
*
document generation
*
named entity recognition
Named-entity recognition (NER) (also known as (named) entity identification, entity chunking, and entity extraction) is a subtask of information extraction that seeks to locate and classify named entities mentioned in unstructured text into pre ...
(NER)
*
biological sequence analysis
*
video understanding.
In 2020, it was shown that the transformer architecture, more specifically GPT-2, could be tuned to play chess. Transformers have been applied to image processing with results competitive with
convolutional neural networks
In deep learning, a convolutional neural network (CNN, or ConvNet) is a class of artificial neural network (ANN), most commonly applied to analyze visual imagery. CNNs are also known as Shift Invariant or Space Invariant Artificial Neural Networ ...
.
Implementations
The transformer model has been implemented in standard deep learning frameworks such as
TensorFlow
TensorFlow is a free and open-source software library for machine learning and artificial intelligence. It can be used across a range of tasks but has a particular focus on training and inference of deep neural networks. "It is machine learnin ...
and
PyTorch
PyTorch is a machine learning framework based on the Torch library, used for applications such as computer vision and natural language processing, originally developed by Meta AI and now part of the Linux Foundation umbrella. It is free and open ...
.
''Transformers'' is a library produced by
Hugging Face
Hugging Face, Inc. is an American company that develops tools for building applications using machine learning. It is most notable for its Transformers library built for natural language processing applications and its platform that allows users ...
that supplies transformer-based architectures and pretrained models.
See also
*
*
*
*
*
References
Further reading
* Hubert Ramsauer ''et al.'' (2020)
"Hopfield Networks is All You Need" preprint submitted for
ICLR 2021. ; see also authors
blog::– Discussion of the effect of a transformer layer as equivalent to a Hopfield update, bringing the input closer to one of the
fixed points (representable patterns) of a continuous-valued
Hopfield network
A Hopfield network (or Ising model of a neural network or Ising–Lenz–Little model) is a form of recurrent artificial neural network and a type of spin glass system popularised by John Hopfield in 1982 as described earlier by Little in 1974 b ...
* Alexander Rush
The Annotated transformer Harvard NLP group, 3 April 2018
{{Differentiable computing
Neural network architectures