Sentence Embedding

Sentence embedding is the collective name for a set of techniques in natural language processing (NLP) where sentences are mapped to vectors of real numbers.

Application

Sentence embedding is used by the deep learning software libraries PyTorch and TensorFlow.

Popular embeddings are based on the hidden layer outputs of transformer models like BERT, see SBERT. An alternative direction is to aggregate word embeddings, such those returned by Word2vec, into sentence embeddings. The most straightforward approach is to simply compute the average of word vectors, known as continuous bag-of-words (CBOW). However, more elaborate solutions based on word vector quantization have also been proposed. One such approach is the vector of locally aggregated word embeddings (VLAWE), which demonstrated performance improvements in downstream text classification tasks.

Evaluation

A way of testing sentence encodings is to apply them on Sentences Involving Compositional Knowledge (SICK) corpus
for both entailment (SICK-E) and relatedness (SICK-R).

In the best results are obtained using a BiLSTM network trained on th
Stanford Natural Language Inference (SNLI) Corpus
The Pearson correlation coefficient for SICK-R is 0.885 and the result for SICK-E is 86.3. A slight improvement over previous scores is presented in: SICK-R: 0.888 and SICK-E: 87.8 using a concatenation of bidirectional Gated recurrent unit.

See also

* Distributional semantics
* Word embedding

External links

InferSent sentence embeddings and training code

Universal Sentence Encoder

Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

References

{{Reflist

Language modeling
Artificial neural networks
Natural language processing
Computational linguistics