Multimodal sentiment analysis is a new dimension of the traditional text-based

sentiment analysis Sentiment analysis (also known as opinion mining or emotion AI) is the use of natural language processing, text analysis, computational linguistics, and biometrics to systematically identify, extract, quantify, and study affective states and subjec ...

, which goes beyond the analysis of texts, and includes other modalities such as audio and visual data. It can be bimodal, which includes different combinations of two modalities, or trimodal, which incorporates three modalities. With the extensive amount of

social media Social media are interactive media technologies that facilitate the creation and sharing of information, ideas, interests, and other forms of expression through virtual communities and networks. While challenges to the definition of ''social medi ...

data available online in different forms such as videos and images, the conventional text-based

has evolved into more complex models of multimodal sentiment analysis, which can be applied in the development of

virtual assistant An intelligent virtual assistant (IVA) or intelligent personal assistant (IPA) is a software agent that can perform tasks or services for an individual based on commands or questions. The term " chatbot" is sometimes used to refer to virtua ...

analysis Analysis ( : analyses) is the process of breaking a complex topic or substance into smaller parts in order to gain a better understanding of it. The technique has been applied in the study of mathematics and logic since before Aristotle (3 ...

of YouTube movie reviews,

of news videos, and

emotion recognition Emotion recognition is the process of identifying human emotion. People vary widely in their accuracy at recognizing the emotions of others. Use of technology to help people with emotion recognition is a relatively nascent research area. Genera ...

(sometimes known as

emotion Emotions are mental states brought on by neurophysiology, neurophysiological changes, variously associated with thoughts, feelings, behavioral responses, and a degree of pleasure or suffering, displeasure. There is currently no scientific ...

detection) such as depression monitoring, among others. Similar to the traditional

, one of the most basic task in multimodal sentiment analysis is

sentiment Sentiment may refer to: *Feelings, and emotions *Public opinion, also called sentiment *Sentimentality, an appeal to shallow, uncomplicated emotions at the expense of reason *Sentimental novel, an 18th-century literary genre *Market sentiment, opt ...

classification, which classifies different sentiments into categories such as positive, negative, or neutral. The complexity of

analyzing Analysis (plural, : analyses) is the process of breaking a complexity, complex topic or Substance theory, substance into smaller parts in order to gain a better understanding of it. The technique has been applied in the study of mathematics a ...

text, audio, and visual features to perform such a task requires the application of different fusion techniques, such as feature-level, decision-level, and hybrid fusion. The performance of these fusion techniques and the classification

algorithm In mathematics and computer science, an algorithm () is a finite sequence of rigorous instructions, typically used to solve a class of specific problems or to perform a computation. Algorithms are used as specifications for performing ...

s applied, are influenced by the type of textual, audio, and visual features employed in the analysis.

Features

Feature engineering Feature engineering or feature extraction or feature discovery is the process of using domain knowledge to extract features (characteristics, properties, attributes) from raw data. The motivation is to use these extra features to improve the qua ...

, which involves the selection of features that are fed into

machine learning Machine learning (ML) is a field of inquiry devoted to understanding and building methods that 'learn', that is, methods that leverage data to improve performance on some set of tasks. It is seen as a part of artificial intelligence. Machine ...

algorithms, plays a key role in the sentiment classification performance. In multimodal sentiment analysis, a combination of different textual, audio, and visual features are employed.

Textual features

Similar to the conventional text-based

, some of the most commonly used textual features in multimodal sentiment analysis are unigrams and n-grams, which are basically a sequence of words in a given textual document. These features are applied using bag-of-words or bag-of-concepts feature representations, in which words or concepts are represented as vectors in a suitable space.

Audio features

Sentiment Sentiment may refer to: *Feelings, and emotions *Public opinion, also called sentiment *Sentimentality, an appeal to shallow, uncomplicated emotions at the expense of reason *Sentimental novel, an 18th-century literary genre *Market sentiment, opt ...

and

characteristics are prominent in different

phonetic Phonetics is a branch of linguistics that studies how humans produce and perceive sounds, or in the case of sign languages, the equivalent aspects of sign. Linguists who specialize in studying the physical properties of speech are phoneticians. ...

and

prosodic In linguistics, prosody () is concerned with elements of speech that are not individual phonetic segments (vowels and consonants) but are properties of syllables and larger units of speech, including linguistic functions such as intonation, st ...

properties contained in audio features. Some of the most important audio features employed in multimodal sentiment analysis are mel-frequency cepstrum (MFCC), spectral centroid, spectral flux, beat histogram, beat sum, strongest beat, pause duration, and pitch. OpenSMILE and

Praat Praat (; , '' "talk"'') is a free computer software package for speech analysis in phonetics. It was designed, and continues to be developed, by Paul Boersma and David Weenink of the University of Amsterdam. It can run on a wide range of opera ...

are popular open-source toolkits for extracting such audio features.

Visual features

One of the main advantages of analyzing videos with respect to texts alone, is the presence of rich sentiment cues in visual data. Visual features include

facial expression A facial expression is one or more motions or positions of the muscles beneath the skin of the face. According to one set of controversial theories, these movements convey the emotional state of an individual to observers. Facial expressions are ...

s, which are of paramount importance in capturing sentiments and

s, as they are a main channel of forming a person's present state of mind. Specifically,

smile A smile is a facial expression formed primarily by flexing the muscles at the sides of the mouth. Some smiles include a contraction of the muscles at the corner of the eyes, an action known as a Duchenne smile. Among humans, a smile expresses ...

, is considered to be one of the most predictive visual cues in multimodal sentiment analysis. OpenFace is an open-source facial analysis toolkit available for extracting and understanding such visual features.

Fusion techniques

Unlike the traditional text-based

, multimodal sentiment analysis undergo a fusion process in which data from different modalities (text, audio, or visual) are fused and analyzed together. The existing approaches in multimodal sentiment analysis

data fusion Data fusion is the process of integrating multiple data sources to produce more consistent, accurate, and useful information than that provided by any individual data source. Data fusion processes are often categorized as low, intermediate, or hig ...

can be grouped into three main categories: feature-level, decision-level, and hybrid fusion, and the performance of the sentiment classification depends on which type of fusion technique is employed.

Feature-level fusion

Feature-level fusion (sometimes known as early fusion) gathers all the features from each

modality Modality may refer to: Humanities * Modality (theology), the organization and structure of the church, as distinct from sodality or parachurch organizations * Modality (music), in music, the subject concerning certain diatonic scales * Modaliti ...

(text, audio, or visual) and joins them together into a single feature vector, which is eventually fed into a classification algorithm. One of the difficulties in implementing this technique is the integration of the heterogeneous features.

Decision-level fusion

Decision-level fusion (sometimes known as late fusion), feeds data from each modality (text, audio, or visual) independently into its own classification algorithm, and obtains the final sentiment classification results by fusing each result into a single decision vector. One of the advantages of this fusion technique is that it eliminates the need to fuse heterogeneous data, and each

can utilize its most appropriate classification

Hybrid fusion

Hybrid fusion is a combination of feature-level and decision-level fusion techniques, which exploits complementary information from both methods during the classification process. It usually involves a two-step procedure wherein feature-level fusion is initially performed between two modalities, and decision-level fusion is then applied as a second step, to fuse the initial results from the feature-level fusion, with the remaining

Applications

Similar to text-based sentiment analysis, multimodal sentiment analysis can be applied in the development of different forms of

recommender system A recommender system, or a recommendation system (sometimes replacing 'system' with a synonym such as platform or engine), is a subclass of information filtering system that provide suggestions for items that are most pertinent to a particular ...

s such as in the analysis of user-generated videos of movie reviews and general product reviews, to predict the sentiments of customers, and subsequently create product or service recommendations. Multimodal sentiment analysis also plays an important role in the advancement of

s through the application of natural language processing (NLP) and

techniques. In the healthcare domain, multimodal sentiment analysis can be utilized to detect certain medical conditions such as

stress Stress may refer to: Science and medicine * Stress (biology), an organism's response to a stressor such as an environmental condition * Stress (linguistics), relative emphasis or prominence given to a syllable in a word, or to a word in a phrase ...

anxiety Anxiety is an emotion which is characterized by an unpleasant state of inner turmoil and includes feelings of dread over anticipated events. Anxiety is different than fear in that the former is defined as the anticipation of a future threat wh ...

, or depression. Multimodal sentiment analysis can also be applied in understanding the sentiments contained in video news programs, which is considered as a complicated and challenging domain, as sentiments expressed by reporters tend to be less obvious or neutral.

References

{{Reflist Natural language processing Affective computing Social media Machine learning Multimodal interaction