In the context of
human–computer interaction
Human–computer interaction (HCI) is research in the design and the use of computer technology, which focuses on the interfaces between people ( users) and computers. HCI researchers observe the ways humans interact with computers and design ...
, a modality is the classification of a single independent channel of sensory
input/output
In computing, input/output (I/O, or informally io or IO) is the communication between an information processing system, such as a computer, and the outside world, possibly a human or another information processing system. Inputs are the signals ...
between a computer and a human.
A system is designated unimodal if it has only one modality implemented, and
multimodal if it has more than one.
[ When multiple modalities are available for some tasks or aspects of a task, the system is said to have overlapping modalities. If multiple modalities are available for a task, the system is said to have redundant modalities. Multiple modalities can be used in combination to provide complementary methods that may be redundant but convey information more effectively. Modalities can be generally defined in two forms: human-computer and computer-human modalities.
]
Computer–Human modalities
Computers utilize a wide range of technologies to communicate and send information to humans:
* Common modalities
** Vision
Vision, Visions, or The Vision may refer to:
Perception Optical perception
* Visual perception, the sense of sight
* Visual system, the physical mechanism of eyesight
* Computer vision, a field dealing with how computers can be made to gain und ...
– computer graphics typically through a screen
** Audition
An audition is a sample performance by an actor, singer, musician, dancer or other performer. It typically involves the performer displaying their talent through a previously memorized and rehearsed solo piece or by performing a work or piece ...
– various audio outputs
** Tactition – vibrations or other movement
* Uncommon modalities
** Gustation (taste)
** Olfaction (smell)
** Thermoception (heat)
** Nociception
Nociception (also nocioception, from Latin ''nocere'' 'to harm or hurt') is the sensory nervous system's process of encoding noxious stimuli. It deals with a series of events and processes required for an organism to receive a painful stimulus, ...
(pain)
** Equilibrioception
The sense of balance or equilibrioception is the perception of balance and spatial orientation. It helps prevent humans and nonhuman animals from falling over when standing or moving. Equilibrioception is the result of a number of sensory sys ...
(balance)
Any human sense can be used as a computer to human modality. However, the modalities of seeing
Seeing may refer to:
* Visual perception
* Astronomical seeing, the blurring effects of air turbulence in the atmosphere
* In the occult seeing refers to "the sight" or the ability to see auras or to predict the future; see fortune-telling
* ' ...
and hearing
Hearing, or auditory perception, is the ability to perceive sounds through an organ, such as an ear, by detecting vibrations as periodic changes in the pressure of a surrounding medium. The academic field concerned with hearing is audit ...
are the most commonly employed since they are capable of transmitting information at a higher speed than other modalities, 250 to 300 and 150 to 160 words per minute, respectively. Though not commonly implemented as computer-human modality, tactition can achieve an average of 125 wpm through the use of a refreshable Braille display
A refreshable braille display or braille terminal is an electro-mechanical device for displaying braille characters, usually by means of round-tipped pins raised through holes in a flat surface. Visually impaired computer users who cannot use a s ...
. Other more common forms of tactition are smartphone and game controller vibrations.
Human–computer modalities
Computers can be equipped with various types of input devices and sensors to allow them to receive information from humans. Common input devices are often interchangeable if they have a standardized method of communication with the computer and afford
Affordance is what the environment offers the individual. American psychologist James J. Gibson coined the term in his 1966 book, ''The Senses Considered as Perceptual Systems'', and it occurs in many of his earlier essays. However, his best-know ...
practical adjustments to the user. Certain modalities can provide a richer interaction depending on the context, and having options for implementation allows for more robust systems.
* Simple modalities
** Keyboard
** Pointing device
** Touchscreen
A touchscreen or touch screen is the assembly of both an input ('touch panel') and output ('display') device. The touch panel is normally layered on the top of an electronic visual display of an information processing system. The display is ofte ...
* Complex modalities
** Computer vision
Computer vision is an Interdisciplinarity, interdisciplinary scientific field that deals with how computers can gain high-level understanding from digital images or videos. From the perspective of engineering, it seeks to understand and automate t ...
** Speech recognition
Speech recognition is an interdisciplinary subfield of computer science and computational linguistics that develops methodologies and technologies that enable the recognition and translation of spoken language into text by computers with the ma ...
** Motion
In physics, motion is the phenomenon in which an object changes its position with respect to time. Motion is mathematically described in terms of displacement, distance, velocity, acceleration, speed and frame of reference to an observer and mea ...
** Orientation
Orientation may refer to:
Positioning in physical space
* Map orientation, the relationship between directions on a map and compass directions
* Orientation (housing), the position of a building with respect to the sun, a concept in building desi ...
With the increasing popularity of smartphones
A smartphone is a Mobile device, portable computer device that combines Mobile phone, mobile telephone and Mobile computing, computing functions into one unit. They are distinguished from feature phones by their stronger hardware capabilities ...
, the general public are becoming more comfortable with the more complex modalities. Speech recognition was a major selling point of the iPhone 4S and following Apple
An apple is an edible fruit produced by an apple tree (''Malus domestica''). Apple trees are cultivated worldwide and are the most widely grown species in the genus '' Malus''. The tree originated in Central Asia, where its wild ances ...
products, with the introduction of Siri
Siri ( ) is a virtual assistant that is part of Apple Inc.'s iOS, iPadOS, watchOS, macOS, tvOS, and audioOS operating systems. It uses voice queries, gesture based control, focus-tracking and a natural-language user interface to answer ques ...
.
Using multiple modalities
Having multiple modalities in a system gives more affordance
Affordance is what the environment offers the individual. American psychologist James J. Gibson coined the term in his 1966 book, ''The Senses Considered as Perceptual Systems'', and it occurs in many of his earlier essays. However, his best-kno ...
to users and can contribute to a more robust system. Having more also allows for greater accessibility for users who work more effectively with certain modalities. Multiple modalities can be used as backup when certain forms of communication are not possible. This is especially true in the case of redundant modalities in which two or more modalities are used to communicate the same information. Certain combinations of modalities can add to the expression of a computer-human or human-computer interaction because the modalities each may be more effective at expressing one form or aspect of information than others.
There are six types of cooperation between modalities, and they help define how a combination or fusion of modalities work together to convey information more effectively.
* Equivalence: information is presented in multiple ways and can be interpreted as the same information
* Specialization: when a specific kind of information is always processed through the same modality
* Redundancy: multiple modalities process the same information
* Complementarity: multiple modalities take separate information and merge it
* Transfer: a modality produces information that another modality consumes
* Concurrency: multiple modalities take in separate information that is not merged
Complementary-redundant systems are those which have multiple sensors to form one understanding or dataset, and the more effectively the information can be combined without duplicating data, the more effectively the modalities cooperate. Having multiple modalities for communication is common, particularly in smartphones, and often their implementations work together towards the same goal, for example gyroscopes and accelerometers working together to track movement.[
]
See also
*
*
*
*
References
{{DEFAULTSORT:Modality (Human-Computer Interaction)
Multimodal interaction