Mixture of experts (MoE) refers to a
machine learning technique where multiple expert networks (learners) are used to divide a problem space into homogeneous regions. It differs from
ensemble techniques in that typically only a few, or 1, expert model will be run, rather than combining results from all models.
An example from
computer vision
Computer vision is an interdisciplinary scientific field that deals with how computers can gain high-level understanding from digital images or videos. From the perspective of engineering, it seeks to understand and automate tasks that the hum ...
is combining one
neural network
A neural network is a network or circuit of biological neurons, or, in a modern sense, an artificial neural network, composed of artificial neurons or nodes. Thus, a neural network is either a biological neural network, made up of biological ...
model for human detection with another for
pose estimation.
Hierarchical mixture
If the output is conditioned on multiple levels of (probabilistic) gating functions, the mixture is called a hierarchical mixture of experts.
A gating network decides which expert to use for each input region. Learning thus consists of learning the parameters of:
* individual learners and
* gating network.
Applications
Meta
Meta (from the Greek μετά, '' meta'', meaning "after" or "beyond") is a prefix meaning "more comprehensive" or "transcending".
In modern nomenclature, ''meta''- can also serve as a prefix meaning self-referential, as a field of study or ende ...
uses MoE in its NLLB-200 system. It uses multiple MoE models that share capacity for use by low-resource
language models with relatively little data.
References
Extra reading
*{{cite journal, last1=Masoudnia, first1=Saeed, last2=Ebrahimpour, first2=Reza, title=Mixture of experts: a literature survey, journal=Artificial Intelligence Review, date=12 May 2012, volume=42, issue=2, pages=275–293, doi=10.1007/s10462-012-9338-y, s2cid=3185688
Machine learning algorithms