Parallel Distributed Processing

Connectionism refers to both an approach in the field of cognitive science that hopes to explain mental phenomena using artificial neural networks (ANN) and to a wide range of techniques and algorithms using ANNs in the context of artificial intelligence to build more intelligent machines. Connectionism presents a cognitive theory based on simultaneously occurring, distributed signal activity via connections that can be represented numerically, where learning occurs by modifying connection strengths based on experience.

Some advantages of the connectionist approach include its applicability to a broad array of functions, structural approximation to biological neurons, low requirements for innate structure, and capacity for graceful degradation. Some disadvantages include the difficulty in deciphering how ANNs process information, or account for the compositionality of mental representations, and a resultant difficulty explaining phenomena at a higher level.

The success of deep learning networks in the past decade has greatly increased the popularity of this approach, but the complexity and scale of such networks has brought with them increased interpretability problems. Connectionism is seen by many to offer an alternative to classical theories of mind based on symbolic computation, but the extent to which the two approaches are compatible has been the subject of much debate since their inception.

Basic principles

The central connectionist principle is that mental phenomena can be described by interconnected networks of simple and often uniform units. The form of the connections and the units can vary from model to model. For example, units in the network could represent neurons and the connections could represent synapses, as in the human brain.

Spreading activation

In most connectionist models, networks change over time. A closely related and very common aspect of connectionist models is ''activation''. At any time, a unit in the network has an activation, which is a numerical value intended to represent some aspect of the unit. For example, if the units in the model are neurons, the activation could represent the probability that the neuron would generate an action potential spike. Activation typically spreads to all the other units connected to it. Spreading activation is always a feature of neural network models, and it is very common in connectionist models used by cognitive psychologists.

Neural networks

Neural networks are by far the most commonly used connectionist model today. Though there are a large variety of neural network models, they almost always follow two basic principles regarding the mind:

# Any mental state can be described as an (N)-dimensional vector of numeric activation values over neural units in a network.
# Memory is created by modifying the strength of the connections between neural units. The connection strengths, or "weights", are generally represented as an N×M matrix.

Most of the variety among neural network models comes from:
* ''Interpretation of units'': Units can be interpreted as neurons or groups of neurons.
* ''Definition of activation'': Activation can be defined in a variety of ways. For example, in a Boltzmann machine, the activation is interpreted as the probability of generating an action potential spike, and is determined via a logistic function on the sum of the inputs to a unit.
* ''Learning algorithm'': Different networks modify their connections differently. In general, any mathematically defined change in connection weights over time is referred to as the "learning algorithm".

Connectionists are in agreement that recurrent neural networks (directed networks wherein connections of the network can form a directed cycle) are a better model of the brain than feedforward neural networks (directed networks with no cycles, called DAG). Many recurrent connectionist models also incorporate dynamical systems theory. Many researchers, such as the connectionist Paul Smolensky, have argued that connectionist models will evolve toward fully continuous, high-dimensional, non-linear, dynamic systems approaches.

Biological realism

Connectionist work in general does not need to be biologically realistic and therefore suffers from a lack of neuroscientific plausibility. However, the structure of neural networks is derived from that of biological neurons, and this parallel in low-level structure is often argued to be an advantage of connectionism in modeling cognitive structures compared with other approaches. One area where connectionist models are thought to be biologically implausible is with respect to error-propagation networks that are needed to support learning, but error propagation can explain some of the biologically-generated electrical activity seen at the scalp in event-related potentials such as the N400 and P600, and this provides some biological support for one of the key assumptions of connectionist learning procedures.

Learning

The weights in a neural network are adjusted according to some learning rule or algorithm, such as Hebbian learning. Thus, connectionists have created many sophisticated learning procedures for neural networks. Learning always involves modifying the connection weights. In general, these involve mathematical formulas to determine the change in weights when given sets of data consisting of activation vectors for some subset of the neural units. Several studies have been focused on designing teaching-learning methods based on connectionism.

By formalizing learning in such a way, connectionists have many tools. A very common strategy in connectionist learning methods is to incorporate gradient descent over an error surface in a space defined by the weight matrix. All gradient descent learning in connectionist models involves changing each weight by the partial derivative of the error surface with respect to the weight. Backpropagation (BP), first made popular in the 1980s, is probably the most commonly known connectionist gradient descent algorithm today.

Connectionism can be traced to ideas more than a century old, which were little more than speculation until the mid-to-late 20th century.

Parallel distributed processing

The prevailing connectionist approach today was originally known as parallel distributed processing (PDP). It was an artificial neural network approach that stressed the parallel nature of neural processing, and the distributed nature of neural representations. It provided a general mathematical framework for researchers to operate in. The framework involved eight major aspects:

* A set of ''processing units'', represented by a set of integers.
* An ''activation'' for each unit, represented by a vector of time-dependent functions.
* An ''output function'' for each unit, represented by a vector of functions on the activations.
* A ''pattern of connectivity'' among units, represented by a matrix of real numbers indicating connection strength.
* A ''propagation rule'' spreading the activations via the connections, represented by a function on the output of the units.
* An ''activation rule'' for combining inputs to a unit to determine its new activation, represented by a function on the current activation and propagation.
* A '' learning rule'' for modifying connections based on experience, represented by a change in the weights based on any number of variables.
* An ''environment'' that provides the system with experience, represented by sets of activation vectors for some subset of the units.

A lot of the research that led to the development of PDP was done in the 1970s, but PDP became popular in the 1980s with the release of the books ''Parallel Distributed Processing: Explorations in the Microstructure of Cognition - Volume 1 (foundations)'' and ''Volume 2 (Psychological and Biological Models)'', by James L. McClelland,