In
deep learning
Deep learning is a subset of machine learning that focuses on utilizing multilayered neural networks to perform tasks such as classification, regression, and representation learning. The field takes inspiration from biological neuroscience a ...
, pruning is the practice of removing
parameter
A parameter (), generally, is any characteristic that can help in defining or classifying a particular system (meaning an event, project, object, situation, etc.). That is, a parameter is an element of a system that is useful, or critical, when ...
s from an existing
artificial neural network
In machine learning, a neural network (also artificial neural network or neural net, abbreviated ANN or NN) is a computational model inspired by the structure and functions of biological neural networks.
A neural network consists of connected ...
. The goal of this process is to reduce the size (parameter count) of the neural network (and therefore the
computational resource
In computational complexity theory, a computational resource is a resource used by some computational models in the solution of computational problems.
The simplest computational resources are computation time, the number of steps necessary to ...
s required to run it) whilst maintaining accuracy. This can be compared to the biological process of
synaptic pruning
Synaptic pruning is the process of synapse elimination or weakening. Though it occurs throughout the lifespan of a mammal, the most active period of synaptic pruning in the development of the nervous system occurs between early childhood and the o ...
which takes place in
mammalian
A mammal () is a vertebrate animal of the Class (biology), class Mammalia (). Mammals are characterised by the presence of milk-producing mammary glands for feeding their young, a broad neocortex region of the brain, fur or hair, and three ...
brains during development.
Node (neuron) pruning
A basic algorithm for pruning is as follows:
#Evaluate the importance of each neuron.
#Rank the neurons according to their importance (assuming there is a clearly defined measure for "importance").
#Remove the least important neuron.
#Check a termination condition (to be determined by the user) to see whether to continue pruning.
Edge (weight) pruning
Most work on neural network pruning focuses on removing weights, namely, setting their values to zero. Early work suggested to also change the values of non-pruned weights.
See also
*
Knowledge distillation
In machine learning, knowledge distillation or model distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have more knowledge ...
*
Neural Darwinism
References
Artificial neural networks
{{deep-learning-stub