Intrinsic motivation, in the study of

artificial intelligence Artificial intelligence (AI) is the capability of computer, computational systems to perform tasks typically associated with human intelligence, such as learning, reasoning, problem-solving, perception, and decision-making. It is a field of re ...

and

robotics Robotics is the interdisciplinary study and practice of the design, construction, operation, and use of robots. Within mechanical engineering, robotics is the design and construction of the physical structures of robots, while in computer s ...

, is a mechanism for enabling artificial agents (including

robot A robot is a machine—especially one Computer program, programmable by a computer—capable of carrying out a complex series of actions Automation, automatically. A robot can be guided by an external control device, or the robot control, co ...

s) to exhibit inherently rewarding behaviours such as exploration and curiosity, grouped under the same term in the study of

psychology Psychology is the scientific study of mind and behavior. Its subject matter includes the behavior of humans and nonhumans, both consciousness, conscious and Unconscious mind, unconscious phenomena, and mental processes such as thoughts, feel ...

. Psychologists consider intrinsic motivation in humans to be the drive to perform an activity for inherent satisfaction – just for the fun or challenge of it.

Definition

intelligent agent In artificial intelligence, an intelligent agent is an entity that Machine perception, perceives its environment, takes actions autonomously to achieve goals, and may improve its performance through machine learning or by acquiring knowledge r ...

is intrinsically motivated to act if the information content alone, or the experience resulting from the action, is the motivating factor. Information content in this context is measured in the

information-theoretic Information theory is the mathematical study of the quantification, storage, and communication of information. The field was established and formalized by Claude Shannon in the 1940s, though early contributions were made in the 1920s through ...

sense of quantifying uncertainty. A typical intrinsic motivation is to search for unusual, surprising situations (exploration), in contrast to a typical extrinsic motivation such as the search for food (homeostasis). Extrinsic motivations are typically described in artificial intelligence as ''task-dependent'' or ''goal-directed''.

Origins in psychology

The study of intrinsic motivation in psychology and neuroscience began in the 1950s with some psychologists explaining exploration through drives to manipulate and explore, however, this homeostatic view was criticised by White. An alternative explanation from Berlyne in 1960 was the pursuit of an optimal balance between novelty and familiarity. Festinger described the difference between internal and external view of the world as dissonance that organisms are motivated to reduce. A similar view was expressed in the '70s by Kagan as the desire to reduce the incompatibility between cognitive structure and experience. In contrast to the idea of optimal incongruity,

Deci ''Deci'' (symbol d) is a decimal unit prefix in the metric system denoting a factor of one tenth. Proposed in 1793, and adopted in 1795, the prefix comes from the Latin , meaning "tenth". Since 1960, the prefix is part of the International System ...

and Ryan identified in the mid 80's an intrinsic motivation based on competence and

self-determination Self-determination refers to a people's right to form its own political entity, and internal self-determination is the right to representative government with full suffrage. Self-determination is a cardinal principle in modern international la ...

Computational models

An influential early computational approach to implement artificial curiosity in the early 1990s by Schmidhuber, has since been developed into a "Formal theory of creativity, fun, and intrinsic motivation”. Intrinsic motivation is often studied in the framework of computational

reinforcement learning Reinforcement learning (RL) is an interdisciplinary area of machine learning and optimal control concerned with how an intelligent agent should take actions in a dynamic environment in order to maximize a reward signal. Reinforcement learnin ...

(introduced by

Sutton Sutton (''south settlement'' or ''south town'' in Old English) may refer to: Places United Kingdom England In alphabetical order by county: * Sutton, Bedfordshire * Sutton, Berkshire, a List of United Kingdom locations: Stu-Sz#Su, location * S ...

and Barto), where the rewards that drive agent behaviour are intrinsically derived rather than externally imposed and must be learnt from the environment. Reinforcement learning is agnostic to how the reward is generated - an agent will learn a policy (action strategy) from the distribution of rewards afforded by actions and the environment. Each approach to intrinsic motivation in this scheme is essentially a different way of generating the reward function for the agent.

Curiosity vs. exploration

Intrinsically motivated artificial agents exhibit behaviour that resembles

curiosity Curiosity (from Latin , from "careful, diligent, curious", akin to "care") is a quality related to inquisitive thinking, such as exploration, investigation, and learning, evident in humans and other animals. Curiosity helps Developmental psyc ...

exploration Exploration is the process of exploring, an activity which has some Expectation (epistemic), expectation of Discovery (observation), discovery. Organised exploration is largely a human activity, but exploratory activity is common to most organis ...

Exploration Exploration is the process of exploring, an activity which has some Expectation (epistemic), expectation of Discovery (observation), discovery. Organised exploration is largely a human activity, but exploratory activity is common to most organis ...

in artificial intelligence and robotics has been extensively studied in reinforcement learning models, usually by encouraging the agent to explore as much of the environment as possible, to reduce uncertainty about the dynamics of the environment (learning the transition function) and how best to achieve its goals (learning the reward function). Intrinsic motivation, in contrast, encourages the agent to first explore aspects of the environment that confer more information, to seek out novelty. Recent work unifying state visit count exploration and intrinsic motivation has shown faster learning in a video game setting.

Types of models

Oudeyer and Kaplan have made a substantial contribution to the study of intrinsic motivation. They define intrinsic motivation based on Berlyne's theory, and divide approaches to the implementation of intrinsic motivation into three categories that broadly follow the roots in psychology: "knowledge-based models", "competence-based models" and "morphological models". Knowledge-based models are further subdivided into "information-theoretic" and "predictive". Baldassare and Mirolli present a similar typology, differentiating knowledge-based models between prediction-based and novelty-based.

Information-theoretic intrinsic motivation

The quantification of prediction and novelty to drive behaviour is generally enabled through the application of information-theoretic models, where agent state and strategy (policy) over time are represented by probability distributions describing a markov decision process and the cycle of perception and action treated as an information channel. These approaches claim biological feasibility as part of a family of bayesian approaches to brain function. The main criticism and difficulty of these models is the intractability of computing probability distributions over large discrete or continuous state spaces. Nonetheless, a considerable body of work has built up modelling the flow of information around the sensorimotor cycle, leading to de facto reward functions derived from the reduction of uncertainty, including most notably active inference, but also infotaxis, predictive information, and

empowerment Empowerment is the degree of autonomy and self-determination in people and in communities. This enables them to represent their interests in a responsible and self-determined way, acting on their own authority. It is the process of becoming strong ...

Competence-based models

Steels' autotelic principle is an attempt to formalise

flow (psychology) Flow in positive psychology, also known colloquially as being in the zone or locked in, is the mental state in which a person performing some activity is fully immersed in a feeling of energized Attention, focus, full involvement, and enjoyment ...

Achievement, affiliation and power models

Other intrinsic motives that have been modelled computationally include achievement, affiliation and power motivation. These motives can be implemented as functions of probability of success or incentive. Populations of agents can include individuals with different profiles of achievement, affiliation and power motivation, modelling population diversity and explaining why different individuals take different actions when faced with the same situation.

Beyond achievement, affiliation and power

A more recent computational theory of intrinsic motivation attempts to explain a large variety of psychological findings based on such motives. Notably this model of intrinsic motivation goes beyond just achievement, affiliation and power, by taking into consideration other important human motives. Empirical data from psychology were computationally simulated and accounted for using this model.

Intrinsically Motivated Learning

Intrinsically motivated (or curiosity-driven) learning is an emerging research topic in artificial intelligence and developmental robotics that aims to develop agents that can learn general skills or behaviours, that can be deployed to improve performance in extrinsic tasks, such as acquiring resources. Intrinsically motivated learning has been studied as an approach to autonomous lifelong learning in machines and open-ended learning in computer game characters. In particular, when the agent learns a meaningful abstract representation, a notion of distance between two representations can be used to gauge novelty, hence allowing for an efficient exploration of its environment. Despite the impressive success of

deep learning Deep learning is a subset of machine learning that focuses on utilizing multilayered neural networks to perform tasks such as classification, regression, and representation learning. The field takes inspiration from biological neuroscience a ...

in specific domains (e.g. AlphaGo), many in the field (e.g.

Gary Marcus Gary Fred Marcus (born 1970) is an American psychologist, cognitive scientist, and author, known for his research on the intersection of cognitive psychology, neuroscience, and artificial intelligence (AI). Marcus is professor ''emeritus'' of ps ...

) have pointed out that the ability to generalise remains a fundamental challenge in artificial intelligence. Intrinsically motivated learning, although promising in terms of being able to generate goals from the structure of the environment without externally imposed tasks, faces the same challenge of generalisation – how to reuse policies or action sequences, how to compress and represent continuous or complex state spaces and retain and reuse the salient features that have been learnt.

References

{{reflist, refs= {{cite journal , last1=Ryan , first1=Richard M , last2=Deci , first2=Edward L , date=2000 , title=Intrinsic and extrinsic motivations: Classic definitions and new directions , journal=Contemporary Educational Psychology , volume=25 , issue=1 , pages=54–67, doi=10.1006/ceps.1999.1020 , pmid=10620381 , s2cid=1098145 , hdl=20.500.12799/2958 , hdl-access=free {{cite book , last1=Oudeyer , first1=Pierre-Yves , last2=Kaplan , first2=Frederic , date=2008 , chapter=How can we define intrinsic motivation? , title=Proc. of the 8th Conf. on Epigenetic Robotics , volume=5 , pages=29–31 {{cite book , last1=Baldassarre , first1=Gianluca , last2=Mirolli , first2=Marco , title=Intrinsically Motivated Learning in Natural 1 and Artificial Systems , date=2013 , publisher=Springer , location=Rome, Italy , pages=1–14 , chapter=Intrinsically Motivated Learning Systems: An Overview {{cite journal , last1=Schmidhuber , first1=J , title=Formal theory of creativity, fun, and intrinsic motivation (1990-2010) , journal=IEEE Trans. Auton. Mental Dev. , date=2010 , volume=2 , issue=3 , pages=230–247, doi=10.1109/TAMD.2010.2056368 , s2cid=234198 {{cite journal , last1=White , first1=R. , title=Motivation reconsidered: The concept of competence. , journal=Psychological Review , date=1959 , volume=66 , issue=5 , pages=297–333, doi=10.1037/h0040934 , pmid=13844397 , s2cid=37385966 Berlyne, D.: Conflict, Arousal and Curiosity. McGraw-Hill, New York (1960) Festinger, L.: A theory of cognitive dissonance. Evanston, Row, Peterson (1957) Kagan, J.: Motives and development. Journal of Personality and Social Psychology 22, 51–66 Deci, E.L., Ryan, R.M.: Intrinsic motivation and self-determination in human behavior. Plenum, New York (1985) Barto, A., Singh, S., Chentanez, N.: Intrinsically motivated learn- ing of hierarchical collections of skills. In: ICDL 2004. Proceedings of the 3rd International Conference on Development and Learning, Salk Institute, San Diego (2004) {{cite journal , last1=Friston , first1=Karl , last2=Kilner , first2=James , last3=Harrison , first3=Lee , title=A free energy principle for the brain , journal=Journal of Physiology-Paris , publisher=Elsevier BV , volume=100 , issue=1–3 , year=2006 , issn=0928-4257 , doi=10.1016/j.jphysparis.2006.10.001 , pmid=17097864 , pages=70–87, s2cid=637885 , url=http://www.fil.ion.ucl.ac.uk/~karl/A%20free%20energy%20principle%20for%20the%20brain.pdf {{cite book , last1=Salge , first1=C , last2=Glackin , first2=C , last3=Polani , first3=D , chapter=Empowerment–An Introduction , date=2014 , editor-last=Prokopenko , editor-first=M , title=Guided Self-Organization: Inception. Emergence, Complexity and Computation , volume=9 , publisher=Springer , pages=67–114 , doi=10.1007/978-3-642-53734-9_4 , arxiv=1310.1863 , isbn=978-3-642-53733-2, s2cid=9662065 Barto, A.G.: Intrinsic motivation and reinforcement learning. In: Baldassarre, G., Mirolli, M. (eds.) Intrinsically Motivated Learning in Natural and Artificial Systems. Springer, Berlin (2012) Steels, Luc: The autotelic principle. In: Iida, F., Pfeifer, R., Steels, L., Kuniyoshi, Y. (eds.) Embodied Artificial Intelligence. LNCS (LNAI), vol. 3139, pp. 231–242. Springer, Heidelberg (2004) Ay, N., Bertschinger, N., Der, R., Güttler, F. and Olbrich, E. (2008), ‘Predictive information and explorative behavior of autonomous robots’, The European Physical Journal B 63(3), 329–339. Oudeyer, P. Y., & Kaplan, F. (2009). What is intrinsic motivation? A typology of computational approaches. Frontiers in Neurorobotics, 3(NOV). https://doi.org/10.3389/neuro.12.006.2007 Vergassola, M., Villermaux, E., & Shraiman, B. I. (2007). ‘Infotaxis’ as a strategy for searching without gradients. Nature, 445(7126), 406–409. https://doi.org/10.1038/nature05464 Kaplan, F. and Oudeyer, P. (2004). Maximizing learning progress: an internal reward system for development. Embodied artificial intelligence, pages 629–629. Singh, S., Barto, A. G., and Chentanez, N. (2005). Intrinsically motivated reinforcement learning. In Proceedings of the 18th Annual Conference on Neural Information Processing Systems (NIPS), Vancouver, B.C., Canada. Klyubin, A., Polani, D., and Nehaniv, C. (2008). Keep your options open: an information-based driving principle for sensorimotor systems. PLOS ONE, 3(12):e4018. https://dx.doi.org/10.1371%2Fjournal.pone.0004018 {{cite journal, last1=Biehl, first1=Martin, last2=Guckelsberger, first2=Christian, last3=Salge, first3=Christoph, last4=Smith, first4=Simón C., last5=Polani, first5=Daniel, title=Expanding the Active Inference Landscape: More Intrinsic Motivations in the Perception-Action Loop , journal=Frontiers in Neurorobotics , volume=12 , year=2018 , pages=45 , doi=10.3389/fnbot.2018.00045 , pmid=30214404, pmc=6125413, arxiv=1806.08083, issn=1662-5218 , doi-access=free Csikszentmihalyi, M. (2000). Beyond boredom and anxiety. Jossey-Bass. Lungarella, M., Metta, G., Pfeifer, R., and Sandini, G. (2003). Developmental robotics: a survey. Connect. Sci. 15, 151–190. doi: 10.1080/09540090310001655110 Barto, A. G. (2013). “Intrinsic motivation and reinforcement learning,” in Intrinsically Motivated Learning in Natural and Artificial Systems (Berlin; Heidelberg: Springer), 17–47 Martius, G., Der, R., and Ay, N. (2013). Information driven self-organization of complex robotic behaviors. PLOS ONE 8:e63400. doi: 10.1371/journal.pone.0063400 Mirolli, M., and Baldassarre, G. (2013). “Functions and mechanisms of intrinsic motivations,” in Intrinsically Motivated Learning in Natural and Artificial Systems, eds G. Baldassarre and M. Mirolli (Berlin; Heidelberg: Springer), 49–72 Santucci, V. G., Oudeyer, P. Y., Barto, A., & Baldassarre, G. (2020). Editorial: Intrinsically motivated open-ended learning in autonomous robots. Frontiers in Neurorobotics, 13(January), 2019–2021. https://doi.org/10.3389/fnbot.2019.00115 Sun, R., Bugrov, S, and Dai, D. (2022). A unified framework for interpreting a range of motivation-performance phenomena. Cognitive Systems Research, 71, 24–40. Tao, Ruo Yu and Francois-Lavet, Vincent and Pineau, Joelle (2020). Novelty search in representational space for sample efficient exploration. Neural Information Processing Systems, 2020. https://arxiv.org/abs/2009.13579 Bellemare, M. G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., & Munos, R. (2016). Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Processing Systems, 1479–1487. Thrun, S. B. (1992). Efficient Exploration in Reinforcement Learning. https://doi.org/10.1007/978-1-4899-7687-1_244 Merrick, K. E., Maher, M-L (2009). Motivated Reinforcement Learning: Curious Characters for Multiuser Games. Springer-Verlag Berlin Heidelberg, https://doi.org/10.1007/978-3-540-89187-1. Merrick, K. E. (2016). Computational Models of Motivation for Game-Playing Agents. Springer International Publishing, https://doi.org/10.1007/978-3-319-33459-2. Artificial intelligence Cognitive science Robotics engineering