machine learning Machine learning (ML) is a field of inquiry devoted to understanding and building methods that 'learn', that is, methods that leverage data to improve performance on some set of tasks. It is seen as a part of artificial intelligence. Machine ...

, grokking, or delayed generalization, is a transition to

generalization A generalization is a form of abstraction whereby common properties of specific instances are formulated as general concepts or claims. Generalizations posit the existence of a domain or set of elements, as well as one or more common character ...

that occurs many training iterations after the interpolation threshold, after many iterations of seemingly little progress, as opposed to the usual process where generalization occurs slowly and progressively once the interpolation threshold has been reached. The term derives from the word ''

grok ''Grok'' is a neologism coined by American writer Robert A. Heinlein for his 1961 science fiction novel ''Stranger in a Strange Land''. While the ''Oxford English Dictionary'' summarizes the meaning of ''grok'' as "to understand intuitively or ...

'' coined by Robert Heinlein in his novel ''

Stranger in a Strange Land ''Stranger in a Strange Land'' is a 1961 science fiction novel by American author Robert A. Heinlein. It tells the story of Valentine Michael Smith, a human who comes to Earth in early adulthood after being born on the planet Mars and raised by ...

''. Grokking can be understood as a

phase transition In chemistry, thermodynamics, and other related fields, a phase transition (or phase change) is the physical process of transition between one state of a medium and another. Commonly the term is used to refer to changes among the basic states ...

during the training process. While grokking has been thought of as largely a phenomenon of relatively shallow models, grokking has been observed in deep neural networks and non-neural models and is the subject of active research.

References

See also