Differentiable programming is a
Programming paradigms are a way to classify programming languages based on their features. Languages can be classified into multiple paradigms.
Some paradigms are concerned mainly with implications for the execution model of the language, suc ...
in which a numeric computer program can be differentiated
automatic differentiationIn mathematics
Mathematics (from Ancient Greek, Greek: ) includes the study of such topics as quantity (number theory), mathematical structure, structure (algebra), space (geometry), and calculus, change (mathematical analysis, analysis). It ha ...
This allows for
of parameters in the program, often via
Gradient descent (also often called steepest descent) is a first-order iterative
Iteration is the repetition of a process in order to generate a (possibly unbounded) sequence of outcomes. Each repetition of the process is a single iteration, ...
. Differentiable programming has found use in a wide variety of areas, particularly
Computational science, also known as scientific computing or scientific computation (SC), is a field that uses advanced computing
Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes ...
Artificial intelligence (AI) is intelligence
Intelligence has been defined in many ways: the capacity for abstraction
Abstraction in its main sense is a conceptual process where general rules and concept
Concepts are defined as abstra ...
Most differentiable programming frameworks work by constructing a graph containing the control flow and
In computer science
Computer science deals with the theoretical foundations of information, algorithms and the architectures of its computation as well as practical techniques for their application.
Computer science is the study of ...
in the program.
Earlier attempts generally fall into two groups:
In computing, a compiler is a computer program that Translator (computing), translates computer code written in one programming language (the ''source'' language) into another language (the ''target'' language). The name "compiler" is primarily ...
graph based approaches such as
TensorFlow is a Free and open-source software, free and open-source Library (computing), software library for machine learning and artificial intelligence. It can be used across a range of tasks but has a particular focus on Types of artificial ...
[TensorFlow 1 uses the static graph approach, whereas TensorFlow 2 uses the dynamic graph approach by default.] Theano
In Greek mythology, Theano (; Ancient Greek: Θεανώ) may refer to the following personages:
*Theano, wife of Metapontus, king of Icaria. Metapontus demanded that she bear him children, or leave the kingdom. She presented the children of Melani ...
, and MXNet
. They tend to allow for good
Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithmic processes and development of both computer hardware , hardware and softw ...
and easier scaling to large systems, but their static nature limits interactivity and the types of programs that can be created easily (e.g. those involving loops
Recursion (adjective: ''recursive'') occurs when a thing is defined in terms of itself or of its type. Recursion is used in a variety of disciplines ranging from linguistics
Linguistics is the scientific study of language, meaning tha ...
), as well as making it harder for users to reason effectively about their programs.
In computer programming
Computer programming is the process of designing and building an executable computer program to accomplish a specific computing result or to perform a particular task. Programming involves tasks such as analysis, gen ...
, dynamic graph based approaches such as PyTorch
. Their dynamic and interactive nature lets most programs be written and reasoned about more easily. However, they lead to interpreter
overhead (particularly when composing many small operations), poorer scalability, and struggle to gain benefit from compiler optimization.
Both of these early approaches are only able to differentiate code written in a suitable manner for the framework, limiting their interoperability with other programs.
A more recent package for the
Julia is usually a feminine given name. It is a Latinate feminine form of the name Julio (given name), Julio and Julius. (For further details on etymology, see wikt:Iulius#Latin, Wiktionary entry “Julius”.) The given name ''Julia'' had been ...
resolves the issues that earlier attempts faced by treating the language's syntax as the graph. The
An intermediate representation (IR) is the data structure or code used internally by a compiler or virtual machine to represent source code. An IR is designed to be conducive for further processing, such as Compiler optimization, optimization a ...
of arbitrary code can then be differentiated directly, optimized
, and compiled.
A programming language "currently under development and is not yet ready for use" calleMyia
ref name="myia1" /> allows defining a model using a subset of
Python may refer to:
The Pythonidae, commonly known as pythons, are a family of nonvenomous snakes found in Africa, Asia, and Australia. Among its members are some of the largest snakes in the world. Ten genera and 42 species
, which is compiled to Myia.
Differentiable programming has been applied in areas such as combining
deep learning #REDIRECT Deep learning#REDIRECT Deep learning
Deep learning (also known as deep structured learning) is part of a broader family of machine learning methods based on artificial neural networks with representation learning. Learning can be ...
with physics engines
Robotics is an interdisciplinary
Interdisciplinarity or interdisciplinary studies involves the combination of two or more academic disciplines into one activity (e.g., a research project). It draws knowledge from several other fields like ...
, solving electronic structure problems with differentiable
density functional theory
Density-functional theory (DFT) is a computational quantum mechanical modelling method used in physics
Physics (from grc, φυσική (ἐπιστήμη), physikḗ (epistḗmē), knowledge of nature, from ''phýsis'' 'nature'), , is ...
, differentiable ray tracing
Digital image processing is the use of a digital computer
A computer is a machine that can be programmed to Execution (computing), carry out sequences of arithmetic or logical operations automatically. Modern computers can perform generic se ...
Probabilistic programming (PP) is a programming paradigm in which probabilistic models are specified and inference for these models is performed automatically.
It represents an attempt to unify probabilistic modeling and traditional general purp ...
Mathematics (from Greek: ) includes the study of such topics as numbers (arithmetic and number theory), formulas and related structures (algebra), shapes and spaces in which they are contained (geometry), and quantities and t ...
Machine learning (ML) is the study of computer algorithms that can improve automatically through experience and by the use of data. It is seen as a part of artificial intelligence. Machine learning algorithms build a model based on sample data ...