HOME

TheInfoList




Differentiable programming is a
programming paradigm Programming paradigms are a way to classify programming languages based on their features. Languages can be classified into multiple paradigms. Some paradigms are concerned mainly with implications for the execution model of the language, suc ...
in which a numeric computer program can be differentiated throughout via
automatic differentiationIn mathematics Mathematics (from Ancient Greek, Greek: ) includes the study of such topics as quantity (number theory), mathematical structure, structure (algebra), space (geometry), and calculus, change (mathematical analysis, analysis). It ha ...
. This allows for
gradient based optimization
gradient based optimization
of parameters in the program, often via
gradient descent Gradient descent (also often called steepest descent) is a first-order iterative Iteration is the repetition of a process in order to generate a (possibly unbounded) sequence of outcomes. Each repetition of the process is a single iteration, ...

gradient descent
. Differentiable programming has found use in a wide variety of areas, particularly
scientific computing Computational science, also known as scientific computing or scientific computation (SC), is a field that uses advanced computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes ...
and
artificial intelligence Artificial intelligence (AI) is intelligence Intelligence has been defined in many ways: the capacity for abstraction Abstraction in its main sense is a conceptual process where general rules and concept Concepts are defined as abstra ...

artificial intelligence
.


Approaches

Most differentiable programming frameworks work by constructing a graph containing the control flow and
data structures In computer science Computer science deals with the theoretical foundations of information, algorithms and the architectures of its computation as well as practical techniques for their application. Computer science is the study of ...
in the program. Earlier attempts generally fall into two groups: * Static,
compiled In computing, a compiler is a computer program that Translator (computing), translates computer code written in one programming language (the ''source'' language) into another language (the ''target'' language). The name "compiler" is primarily ...
graph based approaches such as
TensorFlow TensorFlow is a Free and open-source software, free and open-source Library (computing), software library for machine learning and artificial intelligence. It can be used across a range of tasks but has a particular focus on Types of artificial ...
,TensorFlow 1 uses the static graph approach, whereas TensorFlow 2 uses the dynamic graph approach by default.
Theano In Greek mythology, Theano (; Ancient Greek: Θεανώ) may refer to the following personages: *Theano, wife of Metapontus, king of Icaria. Metapontus demanded that she bear him children, or leave the kingdom. She presented the children of Melani ...
, and MXNet. They tend to allow for good
compiler optimization In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithmic processes and development of both computer hardware , hardware and softw ...
and easier scaling to large systems, but their static nature limits interactivity and the types of programs that can be created easily (e.g. those involving loops or
recursion Recursion (adjective: ''recursive'') occurs when a thing is defined in terms of itself or of its type. Recursion is used in a variety of disciplines ranging from linguistics Linguistics is the scientific study of language, meaning tha ...

recursion
), as well as making it harder for users to reason effectively about their programs. *
Operator overloading In computer programming Computer programming is the process of designing and building an executable computer program to accomplish a specific computing result or to perform a particular task. Programming involves tasks such as analysis, gen ...
, dynamic graph based approaches such as PyTorch and AutoGrad. Their dynamic and interactive nature lets most programs be written and reasoned about more easily. However, they lead to interpreter overhead (particularly when composing many small operations), poorer scalability, and struggle to gain benefit from compiler optimization. Both of these early approaches are only able to differentiate code written in a suitable manner for the framework, limiting their interoperability with other programs. A more recent package for the
Julia Julia is usually a feminine given name. It is a Latinate feminine form of the name Julio (given name), Julio and Julius. (For further details on etymology, see wikt:Iulius#Latin, Wiktionary entry “Julius”.) The given name ''Julia'' had been ...
programming languag
Zygote
resolves the issues that earlier attempts faced by treating the language's syntax as the graph. The
intermediate representation An intermediate representation (IR) is the data structure or code used internally by a compiler or virtual machine to represent source code. An IR is designed to be conducive for further processing, such as Compiler optimization, optimization a ...
of arbitrary code can then be differentiated directly, optimized, and compiled. A programming language "currently under development and is not yet ready for use" calle
Myia
ref name="myia1" /> allows defining a model using a subset of
Python Python may refer to: * Pythonidae The Pythonidae, commonly known as pythons, are a family of nonvenomous snakes found in Africa, Asia, and Australia. Among its members are some of the largest snakes in the world. Ten genera and 42 species ...
, which is compiled to Myia.


Applications

Differentiable programming has been applied in areas such as combining
deep learning #REDIRECT Deep learning#REDIRECT Deep learning Deep learning (also known as deep structured learning) is part of a broader family of machine learning methods based on artificial neural networks with representation learning. Learning can be ...

deep learning
with physics engines in
robotics Robotics is an interdisciplinary Interdisciplinarity or interdisciplinary studies involves the combination of two or more academic disciplines into one activity (e.g., a research project). It draws knowledge from several other fields like ...

robotics
, solving electronic structure problems with differentiable
density functional theory Density-functional theory (DFT) is a computational quantum mechanical modelling method used in physics Physics (from grc, φυσική (ἐπιστήμη), physikḗ (epistḗmē), knowledge of nature, from ''phýsis'' 'nature'), , is ...
, differentiable ray tracing,
image processing Digital image processing is the use of a digital computer A computer is a machine that can be programmed to Execution (computing), carry out sequences of arithmetic or logical operations automatically. Modern computers can perform generic se ...
, and
probabilistic programming Probabilistic programming (PP) is a programming paradigm in which probabilistic models are specified and inference for these models is performed automatically. It represents an attempt to unify probabilistic modeling and traditional general purp ...
.


See also

*
Differentiable function In mathematics Mathematics (from Greek: ) includes the study of such topics as numbers (arithmetic and number theory), formulas and related structures (algebra), shapes and spaces in which they are contained (geometry), and quantities and t ...

Differentiable function
*
Machine learning Machine learning (ML) is the study of computer algorithms that can improve automatically through experience and by the use of data. It is seen as a part of artificial intelligence. Machine learning algorithms build a model based on sample data ...

Machine learning


Notes


References

{{Differentiable computing Differential calculus Programming paradigms