HOME

TheInfoList



OR:

PyTorch is a
machine learning Machine learning (ML) is a field of inquiry devoted to understanding and building methods that 'learn', that is, methods that leverage data to improve performance on some set of tasks. It is seen as a part of artificial intelligence. Machin ...
framework based on the
Torch A torch is a stick with combustible material at one end, which is ignited and used as a light source. Torches have been used throughout history, and are still used in processions, symbolic and religious events, and in juggling entertainment. In ...
library, used for applications such as
computer vision Computer vision is an interdisciplinary scientific field that deals with how computers can gain high-level understanding from digital images or videos. From the perspective of engineering, it seeks to understand and automate tasks that the human ...
and
natural language processing Natural language processing (NLP) is an interdisciplinary subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to proc ...
, originally developed by
Meta AI Meta AI is an artificial intelligence laboratory that belongs to Meta Platforms Inc. (formerly known as Facebook, Inc.) Meta AI intends to develop various forms of artificial intelligence, improving augmented and artificial reality technologie ...
and now part of the Linux Foundation umbrella. It is
free and open-source software Free and open-source software (FOSS) is a term used to refer to groups of software consisting of both free software and open-source software where anyone is freely licensed to use, copy, study, and change the software in any way, and the sour ...
released under the
modified BSD license BSD licenses are a family of permissive free software licenses, imposing minimal restrictions on the use and distribution of covered software. This is in contrast to copyleft licenses, which have share-alike requirements. The original BSD lice ...
. Although the
Python Python may refer to: Snakes * Pythonidae, a family of nonvenomous snakes found in Africa, Asia, and Australia ** ''Python'' (genus), a genus of Pythonidae found in Africa and Asia * Python (mythology), a mythical serpent Computing * Python (pr ...
interface is more polished and the primary focus of development, PyTorch also has a C++ interface. A number of pieces of
deep learning Deep learning (also known as deep structured learning) is part of a broader family of machine learning methods based on artificial neural networks with representation learning. Learning can be supervised, semi-supervised or unsupervised. D ...
software are built on top of PyTorch, including
Tesla Autopilot Tesla Autopilot is a suite of advanced driver-assistance system (ADAS) features offered by Tesla that amounts to SAE International Level 2 vehicle automation. Its features are lane centering, traffic-aware cruise control, automatic lane ch ...
,
Uber Uber Technologies, Inc. (Uber), based in San Francisco, provides mobility as a service, ride-hailing (allowing users to book a car and driver to transport them in a way similar to a taxi), food delivery (Uber Eats and Postmates), pack ...
's Pyro, Hugging Face's Transformers,
PyTorch Lightning PyTorch Lightning is an open-source Python library that provides a high-level interface for PyTorch, a popular deep learning Deep learning (also known as deep structured learning) is part of a broader family of machine learning methods base ...
, and Catalyst. PyTorch provides two high-level features: * Tensor computing (like NumPy) with strong acceleration via
graphics processing unit A graphics processing unit (GPU) is a specialized electronic circuit designed to manipulate and alter memory to accelerate the creation of images in a frame buffer intended for output to a display device. GPUs are used in embedded systems, m ...
s (GPU) * Deep neural networks built on a tape-based
automatic differentiation In mathematics and computer algebra, automatic differentiation (AD), also called algorithmic differentiation, computational differentiation, auto-differentiation, or simply autodiff, is a set of techniques to evaluate the derivative of a function ...
system


History

Meta (formerly known as Facebook) operates both ''PyTorch'' and ''Convolutional Architecture for Fast Feature Embedding'' (
Caffe2 Caffe (Convolutional Architecture for Fast Feature Embedding) is a deep learning framework, originally developed at University of California, Berkeley. It is open source, under a BSD license. It is written in C++, with a Python interface. ...
), but models defined by the two frameworks were mutually incompatible. The Open Neural Network Exchange (
ONNX The Open Neural Network Exchange (ONNX) [] is an Open-source software, open-source artificial intelligence ecosystem of technology companies and research organizations that establish open standards for representing machine learning algorithms and ...
) project was created by Meta and
Microsoft Microsoft Corporation is an American multinational technology corporation producing computer software, consumer electronics, personal computers, and related services headquartered at the Microsoft Redmond campus located in Redmond, Washi ...
in September 2017 for converting models between frameworks. Caffe2 was merged into PyTorch at the end of March 2018. In September 2022, Meta announced that ''PyTorch'' would be governed by PyTorch Foundation, a newly created independent organizationa subsidiary of Linux Foundation.


PyTorch tensors

PyTorch defines a class called Tensor (torch.Tensor) to store and operate on homogeneous multidimensional rectangular arrays of numbers. PyTorch Tensors are similar to NumPy Arrays, but can also be operated on a
CUDA CUDA (or Compute Unified Device Architecture) is a parallel computing platform and application programming interface (API) that allows software to use certain types of graphics processing units (GPUs) for general purpose processing, an approach c ...
-capable
NVIDIA Nvidia CorporationOfficially written as NVIDIA and stylized in its logo as VIDIA with the lowercase "n" the same height as the uppercase "VIDIA"; formerly stylized as VIDIA with a large italicized lowercase "n" on products from the mid 1990s to ...
GPU. PyTorch has also been developing support for other GPU platforms, for example, AMD's
ROCm ROCm is an Advanced Micro Devices (AMD) software stack for graphics processing unit (GPU) programming. ROCm spans several domains: general-purpose computing on graphics processing units (GPGPU), high performance computing (HPC), heterogeneous c ...
and Apple's Metal Framework. PyTorch supports various sub-types of Tensors. Note that the term "tensor" here does not carry the same meaning as
tensor In mathematics, a tensor is an algebraic object that describes a multilinear relationship between sets of algebraic objects related to a vector space. Tensors may map between different objects such as vectors, scalars, and even other tenso ...
in mathematics or physics. The meaning of the word in those areas, that is, a certain kind of object in
linear algebra Linear algebra is the branch of mathematics concerning linear equations such as: :a_1x_1+\cdots +a_nx_n=b, linear maps such as: :(x_1, \ldots, x_n) \mapsto a_1x_1+\cdots +a_nx_n, and their representations in vector spaces and through matrices ...
, is only tangentially related to the one in Machine Learning.


Modules


Autograd module

PyTorch uses a method called
automatic differentiation In mathematics and computer algebra, automatic differentiation (AD), also called algorithmic differentiation, computational differentiation, auto-differentiation, or simply autodiff, is a set of techniques to evaluate the derivative of a function ...
. A recorder records what operations have performed, and then it replays it backward to compute the gradients. This method is especially powerful when building neural networks to save time on one epoch by calculating differentiation of the parameters at the forward pass.


Optim module

torch.optim is a module that implements various optimization algorithms used for building neural networks. Most of the commonly used methods are already supported, so there is no need to build them from scratch.


nn module

PyTorch autograd makes it easy to define computational graphs and take gradients, but raw autograd can be a bit too low-level for defining complex neural networks. This is where the nn module can help. The nn module provides layers and tools to easily create a neural networks by just defining the layers of the network. PyTorch also contains many other useful submodules such as data loading utilities and distributed training functions.


Example

The following program shows the low-level functionality of the library with a simple example import torch dtype = torch.float device = torch.device("cpu") # This executes all calculations on the CPU # device = torch.device("cuda:0") # This executes all calculations on the GPU # Creation of a tensor and filling of a tensor with random numbers a = torch.randn(2, 3, device=device, dtype=dtype) print(a) # Output of tensor A # Output: tensor( -1.1884, 0.8498, -1.7129 # 0.8816, 0.1944, 0.5847) # Creation of a tensor and filling of a tensor with random numbers b = torch.randn(2, 3, device=device, dtype=dtype) print(b) # Output of tensor B # Output: tensor( 0.7178, -0.8453, -1.3403 # 1.3262, 1.1512, -1.7070) print(a*b) # Output of a multiplication of the two tensors # Output: tensor( -0.8530, -0.7183, 2.58 # 1.1692, 0.2238, -0.9981) print(a.sum()) # Output of the sum of all elements in tensor A # Output: tensor(-2.1540) print(a ,2 # Output of the element in the third column of the second row (zero based) # Output: tensor(0.5847) print(a.max()) # Output of the maximum value in tensor A # Output: tensor(-1.7129) The following code-block shows an example of the higher level functionality provided nn module. A neural network with linear layers is defined in the example. import torch from torch import nn # Import the nn sub-module from PyTorch class NeuralNetwork(nn.Module): # Neural networks are defined as classes def __init__(self): # Layers and variables are defined in the __init__ method super(NeuralNetwork, self).__init__() # Must be in every network. self.flatten = nn.Flatten() # Defining a flattening layer. self.linear_relu_stack = nn.Sequential( # Defining a stack of layers. nn.Linear(28*28, 512), # Linear Layers have an input and output shape nn.ReLU(), # ReLU is one of many activation functions provided by nn nn.Linear(512, 512), nn.ReLU(), nn.Linear(512, 10), ) def forward(self, x): # This function defines the forward pass. x = self.flatten(x) logits = self.linear_relu_stack(x) return logits


See also

*
Comparison of deep learning software The following table compares notable software frameworks, libraries and computer programs for deep learning. Deep-learning software by name Comparison of compatibility of machine learning models See also *Comparison of numerical-analy ...
*
Differentiable programming Differentiable programming is a programming paradigm in which a numeric computer program can be differentiated throughout via automatic differentiation. This allows for gradient-based optimization of parameters in the program, often via gradie ...
* DeepSpeed *
Torch (machine learning) Torch is an open-source machine learning library, a scientific computing framework, and a script language based on the Lua programming language. It provides a wide range of algorithms for deep learning, and uses the scripting language LuaJIT, ...


References


External links

* {{Differentiable computing Deep learning software Facebook software Free science software Free software programmed in C Free software programmed in Python Open-source artificial intelligence Python (programming language) scientific libraries Software using the BSD license