Mixed-precision arithmetic is a form of
floating-point arithmetic
In computing, floating-point arithmetic (FP) is arithmetic that represents real numbers approximately, using an integer with a fixed precision, called the significand, scaled by an integer exponent of a fixed base. For example, 12.345 can be ...
that uses numbers with varying widths in a single operation.
Arithmetic
A common usage of mixed-precision arithmetic is for operating on inaccurate numbers with a small width and expanding them to a larger, more accurate representation. For example, two
half-precision
In computing, half precision (sometimes called FP16) is a binary floating-point computer number format that occupies 16 bits (two bytes in modern computers) in computer memory. It is intended for storage of floating-point values in applications wh ...
or
bfloat16
The bfloat16 (Brain Floating Point) floating-point format is a computer number format occupying 16 bits in computer memory; it represents a wide dynamic range of numeric values by using a floating radix point. This format is a truncated (16-b ...
(16-bit) floating-point numbers may be multiplied together to result in a more accurate
single-precision (32-bit) float.
In this way, mixed-precision arithmetic approximates
arbitrary-precision arithmetic, albeit with a low number of possible precisions.
Mixed-precision arithmetic is touted in the field of
machine learning, since
gradient descent algorithms can use coarse and efficient half-precision floats for certain tasks, but can be more accurate if they use more precise but slower single- or
double-precision
Double-precision floating-point format (sometimes called FP64 or float64) is a floating-point number format, usually occupying 64 bits in computer memory; it represents a wide dynamic range of numeric values by using a floating radix point.
Flo ...
floats. Some platforms, including
Nvidia and
AMD GPUs, provide mixed-precision arithmetic for this purpose, using coarse floats when possible, but expanding them to higher precision when necessary.
(like
gradient descent) are good candidates for mixed-precision arithmetic. In an iterative algorithm like
square root, a coarse integral guess can be made and refined over many iterations until the error in precision makes it such that the smallest addition or subtraction to the guess is still too coarse to be an acceptable answer. When this happens, the precision can be increased to something more precise, which allows for smaller increments to be used for the approximation.
Supercomputer
A supercomputer is a computer with a high level of performance as compared to a general-purpose computer. The performance of a supercomputer is commonly measured in floating-point operations per second ( FLOPS) instead of million instructions ...
s such as
Summit
A summit is a point on a surface that is higher in elevation than all points immediately adjacent to it. The topography, topographic terms acme, apex, peak (mountain peak), and zenith are synonymous.
The term (mountain top) is generally used ...
utilize mixed-precision arithmetic to be more efficient with regards to memory and processing time, as well as power consumption.
References
{{Reflist
Floating point
Computer arithmetic