In
computer science
Computer science is the study of computation, information, and automation. Computer science spans Theoretical computer science, theoretical disciplines (such as algorithms, theory of computation, and information theory) to Applied science, ...
, the precision of a numerical quantity is a measure of the detail in which the quantity is expressed. This is usually measured in bits, but sometimes in decimal digits. It is related to
precision in mathematics, which describes the number of digits that are used to express a value.
Some of the standardized precision formats are:
*
Half-precision floating-point format
*
Single-precision floating-point format
Single-precision floating-point format (sometimes called FP32 or float32) is a computer number format, usually occupying 32 bits in computer memory; it represents a wide dynamic range of numeric values by using a floating radix point.
A floa ...
*
Double-precision floating-point format
Double-precision floating-point format (sometimes called FP64 or float64) is a floating-point number format, usually occupying 64 bits in computer memory; it represents a wide range of numeric values by using a floating radix point.
Double pre ...
*
Quadruple-precision floating-point format
In computing, quadruple precision (or quad precision) is a binary floating-point–based computer number format that occupies 16 bytes (128 bits) with precision at least twice the 53-bit double precision.
This 128-bit quadruple precision is des ...
*
Octuple-precision floating-point format
Of these, octuple-precision format is rarely used. The single- and double-precision formats are most widely used and supported on nearly all platforms. The use of half-precision format and
minifloat formats has been increasing especially in the field of
machine learning
Machine learning (ML) is a field of study in artificial intelligence concerned with the development and study of Computational statistics, statistical algorithms that can learn from data and generalise to unseen data, and thus perform Task ( ...
since many machine learning algorithms are inherently error-tolerant.
Rounding errors
Precision is often the source of
rounding errors in
computation
A computation is any type of arithmetic or non-arithmetic calculation that is well-defined. Common examples of computation are mathematical equation solving and the execution of computer algorithms.
Mechanical or electronic devices (or, hist ...
. The number of bits used to store a number will often cause some loss of accuracy. An example would be to store "sin(0.1)" in IEEE single-precision floating point standard. The error is then often magnified as subsequent computations are made using the data (although it can also be reduced).
See also
*
Approximate computing
*
Arbitrary-precision arithmetic
In computer science, arbitrary-precision arithmetic, also called bignum arithmetic, multiple-precision arithmetic, or sometimes infinite-precision arithmetic, indicates that calculations are performed on numbers whose digits of precision are po ...
*
Extended precision
*
Granularity
Granularity (also called graininess) is the degree to which a material or system is composed of distinguishable pieces, "granules" or "grains" (metaphorically).
It can either refer to the extent to which a larger entity is subdivided, or the ...
*
IEEE754 (IEEE floating point standard)
*
Integer (computer science)
In computer science, an integer is a datum of integral data type, a data type that represents some interval (mathematics), range of mathematical integers. Integral data types may be of different sizes and may or may not be allowed to contain negati ...
*
Minifloat (8-bit and less)
*
Significant figures
Significant figures, also referred to as significant digits, are specific digits within a number that is written in positional notation that carry both reliability and necessity in conveying a particular quantity. When presenting the outcom ...
*
Truncation
References
{{Reflist, 60em
Approximations
Computer data