
An image processor, also known as an image processing engine, image processing unit (IPU), or image signal processor (ISP), is a type of
media processor or specialized
digital signal processor
A digital signal processor (DSP) is a specialized microprocessor chip, with its architecture optimized for the operational needs of digital signal processing. DSPs are fabricated on metal–oxide–semiconductor (MOS) integrated circuit chips. ...
(DSP) used for
image processing
An image or picture is a visual representation. An image can be two-dimensional, such as a drawing, painting, or photograph, or three-dimensional, such as a carving or sculpture. Images may be displayed through other media, including a pr ...
, in
digital camera
A digital camera, also called a digicam, is a camera that captures photographs in Digital data storage, digital memory. Most cameras produced today are digital, largely replacing those that capture images on photographic film or film stock. Dig ...
s or other devices.
Image processors often employ
parallel computing
Parallel computing is a type of computing, computation in which many calculations or Process (computing), processes are carried out simultaneously. Large problems can often be divided into smaller ones, which can then be solved at the same time. ...
even with
SIMD
Single instruction, multiple data (SIMD) is a type of parallel computer, parallel processing in Flynn's taxonomy. SIMD describes computers with multiple processing elements that perform the same operation on multiple data points simultaneousl ...
or
MIMD technologies to increase speed and efficiency. The
digital image
A digital image is an image composed of picture elements, also known as pixels, each with '' finite'', '' discrete quantities'' of numeric representation for its intensity or gray level that is an output from its two-dimensional functions f ...
processing engine can perform a range of tasks.
To increase the system integration on
embedded devices, often it is a
system on a chip
A system on a chip (SoC) is an integrated circuit that combines most or all key components of a computer or Electronics, electronic system onto a single microchip. Typically, an SoC includes a central processing unit (CPU) with computer memory, ...
with
multi-core processor
A multi-core processor (MCP) is a microprocessor on a single integrated circuit (IC) with two or more separate central processing units (CPUs), called ''cores'' to emphasize their multiplicity (for example, ''dual-core'' or ''quad-core''). Ea ...
architecture.
Function
Bayer transformation
The
photodiode
A photodiode is a semiconductor diode sensitive to photon radiation, such as visible light, infrared or ultraviolet radiation, X-rays and gamma rays. It produces an electrical current when it absorbs photons. This can be used for detection and me ...
s employed in an
image sensor An image sensor or imager is a sensor that detects and conveys information used to form an image. It does so by converting the variable attenuation of light waves (as they refraction, pass through or reflection (physics), reflect off objects) into s ...
are color-blind by nature: they can only record
shades of grey. To get
color
Color (or colour in English in the Commonwealth of Nations, Commonwealth English; American and British English spelling differences#-our, -or, see spelling differences) is the visual perception based on the electromagnetic spectrum. Though co ...
into the picture, they are covered with different color filters:
red,
green
Green is the color between cyan and yellow on the visible spectrum. It is evoked by light which has a dominant wavelength of roughly 495570 nm. In subtractive color systems, used in painting and color printing, it is created by a com ...
and
blue
Blue is one of the three primary colours in the RYB color model, RYB colour model (traditional colour theory), as well as in the RGB color model, RGB (additive) colour model. It lies between Violet (color), violet and cyan on the optical spe ...
(
RGB) according to the pattern designated by the
Bayer filter
A Bayer filter mosaic is a color filter array (CFA) for arranging RGB color model, RGB color filters on a square grid of photosensors. Its particular arrangement of color filters is used in most single-chip digital image sensors used in digit ...
. As each photodiode records the color information for exactly one
pixel
In digital imaging, a pixel (abbreviated px), pel, or picture element is the smallest addressable element in a Raster graphics, raster image, or the smallest addressable element in a dot matrix display device. In most digital display devices, p ...
of the image, without an image processor there would be a green pixel next to each red and blue pixel.
This process, however, is quite complex, and involves a number of different operations. Its quality depends largely on the effectiveness of the
algorithms
In mathematics and computer science, an algorithm () is a finite sequence of mathematically rigorous instructions, typically used to solve a class of specific problems or to perform a computation. Algorithms are used as specifications for per ...
applied to the raw data coming from the sensor. The mathematically manipulated data becomes the recorded photo file.
Demosaicing
As stated above, the image processor evaluates the color and
brightness
Brightness is an attribute of visual perception in which a source appears to be radiating/reflecting light. In other words, brightness is the perception dictated by the luminance of a visual target. The perception is not linear to luminance, and ...
data of a given pixel, compares them with the data from neighboring pixels, and then uses a
demosaicing
Demosaicing (or de-mosaicing, demosaicking), also known as color reconstruction, is a digital image processing algorithm used to reconstruct a full color image from the incomplete color samples output from an image sensor overlaid with a color fil ...
algorithm to produce an appropriate color and brightness value for the pixel. The image processor also assesses the whole picture to guess at the correct distribution of
contrast. By adjusting the
gamma
Gamma (; uppercase , lowercase ; ) is the third letter of the Greek alphabet. In the system of Greek numerals it has a value of 3. In Ancient Greek, the letter gamma represented a voiced velar stop . In Modern Greek, this letter normally repr ...
value (heightening or lowering the contrast range of an image's mid-tones), subtle tonal gradations, such as in
human skin
The human skin is the outer covering of the body and is the largest organ of the integumentary system. The skin has up to seven layers of ectodermal tissue (biology), tissue guarding Skeletal muscle, muscles, bones, ligaments and organ (anato ...
or the blue of the
sky
The sky is an unobstructed view upward from the planetary surface, surface of the Earth. It includes the atmosphere of Earth, atmosphere and outer space. It may also be considered a place between the ground and outer space, thus distinct from ...
, become much more realistic.
Noise reduction
Noise
Noise is sound, chiefly unwanted, unintentional, or harmful sound considered unpleasant, loud, or disruptive to mental or hearing faculties. From a physics standpoint, there is no distinction between noise and desired sound, as both are vibrat ...
is a phenomenon found in any
electronic circuitry. In
digital photography
Digital photography uses cameras containing arrays of electronic photodetectors interfaced to an analog-to-digital converter (ADC) to produce images focused by a lens, as opposed to an exposure on photographic film. The digitized image is ...
its effect is often visible as random spots of obviously wrong color in an otherwise smoothly-colored area. Noise increases with temperature and
exposure times. When higher
ISO
The International Organization for Standardization (ISO ; ; ) is an independent, non-governmental, international standard development organization composed of representatives from the national standards organizations of member countries.
Me ...
settings are chosen the electronic signal in the image sensor is amplified, which at the same time increases the noise level, leading to a lower
signal-to-noise ratio
Signal-to-noise ratio (SNR or S/N) is a measure used in science and engineering that compares the level of a desired signal to the level of background noise. SNR is defined as the ratio of signal power to noise power, often expressed in deci ...
. The image processor attempts to separate the noise from the image information and to remove it. This can be quite a challenge, as the image may contain areas with fine textures which, if treated as noise, may lose some of their definition.
Image sharpening
As the color and brightness values for each pixel are
interpolated some
image sharpening is applied to even out any fuzziness that has occurred. To preserve the impression of
depth, clarity and fine details, the image processor must sharpen edges and contours. It therefore must
detect edges correctly and reproduce them smoothly and without over-sharpening.
Models
Image processor users are using industry standard products, application-specific standard products (ASSP) or even
application-specific integrated circuit
An application-specific integrated circuit (ASIC ) is an integrated circuit (IC) chip customized for a particular use, rather than intended for general-purpose use, such as a chip designed to run in a digital voice recorder or a high-efficienc ...
s (ASIC) with trade names: Canon's is called
DIGIC, Nikon's
Expeed, Olympus' TruePic, Panasonic's
Venus Engine and Sony's
Bionz. Some are known to be based on the
Fujitsu Milbeaut, the
Texas Instruments
Texas Instruments Incorporated (TI) is an American multinational semiconductor company headquartered in Dallas, Texas. It is one of the top 10 semiconductor companies worldwide based on sales volume. The company's focus is on developing analog ...
OMAP,
Panasonic
is a Japanese multinational electronics manufacturer, headquartered in Kadoma, Osaka, Kadoma, Japan. It was founded in 1918 as in Fukushima-ku, Osaka, Fukushima by Kōnosuke Matsushita. The company was incorporated in 1935 and renamed and c ...
MN103,
Zoran Coach, Altek Sunny or
Sanyo
is a former Japanese electronics manufacturer founded in 1947 by Toshio Iue, the brother-in-law of Kōnosuke Matsushita, the founder of Matsushita Electric Industrial, now known as Panasonic. Iue left Matsushita Electric to start his own bu ...
image/video processors.
ARM architecture
ARM (stylised in lowercase as arm, formerly an acronym for Advanced RISC Machines and originally Acorn RISC Machine) is a family of reduced instruction set computer, RISC instruction set architectures (ISAs) for central processing unit, com ...
processors with its
NEON SIMD Media Processing Engines (MPE) are often used in
mobile phone
A mobile phone or cell phone is a portable telephone that allows users to make and receive calls over a radio frequency link while moving within a designated telephone service area, unlike fixed-location phones ( landline phones). This rad ...
s.
Processor brand names
*ATI -
Imageon (graphics co-processor used in many early mobile photos to offer camera image signal processing)
*Canon -
DIGIC (based on Texas Instruments
OMAP)
*Casio - EXILIM engine
*Epson - EDiART
*Fujifilm - EXR III or X Processor Pro
*Google -
Pixel Visual Core
*HTC - ImageSense
*Intel - IPU
*MediaTek - Imagiq
*Minolta / Konica Minolta -
SUPHEED with CxProcess
*Leica - MAESTRO (based on Fujitsu
Milbeaut)
*Nikon -
Expeed (based on Fujitsu
Milbeaut)
Milbeaut and EXPEED
byThom
*Olympus - TruePic (based on Panasonic MN103/MN103S)
*OPPO - MariSilicon X
*Panasonic - Venus Engine (based on Panasonic MN103/MN103S)
*Pentax - PRIME (Pentax Real IMage Engine) (newer variants based on Fujitsu Milbeaut)
*Qualcomm - Qualcomm Spectra (based on Qualcomm Snapdragon)
*Ricoh - GR engine (GR digital), Smooth Imaging Engine
*Samsung - DRIMe (based on Samsung
Samsung Group (; stylised as SΛMSUNG) is a South Korean Multinational corporation, multinational manufacturing Conglomerate (company), conglomerate headquartered in the Samsung Town office complex in Seoul. The group consists of numerous a ...
Exynos
The Samsung Exynos (stylized as SΛMSUNG Exynos), formerly Hummingbird (), is a series of ARM architecture, Arm-based System on a chip, system-on-chips developed by Samsung Electronics' System LSI division and manufactured by Samsung Foundry. I ...
)
*Sanyo - Platinum engine
*Sigma - True
*Sharp - ProPix
*Socionext
Milbeaut
Family of ISPs - SC2000 (M-10V), SC2002 (M-11S)
*Sony - Bionz
*THine - THP serie
with compatible SDK Kit for developing firmwar
*UNISOC - Vivimagic
Speed
With the ever-higher pixel count in image sensors, the image processor's speed becomes more critical: Photographer, photographers don't want to wait for the camera's image processor to complete its job before they can carry on shooting - they don't even want to notice some processing is going on inside the camera. Therefore, image processors must be optimised to cope with more data in the same or even a shorter period of time.
Software
libcamera is a software library that supports using image signal processors for the capture of pictures.
See also
* Color image pipeline
*Image processing
An image or picture is a visual representation. An image can be two-dimensional, such as a drawing, painting, or photograph, or three-dimensional, such as a carving or sculpture. Images may be displayed through other media, including a pr ...
*Digital image processing
Digital image processing is the use of a digital computer to process digital images through an algorithm. As a subcategory or field of digital signal processing, digital image processing has many advantages over analog image processing. It allo ...
* Digital image editing
*Demosaicing
Demosaicing (or de-mosaicing, demosaicking), also known as color reconstruction, is a digital image processing algorithm used to reconstruct a full color image from the incomplete color samples output from an image sensor overlaid with a color fil ...
References
{{System on a chip
Digital signal processors
Photography equipment