Image Rescaling
   HOME

TheInfoList



OR:

In
computer graphics Computer graphics deals with generating images and art with the aid of computers. Computer graphics is a core technology in digital photography, film, video games, digital art, cell phone and computer displays, and many specialized applications. ...
and
digital imaging Digital imaging or digital image acquisition is the creation of a digital representation of the visual characteristics of an object, such as a physical scene or the interior structure of an object. The term is often assumed to imply or include ...
, image scaling refers to the resizing of a digital image. In video technology, the magnification of digital material is known as upscaling or resolution enhancement. When scaling a
vector graphic Vector graphics are a form of computer graphics in which visual images are created directly from geometric shapes defined on a Cartesian plane, such as points, lines, curves and polygons. The associated mechanisms may include vector display ...
image, the graphic primitives that make up the image can be scaled using geometric transformations with no loss of
image quality Image quality can refer to the level of accuracy with which different imaging systems capture, process, store, compress, transmit and display the signals that form an image. Another definition refers to image quality as "the weighted combination of ...
. When scaling a
raster graphics upright=1, The Smiley, smiley face in the top left corner is a raster image. When enlarged, individual pixels appear as squares. Enlarging further, each pixel can be analyzed, with their colors constructed through combination of the values for ...
image, a new image with a higher or lower number of pixels must be generated. In the case of decreasing the pixel number (scaling down), this usually results in a visible quality loss. From the standpoint of
digital signal processing Digital signal processing (DSP) is the use of digital processing, such as by computers or more specialized digital signal processors, to perform a wide variety of signal processing operations. The digital signals processed in this manner are a ...
, the scaling of raster graphics is a two-dimensional example of
sample-rate conversion Sample-rate conversion, sampling-frequency conversion or resampling is the process of changing the sampling rate or sampling frequency of a discrete signal to obtain a new discrete representation of the underlying continuous signal. Application a ...
, the conversion of a
discrete signal In mathematical dynamics, discrete time and continuous time are two alternative frameworks within which variables that evolve over time are modeled. Discrete time Discrete time views values of variables as occurring at distinct, separate "poi ...
from a
sampling rate In signal processing, sampling is the reduction of a continuous-time signal to a discrete-time signal. A common example is the conversion of a sound wave to a sequence of "samples". A sample is a value of the signal at a point in time and/or s ...
(in this case, the local sampling rate) to another.


Mathematical

Image scaling can be interpreted as a form of image resampling or image reconstruction from the view of the Nyquist sampling theorem. According to the theorem, downsampling to a smaller image from a higher-resolution original can only be carried out after applying a suitable 2D
anti-aliasing filter An anti-aliasing filter (AAF) is a filter used before a signal sampler to restrict the bandwidth of a signal to satisfy the Nyquist–Shannon sampling theorem over the band of interest. Since the theorem states that unambiguous reconstructi ...
to prevent aliasing artifacts. The image is reduced to the information that can be carried by the smaller image. In the case of up sampling, a
reconstruction filter In a mixed-signal system ( analog and digital), a reconstruction filter, sometimes called an anti-imaging filter, is used to construct a smooth analog signal from a digital input, as in the case of a digital to analog converter ( DAC) or other sam ...
takes the place of the anti-aliasing filter. File:160 by 160 thumbnail of 'Green Sea Shell'.png , Original 160x160px image File:160 by 160 thumbnail of 'Green Sea Shell' - 0. in fourier domain.png , Original image in spatial-frequency domain File:160 by 160 thumbnail of 'Green Sea Shell' - 1. fourier filtered for downsampling to 40 x 40.png , 2D
low-pass filter A low-pass filter is a filter that passes signals with a frequency lower than a selected cutoff frequency and attenuates signals with frequencies higher than the cutoff frequency. The exact frequency response of the filter depends on the filt ...
ed, but still at 160x160px File:160 by 160 thumbnail of 'Green Sea Shell' - 1.1. fourier filtered image for downsampling to 40 x 40 in fourier domain.png , Filtered image in spatial-frequency domain
File:160 by 160 thumbnail of 'Green Sea Shell' - 2. downsampling to 40 x 40 (nearest neighour).png , low-pass filtered 160x160px image 4× downsampled to 40x40px, alt=low-pass filtered 160x160px image 4× downsampled to 40x40px File:160 by 160 thumbnail of 'Green Sea Shell' - 3. fourier reconstruction from 40 x 40.png , 4× Fourier upsampling of 40x40px downsampled image to 160x160px (correct reconstruction) File:160 by 160 thumbnail of 'Green Sea Shell' - 4. fourier reconstruction from 40 x 40 (aliasing ).png , 4× Fourier upsampling of 40x40px downsampled image to 160x160px (with aliasing) A more sophisticated approach to upscaling treats the problem as an
inverse problem An inverse problem in science is the process of calculating from a set of observations the causal factors that produced them: for example, calculating an image in X-ray computed tomography, sound source reconstruction, source reconstruction in ac ...
, solving the question of generating a plausible image that, when scaled down, would look like the input image. A variety of techniques have been applied for this, including optimization techniques with
regularization Regularization may refer to: * Regularization (linguistics) * Regularization (mathematics) * Regularization (physics) * Regularization (solid modeling) * Regularization Law, an Israeli law intended to retroactively legalize settlements See also ...
terms and the use of
machine learning Machine learning (ML) is a field of study in artificial intelligence concerned with the development and study of Computational statistics, statistical algorithms that can learn from data and generalise to unseen data, and thus perform Task ( ...
from examples.


Algorithms

An image size can be changed in several ways.


Nearest-neighbor interpolation

One of the simpler ways of increasing image size is
nearest-neighbor interpolation Nearest-neighbor interpolation (also known as proximal interpolation or, in some contexts, point sampling) is a simple method of multivariate interpolation in one or more dimensions. Interpolation is the problem of approximating the value of a ...
, replacing every pixel with the nearest pixel in the output; for upscaling, this means multiple pixels of the same color will be present. This can preserve sharp details but also introduce jaggedness in previously smooth images. 'Nearest' in nearest-neighbor does not have to be the mathematical nearest. One common implementation is to always round toward zero. Rounding this way produces fewer artifacts and is faster to calculate. This algorithm is often preferred for images which have little to no smooth edges. A common application of this can be found in pixel art.


Bilinear and bicubic algorithms

Bilinear interpolation In mathematics, bilinear interpolation is a method for interpolating functions of two variables (e.g., ''x'' and ''y'') using repeated linear interpolation. It is usually applied to functions sampled on a 2D rectilinear grid, though it can be ge ...
works by interpolating pixel color values, introducing a continuous transition into the output even where the original material has discrete transitions. Although this is desirable for continuous-tone images, this algorithm reduces contrast (sharp edges) in a way that may be undesirable for line art.
Bicubic interpolation In mathematics, bicubic interpolation is an extension of cubic spline interpolation (a method of applying cubic interpolation to a data set) for interpolating data points on a two-dimensional regular grid. The interpolated surface (meaning the k ...
yields substantially better results, with an increase in computational cost.


Sinc and Lanczos resampling

Sinc resampling In mathematics, physics and engineering, the sinc function ( ), denoted by , has two forms, normalized and unnormalized.. In mathematics, the historical unnormalized sinc function is defined for by \operatorname(x) = \frac. Alternatively, ...
, in theory, provides the best possible reconstruction for a perfectly bandlimited signal. In practice, the assumptions behind sinc resampling are not completely met by real-world digital images.
Lanczos resampling Lanczos filtering and Lanczos resampling are two applications of a certain mathematical formula. It can be used as a low-pass filter or used to smoothly interpolate the value of a digital signal between its samples. In the latter case, it maps ...
, an approximation to the sinc method, yields better results. Bicubic interpolation can be regarded as a computationally efficient approximation to Lanczos resampling.


Box sampling

One weakness of bilinear, bicubic, and related algorithms is that they sample a specific number of pixels. When downscaling below a certain threshold, such as more than twice for all bi-sampling algorithms, the algorithms will sample non-adjacent pixels, which results in both losing data and rough results. The trivial solution to this issue is box sampling, which is to consider the target pixel a box on the original image and sample all pixels inside the box. This ensures that all input pixels contribute to the output. The major weakness of this algorithm is that it is hard to optimize.


Mipmap

Another solution to the downscale problem of bi-sampling scaling is
mipmap In computer graphics, a mipmap (''mip'' being an acronym of the Latin phrase ''multum in parvo'', meaning "much in little") is a pre-calculated, optimized sequence of images, each of which has an image resolution which is a factor of two small ...
s. A mipmap is a prescaled set of downscaled copies. When downscaling, the nearest larger mipmap is used as the origin to ensure no scaling below the useful threshold of bilinear scaling. This algorithm is fast and easy to optimize. It is standard in many frameworks, such as
OpenGL OpenGL (Open Graphics Library) is a Language-independent specification, cross-language, cross-platform application programming interface (API) for rendering 2D computer graphics, 2D and 3D computer graphics, 3D vector graphics. The API is typic ...
. The cost is using more image memory, exactly one-third more in the standard implementation.


Fourier-transform methods

Simple interpolation based on the
Fourier transform In mathematics, the Fourier transform (FT) is an integral transform that takes a function as input then outputs another function that describes the extent to which various frequencies are present in the original function. The output of the tr ...
pads the
frequency domain In mathematics, physics, electronics, control systems engineering, and statistics, the frequency domain refers to the analysis of mathematical functions or signals with respect to frequency (and possibly phase), rather than time, as in time ser ...
with zero components (a smooth window-based approach would reduce the
ringing Ringing may mean: Vibrations * Ringing (signal), unwanted oscillation of a signal, leading to ringing artifacts * Vibration of a harmonic oscillator ** Bell ringing * Ringing (telephony), the sound of a telephone bell * Ringing (medicine), a ri ...
). Besides the good conservation (or recovery) of details, notable are the ringing and the circular bleeding of content from the left border to the right border (and the other way around).


Edge-directed interpolation

Edge-directed interpolation algorithms aim to preserve edges in the image after scaling, unlike other algorithms, which can introduce staircase artifacts. Examples of algorithms for this task include New Edge-Directed Interpolation (NEDI), Edge-Guided Image Interpolation (EGGI), Iterative Curvature-Based Interpolation (ICBI), and Directional Cubic Convolution Interpolation (DCCI). A 2013 analysis found that DCCI had the best scores in
peak signal-to-noise ratio Peak signal-to-noise ratio (PSNR) is an engineering term for the ratio between the maximum possible power of a signal and the power of corrupting noise that affects the fidelity of its representation. Because many signals have a very wide dynamic ...
and
structural similarity The structural similarity index measure (SSIM) is a method for predicting the perceived quality of digital television and cinematic pictures, as well as other kinds of digital images and videos. It is also used for measuring the similarity betwe ...
on a series of test images.


hqx

For magnifying computer graphics with low resolution and/or few colors (usually from 2 to 256 colors), better results can be achieved by hqx or other
pixel-art scaling algorithms Pixel art scaling algorithms are graphical filters that attempt to enhance the appearance of hand-drawn 2D pixel art graphics. These algorithms are a form of automatic image enhancement. Pixel art scaling algorithms employ methods significantly ...
. These produce sharp edges and maintain a high level of detail.


Vectorization

Vector extraction, or
vectorization Vectorization may refer to: Computing * Array programming, a style of computer programming where operations are applied to whole arrays instead of individual elements * Automatic vectorization, a compiler optimization that transforms loops to vec ...
, offers another approach. Vectorization first creates a resolution-independent vector representation of the graphic to be scaled. Then the resolution-independent version is rendered as a raster image at the desired resolution. This technique is used by
Adobe Illustrator Adobe Illustrator is a vector graphics editor and Computer-aided design, design software developed and marketed by Adobe Inc., Adobe. Originally designed for the Apple Inc., Apple Mac (computer), Macintosh, development of Adobe Illustrator began ...
, Live Trace, and
Inkscape Inkscape is a vector graphics editor. It is used for both artistic and technical illustrations such as cartoons, clip art, logos, typography, diagrams, and flowcharts. It uses vector graphics to allow for sharp printouts and renderings at ...
.
Scalable Vector Graphics Scalable Vector Graphics (SVG) is an XML-based vector graphics format for defining two-dimensional graphics, having support for interactivity and animation. The SVG specification is an open standard developed by the World Wide Web Consortium sin ...
are well suited to simple geometric images, while photographs do not fare well with vectorization due to their complexity.


Deep convolutional neural networks

This method uses
machine learning Machine learning (ML) is a field of study in artificial intelligence concerned with the development and study of Computational statistics, statistical algorithms that can learn from data and generalise to unseen data, and thus perform Task ( ...
for more detailed images, such as photographs and complex artwork. Programs that use this method include waifu2x, Imglarger and Neural Enhance. Demonstration of conventional vs. waifu2x upscaling with noise reduction, using a detail of Phosphorus and Hesperus by Evelyn De Morgan. lick image for full size File:Evelyn de Morgan - Phosphorus and Hesperus, (1881) detail.png, Original image File:Evelyn de Morgan - Phosphorus and Hesperus, (1881) detail - upscaled 200% using Paint Shop Pro.png, Image upscaled 200% using
PaintShop Pro PaintShop Pro (PSP) is a raster and vector graphics editor for Microsoft Windows. It was originally published by Jasc Software. In October 2004, Corel purchased Jasc Software and the distribution rights to PaintShop Pro. PSP functionality can ...
File:Evelyn de Morgan - Phosphorus and Hesperus, (1881) detail - upscaled 200% using Waifu2x in Photo mode with Medium noise reduction.png, Image upscaled 200% using waifu2x in Photo mode with Medium noise reduction File:Evelyn de Morgan - Phosphorus and Hesperus, (1881) detail output.png, Image upscaled 400% using Topaz A.I. Gigapixel with Low noise reduction File:Evelyn de Morgan - Phosphorus and Hesperus, (1881) RealSR.png, Image upscaled 400% using RealSR DF2K-JPEG
AI-driven software such as the MyHeritage Photo Enhancer allows detail and sharpness to be added to historical photographs, where it is not present in the original. File:John Howard Lindauer portrait.jpg, Low-quality 270×368 pixel photograph of John Howard Lindauer, scanned from a State Legislature directory File:John Howard Lindauer 1983 (upscaled whitebackground enhanced).jpg, The same image enhanced and upscaled to 2160×2944 pixels by MyHeritage's Photo Enhancer


Applications


General

Image scaling is used in, among other applications,
web browser A web browser, often shortened to browser, is an application for accessing websites. When a user requests a web page from a particular website, the browser retrieves its files from a web server and then displays the page on the user's scr ...
s,
image editor Image editing encompasses the processes of altering images, whether they are Digital photography, digital photographs, traditional Photographic processing, photo-chemical photographs, or illustrations. Traditional analog image editing is known ...
s, image and file viewers, software magnifiers, digital zoom, the process of generating thumbnail images, and when outputting images through screens or printers.


Video

This application is the magnification of images for home theaters for HDTV-ready output devices from PAL-Resolution content, for example, from a DVD player. Upscaling is performed in real time, and the output signal is not saved.


Pixel-art scaling

As pixel-art graphics are usually low-resolution, they rely on careful placement of individual pixels, often with a limited palette of colors. This results in graphics that rely on stylized visual cues to define complex shapes with little resolution, down to individual pixels. This makes scaling pixel art a particularly difficult problem. Specialized algorithms were developed to handle pixel-art graphics, as the traditional scaling algorithms do not take perceptual cues into account. Since a typical application is to improve the appearance of fourth-generation and earlier
video game A video game or computer game is an electronic game that involves interaction with a user interface or input device (such as a joystick, game controller, controller, computer keyboard, keyboard, or motion sensing device) to generate visual fe ...
s on
arcade Arcade most often refers to: * Arcade game, a coin-operated video, pinball, electro-mechanical, redemption, etc., game ** Arcade video game, a coin-operated video game ** Arcade cabinet, housing which holds an arcade video game's hardware ** Arcad ...
and
console emulator A video game console emulator is a type of emulator that allows a computing device to emulate a video game console's hardware and play its games on the emulating platform. More often than not, emulators carry additional features that surpass ...
s, many are designed to run in real time for small input images at 60 frames per second. On fast hardware, these algorithms are suitable for gaming and other real-time image processing. These algorithms provide sharp, crisp graphics, while minimizing blur. Scaling art algorithms have been implemented in a wide range of emulators such as HqMAME and
DOSBox DOSBox is a free and open-source MS-DOS emulator. It supports running programs primarily video games that are otherwise inaccessible since hardware for running a compatible disk operating system (DOS) is obsolete and generally unavailab ...
, as well as 2D
game engine A game engine is a software framework primarily designed for the development of video games which generally includes relevant libraries and support programs such as a level editor. The "engine" terminology is akin to the term " software engine" u ...
s and
game engine recreation Game engine recreation is a type of video game engine remastering process whereby a new game engine is Code rewriting, rewritten from scratch as a Clone (computing), clone of the original with the ability to load the original game's data files suc ...
s such as
ScummVM Script Creation Utility for Maniac Mansion Virtual Machine (ScummVM) is a set of game engine recreations. Originally designed to play LucasArts adventure games that use the SCUMM system, it also supports a variety of non-SCUMM games by companies l ...
. They gained recognition with gamers, for whom these technologies encouraged a revival of 1980s and 1990s gaming experiences. Such filters are currently used in commercial emulators on
Xbox Live The Xbox network, formerly known and commonly referred to as Xbox Live, is an online multiplayer gaming and digital media delivery service created and operated by Microsoft Gaming for the Xbox brand. It was first made available to the origina ...
,
Virtual Console The Virtual Console was a line of downloadable retro video games for Nintendo's Wii and Wii U home video game consoles and the Nintendo 3DS family of handheld systems. The Virtual Console lineup consisted of titles originally released on pa ...
, and PSN to allow classic low-resolution games to be more visually appealing on modern HD displays. Recently released games that incorporate these filters include ''
Sonic's Ultimate Genesis Collection ''Sonic's Ultimate Genesis Collection'' (''Sega Mega Drive Ultimate Collection'' in PAL regions) is a Video game compilation, compilation of video games developed by Backbone Entertainment and published by Sega for PlayStation 3 and Xbox 360. The ...
'', '' Castlevania: The Dracula X Chronicles'', '' Castlevania: Symphony of the Night'', and '' Akumajō Dracula X Chi no Rondo''.


Real-time scaling

A number of companies have developed techniques to upscale video frames in
real-time Real-time, realtime, or real time may refer to: Computing * Real-time computing, hardware and software systems subject to a specified time constraint * Real-time clock, a computer clock that keeps track of the current time * Real-time Control Syst ...
, such as when they are drawn on screen in a video game.
Nvidia Nvidia Corporation ( ) is an American multinational corporation and technology company headquartered in Santa Clara, California, and incorporated in Delaware. Founded in 1993 by Jensen Huang (president and CEO), Chris Malachowsky, and Curti ...
's
deep learning super sampling Deep Learning Super Sampling (DLSS) is a suite of Real-time computing, real-time deep learning image enhancement and Image scaling, upscaling technologies developed by Nvidia that are available in a number of video games. The goal of these technol ...
(DLSS) uses
deep learning Deep learning is a subset of machine learning that focuses on utilizing multilayered neural networks to perform tasks such as classification, regression, and representation learning. The field takes inspiration from biological neuroscience a ...
to upsample lower- resolution images to a higher resolution for display on higher-resolution computer monitors.
AMD Advanced Micro Devices, Inc. (AMD) is an American multinational corporation and technology company headquartered in Santa Clara, California and maintains significant operations in Austin, Texas. AMD is a hardware and fabless company that de ...
's
FidelityFX Super Resolution GPUOpen is a middleware software suite originally developed by AMD's Radeon Technologies Group that offers advanced visual effects for computer games. It was released in 2016. GPUOpen serves as an alternative to, and a direct competitor of N ...
1.0 (FSR) does not employ machine learning, instead using traditional hand-written algorithms to achieve spatial upscaling on traditional shading units. FSR 2.0 utilises temporal upscaling, again with a hand-tuned algorithm. FSR standardized presets are not enforced, and some titles such as ''
Dota 2 ''Dota 2'' is a 2013 multiplayer online battle arena (MOBA) video game by Valve Corporation, Valve. The game is a sequel to ''Defense of the Ancients'' (''DotA''), a community-created Mod (video gaming), mod for Blizzard Entertainment's ''War ...
'' offer resolution sliders. Other technologies include
Intel Intel Corporation is an American multinational corporation and technology company headquartered in Santa Clara, California, and Delaware General Corporation Law, incorporated in Delaware. Intel designs, manufactures, and sells computer compo ...
XeSS and Nvidia Image Scaler (NIS).


See also

*
Bicubic interpolation In mathematics, bicubic interpolation is an extension of cubic spline interpolation (a method of applying cubic interpolation to a data set) for interpolating data points on a two-dimensional regular grid. The interpolated surface (meaning the k ...
*
Bilinear interpolation In mathematics, bilinear interpolation is a method for interpolating functions of two variables (e.g., ''x'' and ''y'') using repeated linear interpolation. It is usually applied to functions sampled on a 2D rectilinear grid, though it can be ge ...
*
Image reconstruction Iterative reconstruction refers to Iteration, iterative algorithms used to reconstruct 2D and 3D reconstruction, 3D images in certain Digital imaging, imaging techniques. For example, in computed tomography an image must be reconstructed from pro ...
*
Lanczos resampling Lanczos filtering and Lanczos resampling are two applications of a certain mathematical formula. It can be used as a low-pass filter or used to smoothly interpolate the value of a digital signal between its samples. In the latter case, it maps ...
*
Spline interpolation In the mathematical field of numerical analysis, spline interpolation is a form of interpolation where the interpolant is a special type of piecewise polynomial called a spline. That is, instead of fitting a single, high-degree polynomial to all ...
*
Seam carving Seam carving (or liquid rescaling) is an algorithm for content-aware image scaling, image resizing, developed by Shai Avidan, of Mitsubishi Electric Research Laboratories (MERL), and Ariel Shamir, of the Interdisciplinary Center and MERL. It fu ...
*
Video scaler A video scaler is a system that converts video signals from one display resolution to another; typically, scalers are used to convert a signal from a lower resolution (such as 480p standard definition) to a higher resolution (such as 1080i hig ...
* Scaling (geometry)


References

{{Video processing Image processing Articles containing video clips