Rate-distortion optimization (RDO) is a method of improving

video quality Video quality is a characteristic of a video passed through a video transmission or processing system that describes perceived video degradation (typically, compared to the original video). Video processing systems may introduce some amount of dist ...

in video compression. The name refers to the optimization of the amount of ''distortion'' (loss of video quality) against the amount of data required to encode the video, the ''rate''. While it is primarily used by video encoders, rate-distortion optimization can be used to improve quality in any encoding situation (image, video, audio, or otherwise) where decisions have to be made that affect both file size and quality simultaneously.

Background

The classical method of making encoding decisions is for the video encoder to choose the result which yields the highest quality output image. However, this has the disadvantage that the choice it makes might require more bits while giving comparatively little quality benefit. One common example of this problem is in

motion estimation Motion estimation is the process of determining ''motion vectors'' that describe the transformation from one 2D image to another; usually from adjacent frames in a video sequence. It is an ill-posed problem as the motion is in three dimensions b ...

, and in particular regarding the use of quarter pixel-precision motion estimation. Adding the extra precision to the motion of a block during motion estimation might increase quality, but in some cases that extra quality isn't worth the extra bits necessary to encode the motion vector to a higher precision.

How it works

Rate-distortion optimization solves the aforementioned problem by acting as a video quality metric, measuring both the deviation from the source material and the bit cost for each possible decision outcome. The bits are mathematically measured by multiplying the bit cost by the

Lagrangian Lagrangian may refer to: Mathematics * Lagrangian function, used to solve constrained minimization problems in optimization theory; see Lagrange multiplier ** Lagrangian relaxation, the method of approximating a difficult constrained problem with ...

, a value representing the relationship between bit cost and quality for a particular quality level. The deviation from the source is usually measured as the

mean squared error In statistics, the mean squared error (MSE) or mean squared deviation (MSD) of an estimator (of a procedure for estimating an unobserved quantity) measures the average of the squares of the errors—that is, the average squared difference betwe ...

, in order to maximize the PSNR video quality metric. Calculating the bit cost is made more difficult by the entropy encoders in modern video codecs, requiring the rate-distortion optimization algorithm to pass each block of video to be tested to the entropy coder to measure its actual bit cost. In

MPEG The Moving Picture Experts Group (MPEG) is an alliance of working groups established jointly by ISO and IEC that sets standards for media coding, including compression coding of audio, video, graphics, and genomic data; and transmission and fi ...

codecs, the full process consists of a discrete cosine transform, followed by quantization and entropy encoding. Because of this, rate-distortion optimization is much slower than most other block-matching metrics, such as the simple sum of absolute differences (SAD) and

sum of absolute transformed differences The sum of absolute transformed differences (SATD) is a block matching criterion widely used in fractional motion estimation for video compression. It works by taking a frequency transform, usually a Hadamard transform, of the differences between ...

(SATD). As such it is usually used only for the final steps of the

process, such as deciding between different partition types in H.264/AVC.

List of encoders that support RDO

Ateme Ateme S.A. is a multinational company that specializes in video delivery, namely software for video compression based on H.265 / HEVC standards; MPEG4; MPEG2 encoding / decoding solutions for contribution links, multi-screen live streaming, OTT ...

H.264 encoder *

Grass Valley A grass valley (also vega and valle) is a meadow located within a forested and relatively small drainage basin such as a headwater. Grass valleys are common in North America, where they are created and maintained principally by the work of be ...

ViBE encoders (SD & HD MPEG-2/MPEG-4) * Harmonic Electra 8000 encoder (SD & HD MPEG-2/MPEG-4) *

libavcodec libavcodec is a free and open-source library of codecs for encoding and decoding video and audio data. libavcodec is an integral part of many open-source multimedia applications and frameworks. The popular MPV, xine and VLC media players u ...

* MainConcept H.264 encoder *

Microsoft Microsoft Corporation is an American multinational corporation, multinational technology company, technology corporation producing Software, computer software, consumer electronics, personal computers, and related services headquartered at th ...

VC-1 SMPTE 421, informally known as VC-1, is a video coding format. Most of it was initially developed as Microsoft's proprietary video format Windows Media Video 9 in 2003. With some enhancements including the development of a new Advanced Profile, ...

encoder * TANDBERG Television SD MPEG-2 EN8100 * TANDBERG Television HD MPEG-4 EN8190 * TANDBERG Television SD & HD MPEG-4 iPlex *

Theora Theora is a free lossy video compression format. It is developed by the Xiph.Org Foundation and distributed without licensing fees alongside their other free and open media projects, including the Vorbis audio format and the Ogg container ...

1.1-alpha1 and later (the "Thusnelda" branch) * x264 H.264 encoder * x265 H.265 encoder *

Xvid Xvid (formerly "XviD") is a video codec library following the MPEG-4 video coding standard, specifically MPEG-4 Part 2 Advanced Simple Profile (ASP). It uses ASP features such as b-frames, global and quarter pixel motion compensation, lumi ma ...

MPEG-4 ASP encoder * H.264/AVC reference software JM (Joint Model) *

HEVC High Efficiency Video Coding (HEVC), also known as H.265 and MPEG-H Part 2, is a video compression standard designed as part of the MPEG-H project as a successor to the widely used Advanced Video Coding (AVC, H.264, or MPEG-4 Part 10). In compa ...

reference software HM (HEVC Test Model) * Kvazaar (partial)

References

{{DEFAULTSORT:Rate-Distortion Optimization Video compression