When you compare two encodes of the same video, judging by eye is indispensable but has two problems: it is slow and varies from person to person, and even within the same person depending on the screen and on tiredness. Objective metrics let you put a number on quality and compare encoders, presets and bitrates in a repeatable way. Used with judgement, they are a valuable tool; used blindly, they can lead to the wrong conclusions.
Full-reference metrics
The most widely used metrics are full-reference: they compare the compressed video with the uncompressed original, frame by frame. They therefore only work when you have the source, as when preparing content for streaming or choosing an encoder's settings.
PSNR
Peak Signal-to-Noise Ratio measures the numeric difference between the pixels of the original and those of the compressed video, expressed in decibels. The higher the value, the more similar the two videos.
Strengths: - simple and very fast to calculate; - useful for comparing settings of the same encoder on the same content.
Limits: - it ignores how human vision works: a slight shift of the whole picture lowers PSNR sharply even if the eye sees no difference; - it penalises visible and invisible errors alike; - some encoders, when tuned for PSNR, produce softer pictures that the eye judges worse.
SSIM
The Structural Similarity Index compares brightness, contrast and structure in small areas of the picture. The result runs from 0 to 1, where 1 means identical. It is closer to human perception than PSNR, because it values the preservation of edges and textures over the exact difference of individual pixels.
There is also a multi-scale variant, MS-SSIM, which evaluates the picture at several resolutions and behaves better on content with details of different sizes.
VMAF
VMAF (Video Multimethod Assessment Fusion) was developed by Netflix and released as open-source software. Instead of a single formula, it combines several elementary metrics through a machine-learning model trained on ratings given by real viewers. The result is a score from 0 to 100.
Strengths: - good correlation with human judgement on many types of content; - it also takes temporal aspects into account; - there are different models for different viewing conditions, for example a living-room TV or a phone.
Limits: - it is slower to calculate; - it was trained on certain kinds of content and defects, and may be less reliable in other cases, such as very unusual animation or uncommon artefacts; - it can be «fooled» by sharpening filters that raise the score without truly improving the picture.
Calculating the metrics with FFmpeg
FFmpeg includes filters for all three metrics. The first input is the video being assessed, the second the reference:
ffmpeg -i encoded.mp4 -i original.mov -lavfi "[0:v][1:v]psnr" -f null -
ffmpeg -i encoded.mp4 -i original.mov -lavfi "[0:v][1:v]ssim" -f null -
ffmpeg -i encoded.mp4 -i original.mov -lavfi "[0:v][1:v]libvmaf" -f null -
The libvmaf filter is only available if FFmpeg was built with the corresponding library.
Conditions to respect
For the comparison to make sense:
- the two videos must have the same resolution: if the encode is 720p and the original 1080p, it must first be scaled back to the viewing resolution;
- they must be aligned in time, frame by frame; a single frame of offset skews everything;
- they must have the same frame rate and, ideally, the same pixel format.
Interpreting the results
- Look at the average, but also at the minimums. A high average score can hide a few badly degraded seconds, and those are exactly what the viewer remembers.
- Compare on the same content. A VMAF of 90 on a cartoon and one of 90 on a football match do not indicate the same experience.
- Treat small differences as insignificant. One or two VMAF points are rarely perceptible.
- Use metrics to narrow the options, your eyes to decide. Metrics help discard clearly worse settings; the final choice should be checked by watching the hardest passages.
A typical use
Suppose you want to choose between two presets of an encoder. You encode the same excerpt with both at several bitrates, calculate the VMAF of each version and plot the quality-bitrate curve. If one preset reaches the same score with 15% less bitrate, the choice is obvious; if the curves overlap, the faster preset wins.
In short
- Full-reference metrics compare the compressed video with the original.
- PSNR is simple but far from human perception.
- SSIM evaluates picture structure and is more realistic.
- VMAF combines several metrics with a model trained on real ratings.
- Align resolution and timing, look at the minimums and always confirm by eye.
