When an online video starts blurry and turns sharp after a few seconds, or loses quality for a moment while you are on a train, you are watching adaptive streaming at work. It is the technique used by almost every modern video platform, and it rests on a simple idea: instead of sending a single file, the server offers the same content in several qualities, cut into small pieces, and the player chooses, each time, the piece best suited to the available connection.
From progressive download to adaptive streaming
The simplest way to put a video online is to upload an MP4 file and let the browser download it while playing. This is called progressive download and it works well for short videos. It has two limits, though:
- there is only one quality: if the connection is slow, the video stalls;
- jumping to the middle requires the server to support byte-range requests, and the player cannot adapt.
Adaptive streaming solves both problems by working on top of ordinary HTTP. No special servers are needed: segments are static files that any web server or CDN can deliver.
The basic elements
- Renditions: the same video encoded in several versions, for example 1080p at 5 Mbps, 720p at 3 Mbps, 480p at 1.2 Mbps and 360p at 600 kbps.
- Segments: each version is split into pieces a few seconds long, typically between 2 and 6.
- Manifest: a text file listing the available versions and where the segments are.
- Player: the software that reads the manifest, measures bandwidth and decides which segment to download.
Since the player can only switch quality between one segment and the next, every segment must start with a keyframe and the cuts must line up across all versions.
HLS
HTTP Live Streaming was introduced by Apple and is today the most compatible format, supported natively by iOS, macOS and many TVs and, through JavaScript libraries, by every browser. It uses text playlists with the .m3u8 extension:
- a master playlist lists the variants with resolution, bitrate and codecs;
- a playlist for each variant lists its segments in order.
An excerpt of a master playlist looks like this:
#EXTM3U
#EXT-X-STREAM-INF:BANDWIDTH=5200000,RESOLUTION=1920x1080
1080p/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=3100000,RESOLUTION=1280x720
720p/index.m3u8
Originally segments were MPEG-TS; today HLS also supports fragmented MP4 segments.
MPEG-DASH
Dynamic Adaptive Streaming over HTTP is a vendor-neutral international standard. Its manifest, called the MPD, is an XML file describing periods, adaptation sets and representations. DASH is codec-agnostic and typically works with fragmented MP4.
It is not supported natively in browsers, but it works through Media Source Extensions with libraries such as dash.js or Shaka Player. Nor is it supported natively in iOS Safari, which explains why many services offer both formats.
CMAF: one set of segments
Encoding and storing the same content twice, once for HLS and once for DASH, doubles the storage needed. The CMAF format defines fragmented MP4 segments usable by both protocols: you produce two different manifests pointing to the same files. It is the most efficient solution for anyone who must serve every device.
How the player decides
The adaptation algorithm, called ABR (Adaptive Bitrate), relies on two main pieces of information:
- measured bandwidth: how quickly the last segments were downloaded;
- buffer level: how many seconds of video are already ready.
With a full buffer the player can afford to try a higher quality; with an almost empty buffer it drops quality to avoid stalling. A good algorithm also avoids constantly flipping between two levels, which viewers find annoying.
Typical problems and causes
| Symptom | Probable cause |
|---|---|
| visible jump when quality changes | keyframes not aligned across renditions |
| frequent buffering on good connections | segments too long or steps too large between renditions |
| slow start | heavy first segment or no light rendition |
| audio disappearing on quality switch | different audio tracks across variants |
| works on a computer but not on an iPhone | DASH manifest with no HLS alternative |
Creating an HLS stream with FFmpeg
For a single quality level the basic command is:
ffmpeg -i master.mp4 -c:v libx264 -crf 21 -g 50 -sc_threshold 0 -c:a aac -b:a 128k -hls_time 4 -hls_playlist_type vod -hls_segment_filename "seg_%03d.ts" index.m3u8
For several levels you use multiple mapping options and the -var_stream_map parameter, or dedicated packaging tools that start from MP4 files already encoded.
In short
- Adaptive streaming offers several qualities in segments and lets the player choose.
- HLS uses .m3u8 playlists and is the most compatible, especially on Apple devices.
- DASH uses an XML manifest and is an open, codec-agnostic standard.
- CMAF lets you serve HLS and DASH from the same segments.
- Aligned keyframes and a well-built quality ladder prevent jumps and buffering.
