In a video file the audio is often treated as a detail, yet it is responsible for a surprising share of problems: tracks that cannot be heard on some devices, the wrong language selected by default, lips moving before the voice. This guide focuses on audio as a component of the video file: which codecs to choose, how to organise tracks and how to diagnose and fix loss of sync.
Audio codecs in video containers
| Codec | Where it is used | Notes |
|---|---|---|
| AAC | MP4, MOV, MKV, HLS | the most compatible with H.264 and HEVC video |
| Opus | WebM, MKV, recent MP4 | very efficient, standard of the modern web |
| AC-3 / E-AC-3 | broadcast, discs, streaming | multichannel audio, widely supported on TVs |
| PCM | MOV, MXF, MKV | uncompressed, for editing and masters |
| FLAC | MKV | lossless, suited to archiving |
| DTS | discs, MKV | multichannel, more limited support |
For general distribution, stereo AAC between 128 and 192 kbps is a solid choice. For the web with WebM or AV1, Opus achieves good results at lower bitrates. For editing, PCM at 48 kHz is preferable, as it avoids losses from one step to the next.
Sample rate
Video conventionally uses 48 kHz, while CD music uses 44.1 kHz. Mixing the two rates without proper conversion in an editing program can produce small offsets or artefacts. A good rule is to record and work at 48 kHz and convert any 44.1 kHz music once, at the start of the project.
Multiple tracks and languages
A container such as MKV or MP4 can hold several audio tracks. Each should have:
- a language code (for example
itaoreng), which lets players choose the right track; - a descriptive title, such as «Director's commentary» or «Audio description»;
- a default track flag, to decide which one starts playing.
With FFmpeg they are set like this:
ffmpeg -i video.mkv -map 0 -c copy -metadata:s:a:0 language=ita -metadata:s:a:1 language=eng -disposition:a:0 default -disposition:a:1 0 output.mkv
If languages are not declared, the player usually picks the first track, which is not always the one you want.
How sound and picture are synchronised
Every audio and video packet in the container has a presentation timestamp (PTS) indicating when it should be played. The player usually uses the audio as its reference clock and shows frames at the right moment relative to it. Sync is lost when these timestamps do not match what was actually recorded.
The two forms of desync
Constant offset
The audio is ahead or behind by the same amount from start to finish. Typical causes:
- different processing delays between the camera and an external microphone;
- capture cards that add a few frames of delay to the video;
- remuxes that lost an initial delay value.
You fix it by shifting the audio. With FFmpeg, to delay the audio by 200 milliseconds without re-encoding:
ffmpeg -i video.mp4 -itsoffset 0.2 -i video.mp4 -map 0:v -map 1:a -c copy fixed.mp4
Progressive drift
Everything is fine at the start, but after ten minutes the voice arrives late, and the gap keeps growing. Typical causes:
- the variable frame rate of smartphones and screen recorders, misread by the editing program;
- external audio recorders whose internal clock differs slightly from the camera's;
- conversions between 23.976 and 24 fps, or between 44.1 and 48 kHz, carried out inconsistently.
Here shifting the audio is not enough: the track must be slightly stretched or shortened, or the video converted to a constant frame rate first. With FFmpeg the aresample=async=1 filter helps correct small irregularities, while frame rate conversion uses the fps filter.
Quick diagnosis
- Watch a scene with a clear, visible sound, such as a hand clap or a clapperboard.
- Check the offset at the start and towards the end of the file.
- If it is the same, it is a constant offset; if it grows, it is drift.
- Check with MediaInfo whether the video has a variable frame rate.
- Try the file in another player: some problems depend only on the playback software.
Recording to avoid problems
- always record the camera's own audio too, as a reference for aligning the external one;
- use a clapperboard or a hand clap at the start of every take;
- set camera and audio recorder to the same sample rate;
- for long takes with a separate recorder, split into shorter clips to limit drift.
In short
- AAC is the most compatible audio codec, Opus the most efficient for the web, PCM the best for editing.
- Work at 48 kHz, the video standard.
- Declare language, title and default flag for every audio track.
- A constant offset is fixed by shifting the audio; drift requires rescaling the track or fixing the frame rate.
- A clap at the start of every take makes alignment easy.
