Audio/Video

Everything you need to know about audio/video inside Chromium and Chromium OS!

Who To Contact

It's best to have discussions on chromium-dev@chromium.org.

We are component Internals>Media on the Chromium bug tracker.

Documentation

Here's our outdated design doc for HTML5 audio/video.

Codec Support

Chromium supports the following:
Vorbis audio codec
Theora video codec
Opus audio codec
VP8 video codec
VP9 video codec
PCM 8-bit unsigned integer
PCM 16-bit signed integer little endian
PCM 32-bit float little endian
Ogg container format
WebM container format
WAV container format

When building Google Chrome, the following codecs/containers are also included:
MP3 audio codec
AAC audio codec (Main only, not AAC-LC, AAC-SSR, HE-AAC)
H.264 video codec
MP4 container format

In terms of file extensions, Chromium recognizes the following (defined in src/net/base/mime_util.cc):
mp4, m4v, m4a, mp3, ogv, ogm, ogg, oga, webm, wav

When build Google Chrome OS, the following codecs/containers are also included:
FLAC audio codec
MPEG-4 video codec
AMR-NB audio codec
AMR-WB audio codec
PCM μ-law

In terms of file extensions, Google Chrome OS recognizes the following (defined in src/net/base/mime_util.cc):
amr, avi, flac, 3gp    

Code Location

Chromium
media/ - Home to all things media!
media/audio - OS audio input/output abstractions
media/video/capture - OS camera input abstraction
media/video - software/hardware video decoder interfaces + implementations
third_party/ffmpeg - Chromium's copy of FFmpeg
third_party/libvpx - Chromium's copy of libvpx

Blink
Source/core/html/HTMLMediaElement.{cpp,h,idl} - media element base class
Source/core/html/HTMLAudioElement.{cpp,h,idl} - audio element implementation
Source/core/html/HTMLVideoElement.{cpp,h,idl} - video element implementation

Particularly Interesting Bits
net/base/mime_util.cc - defines canPlayType() behaviour and file extension mapping
media/blink/buffered_data_source.{cc,h} - Chromium's main implementation of DataSource for the media pipeline
media/blink/buffered_resource_loader.{cc,h} - Implements the sliding window buffering strategy (see below)
third_party/WebKit/public/platform/WebMediaPlayer.h - Blink's media player interface for providing HTML5 audio/video functionality
media/blink/webmediaplayer_impl.{cc,h} - Chromium's main implementation of WebMediaPlayer

How does everything get instantiated?

WebFrameClient::createMediaPlayer() is the Blink embedder API for creating a WebMediaPlayer and passing it back to Blink. Every HTML5 audio/video element will ask the embedder to create a WebMediaPlayer.

For Chromium this is handled in RenderFrameImpl.

GYP/GN Flags

There are a few GYP/GN flags which can alter the behaviour of Chromium's HTML5 audio/video implementation.

ffmpeg_branding
  Overrides which version of FFmpeg to use
  Default: $(branding)
  Values:
    Chrome - includes additional proprietary codecs (MP3, etc..) for use with Google Chrome
    Chromium - builds default set of codecs

proprietary_codecs
  Alters the list of codecs Chromium claims to support, which affects <source> and canPlayType() behaviour
  Default: 0(gyp)/false(gn)
  Values:
    0/false - <source> and canPlayType() assume the default set of codecs
    1/true - <source> and canPlayType() assume they support additional proprietary codecs

LayoutTests
We run the following layout tests:
LayoutTests/media
LayoutTests/http/tests/media

How the %#$& does buffering work?

Chromium uses a combination of range requests and an in-memory sliding window to buffer media. We have a low and high watermark that is used to determine when to purposely stall the HTTP request and when to resume the HTTP request.

It's complicated, so here's a picture:
Subpages (1): Media Internals
Comments