Audio/Video

Everything you need to know about audio/video inside Chromium and Chromium OS!

Who To Contact

It's best to have discussion on chromium-dev@chromium.org, but failing that feel free to email me (scherkus@chromium.org).

We are Cr-Internals-Media on the Chromium bug tracker.

Documentation

Here's our (somewhat outdated) design doc for HTML5 audio/video.

Codec Support

As of February 2014, Chromium supports the following:
Vorbis audio codec
Theora video codec
Opus audio codec
VP8 video codec
VP9 video codec
PCM 8-bit unsigned integer
PCM 16-bit signed integer little endian
PCM 32-bit float little endian
Ogg container format
WebM container format
WAV container format

When building Google Chrome, the following codecs/containers are also included:
MP3 audio codec
AAC audio codec (Main only, not AAC-LC, AAC-SSR, HE-AAC)
H.264 video codec
MP4 container format

In terms of file extensions, Chromium recognizes the following (defined in src/net/base/mime_util.cc):
mp4, m4v, m4a, mp3, ogv, ogm, ogg, oga, webm, wav

When build Google Chrome OS, the following codecs/containers are also included:
FLAC audio codec
MPEG-4 video codec
AMR-NB audio codec
AMR-WB audio codec
PCM μ-law

In terms of file extensions, Google Chrome OS recognizes the following (defined in src/net/base/mime_util.cc):
amr, avi, flac, 3gp    

Code Location

Chromium
src/media/ - Home to all things multimedia!
src/media/audio - OS audio input/output abstractions
src/media/video/capture - OS camera input abstraction
src/media/video - software/hardware video decoder interfaces + implementations
deps/third_party/ffmpeg - Chromium's copy of FFmpeg
deps/third_party/libvpx - Chromium's copy of libvpx

WebKit
Source/WebCore/html/HTMLMediaElement.{cpp,h,idl} - media element base class
Source/WebCore/html/HTMLAudioElement.{cpp,h,idl} - audio element implementation
Source/WebCore/html/HTMLVideoElement.{cpp,h,idl} - video element implementation
Source/WebCore/platform/graphics/MediaPlayer.{cpp,h} - encapsulates media playback inside WebKit

Particularly Interesting Bits
src/net/base/mime_util.cpp - defines canPlayType() behaviour and file extension mapping
src/webkit/media/buffered_data_source.{cc,h} - Chromium's main implementation of DataSource for the media pipeline
src/webkit/media/buffered_resource_loader.{cc,h} - Implements the sliding window buffering strategy (see below)
src/webkit/media/webmediaplayer_impl.{cc,h} - Chromium's implementation of WebMediaPlayer
Source/WebKit/chromium/public/WebMediaPlayer.h - Chromium's interface for providing HTML5 audio/video functionality

How does everything get instantiated?

WebViewClient::createMediaPlayer(WebFrame*, WebMediaPlayerClient*) is the WebKit embedder API for creating a WebMediaPlayer and passing it back to WebKit. Every HTML5 audio/video element will ask to create a WebMediaPlayer object from the embedder.

For Chromium (the entire browser) this is handled in RenderView:
src/content/renderer/render_view.cc 

For Test Shell this is handled in TestWebViewDelegate:
src/webkit/tools/test_shell/test_webview_delegate.h

What's with all these abstraction layers!?

Necessary in order to properly isolate Chromium from WebKit. Although it looks confusing, really all you need to worry about is WebKit's MediaPlayer and Chromium's WebMediaPlayerImpl. The rest of it amounts to simple method forwarding.

GYP Flags

There are a few GYP flags which can alter the behaviour of Chromium's HTML5 audio/video implementation.

ffmpeg_branding
  Overrides which version of FFmpeg to use
  Default: $(branding)
  Values:
    Chrome - includes additional proprietary codecs (MP3, etc..) for use with Google Chrome
    Chromium - builds default set of codecs

proprietary_codecs
  Alters the list of codecs Chromium claims to support, which affects <source> and canPlayType() behaviour
  Default: 0
  Values:
    0 - <source> and canPlayType() assume the default set of codecs
    1 - <source> and canPlayType() assume they support additional proprietary codecs

use_system_ffmpeg
  Will build Chromium using the host system's version of FFmpeg
  Default: 0
  Values:
    0 - Build Chromium using the copy of FFmpeg included with Chromium
    1 - Build Chromium using the host system's version of FFmpeg

LayoutTests

We run the following layout tests:
Source/LayoutTests/media
Source/LayoutTests/http/tests/media

If these are failing, let us know!

How the %#$& does buffering work?

Chromium uses a combination of range requests and an in-memory sliding window to buffer media. We have a low and high watermark that is used to determine when to purposefully stall the HTTP request and when the resume the HTTP request.

It's complicated, so here's a picture:
Subpages (1): Media Internals
Comments