HTML5: Audio and Video Elements

Exclusive offer: get 50% off this eBook here
HTML5 Multimedia Development Cookbook

HTML5 Multimedia Development Cookbook — Save 50%

Recipes for practical, real-world HTML5 multimedia driven development.

$23.99    $12.00
by Dale Cruse | June 2011 | Open Source Web Development

In the open source HTML5 standard, the new audio and video elements are more mature and usable than ever before. That's a good thing, because users' expectations for multimedia are much higher than ever before.

In this article by Dale Cruse, author of HTML5 Multimedia Development Cookbook, we will cover:

  • Understanding audio and video file formats
  • Creating accessible audio and video
  • Crafting a slick audio player

 

HTML5 Multimedia Development Cookbook

HTML5 Multimedia Development Cookbook

Recipes for practical, real-world HTML5 multimedia driven development.

        Read more about this book      

(For more resources on Multimedia development, see here.)

Understanding audio and video file formats

There are plenty of different audio and video file formats. These files may include not just video but also audio and metadata—all in one file. These file types include:

  • .avi – A blast from the past, the Audio Video Interleave file format was invented by Microsoft. Does not support most modern audio and video codecs in use today.
  • .flv – Flash video. This used to be the only video file format Flash fully supported. Now it also includes support for .mp4.
  • .mp4 or .mpv – MPEG4 is based on Apple's QuickTime player and requires that software for playback.

How it works...

Each of the previously mentioned video file formats require a browser plugin or some sort of standalone software for playback. Next, we'll look at new open-source audio and video file formats that don't require plugins or special software and the browsers that support them.

  • H.264 has become of the most commonly used high definition video formats. Used on Blu-ray Discs as well as many Internet video streaming sites including Flash, iTunes Music Store, Silverlight, Vimeo, YouTube, cable television broadcasts, and real-time videoconferencing. In addition, there is a patent on H.264 is therefore, by definition, not open source. Browsers that support H.264 video file format include:

    HTML5 Multimedia Development Cookbook

    Google has now partially rejected the H.264 format and is leaning more toward its support of the new WebM video file format instead.

  • Ogg might be a funny sounding name, but its potential is very serious, I assure you. Ogg is really two things: Ogg Theora, which is a video file format; and Ogg Vorbis, which is an audio file format. Theora is really much more of a video file compression format than it is a playback file format, though it can be used that way also. It has no patents and is therefore considered open source.

    Fun fact: According to Wikipedia, "Theora is named after Theora Jones, Edison Carter's controller on the Max Headroom television program."

    Browsers that support the Ogg video file format include:

    HTML5 Multimedia Development Cookbook

  • WebM is the newest entrant in the online video file format race. This open source audio/video file format development is sponsored by Google. A WebM file contains both an Ogg Vorbis audio stream as well as a VP8 video stream. It is fairly well supported by media players including Miro, Moovidia, VLC, Winamp, and more, including preliminary support by YouTube. The makers of Flash say it will support WebM in the future, as will Internet Explorer 9. Browsers that currently support WebM include:

    HTML5 Multimedia Development Cookbook

There's more...

So far this may seem like a laundry list of audio and video file formats with spotty browser support at best. If you're starting to feel that way, you'd be right.

The truth is no one audio or video file format has emerged as the one true format to rule them all. Instead, we developers will often have to serve up the new audio and video files in multiple formats while letting the browser decide whichever one it's most comfortable and able to play. That's a drag for now but here's hoping in the future we settle on fewer formats with more consistent results.

Audio file formats

There are a number of audio file formats as well. Let's take a look at those.

  • AAC – Advanced Audio Coding files are better known as AACs. This audio file format was created by design to sound better than MP3s using the same bitrate. Apple uses this audio file format for its iTunes Music Store. Since the AAC audio file format supports DRM, Apple offers files in both protected and unprotected formats. There is an AAC patent, so by definition we can't exactly call this audio file format open source. All Apple hardware products, including their mobile iPhone and iPad devices as well as Flash, support the AAC audio file format. Browsers that support AAC include:

    HTML5 Multimedia Development Cookbook

  • MP3 – MPEG-1 Audio Layer 3 files are better known as MP3s. Unless you've been hiding under a rock, you know MP3s are the most ubiquitous audio file format in use today. Capable of playing two channels of sound, these files can be encoded using a variety of bitrates up to 320. Generally, the higher the bitrate, the better the audio file sounds. That also means larger file sizes and therefore slower downloads. There is an MP3 patent, so by definition we can't exactly call this audio file format open source either. Browsers that support MP3 include:

    HTML5 Multimedia Development Cookbook

  • Ogg – We previously discussed the Ogg Theora video file format. Now, let's take a look at the Ogg Vorbis audio format. As mentioned before, there is no patent on Ogg files and are therefore considered open source.

    Another fun fact: According to Wikipedia, "Vorbis is named after a Discworld character, Exquisitor Vorbis in Small Gods by Terry Pratchett."

    HTML5 Multimedia Development Cookbook

File format agnosticism

We've spent a lot of time examining these various video and audio file formats. Each has its own plusses and minuses and are supported (or not) by various browsers. Some work better than others, some sound and look better than others. But here's the good news: The new HTML5 <video> and <audio> elements themselves are file-format agnostic! Those new elements don't care what kind of video or audio file you're referencing. Instead, they serve up whatever you specify and let each browser do whatever it's most comfortable doing.

Can we stop the madness one day?

The bottom line is that until one new HTML5 audio and one new HTML5 video file format emerges as the clear choice for all browsers and devices, audio and video files are going to have to be encoded more than once for playback.

 

Creating accessible audio and video

In this section we will pay attention to those people who rely on assistive technologies.

How to do it...

First, we'll start with Kroc Camen's "Video for Everybody" code chunk and examine how to make it accessibility friendly to ultimately look like this:

<div id="videowrapper">
<video controls height="360" width="640">
<source src='//dgdsbygo8mp3h.cloudfront.net/sites/default/files/blank.gif' data-original="__VIDEO__.MP4" type="video/mp4" />
<source src='//dgdsbygo8mp3h.cloudfront.net/sites/default/files/blank.gif' data-original="__VIDEO__.OGV" type="video/ogg" />
<object width="640" height="360" type="application/
x-shockwave-flash" data="__FLASH__.SWF">
<param name="movie" value="__FLASH__.SWF" />
<param name="flashvars" value="controlbar=over&amp;
image=__POSTER__.JPG&amp;file=__VIDEO__.MP4" />
<img src='//dgdsbygo8mp3h.cloudfront.net/sites/default/files/blank.gif' data-original="__VIDEO__.JPG" width="640" height="360"
alt="__TITLE__" title="No video playback capabilities,
please download the video below" />
</object>
<track kind="captions" src='//dgdsbygo8mp3h.cloudfront.net/sites/default/files/blank.gif' data-original="videocaptions.srt" srclang="en" />
<p>Final fallback content</p>
</video>
<div id="captions"></div>
<p><strong>Download Video:</strong>
Closed Format: <a href="__VIDEO__.MP4">"MP4"</a>
Open Format: <a href="__VIDEO__.OGV">"Ogg"</a>
</p>
</div>

How it works...

The first thing you'll notice is we've wrapped the new HTML5 video element in a wrapper div. While this is not strictly necessary semantically, it will give a nice "hook" to tie our CSS into.

<div id="videowrapper">

Much of the next chunk should be recognizable from the previous section. Nothing has changed here:

<video controls height="360" width="640">
<source src='//dgdsbygo8mp3h.cloudfront.net/sites/default/files/blank.gif' data-original="__VIDEO__.MP4" type="video/mp4" />
<source src='//dgdsbygo8mp3h.cloudfront.net/sites/default/files/blank.gif' data-original="__VIDEO__.OGV" type="video/ogg" />
<object width="640" height="360" type="application/
x-shockwave-flash" data="__FLASH__.SWF">
<param name="movie" value="__FLASH__.SWF" />
<param name="flashvars" value="controlbar=over&amp;
image=__POSTER__.JPG&amp;file=__VIDEO__.MP4" />
<img src='//dgdsbygo8mp3h.cloudfront.net/sites/default/files/blank.gif' data-original="__VIDEO__.JPG" width="640" height="360"
alt="__TITLE__"
title="No video playback capabilities, please download the
video below" />
</object>

So far, we're still using the approach of serving the new HTML5 video element to those browsers capable of handling it and using Flash as our first fallback option. But what happens next if Flash isn't an option gets interesting:

<track kind="captions" src='//dgdsbygo8mp3h.cloudfront.net/sites/default/files/blank.gif' data-original="videocaptions.srt" srclang="en" />

What the heck is that, you might be wondering.

"The track element allows authors to specify explicit external timed text tracks for media elements. It does not represent anything on its own." - W3C HTML5 specification

Here's our chance to use another new part of the HTML5 spec: the new <track> element. Now, we can reference the type of external file specified in the kind="captions". As you can guess, kind="captions" is for a caption file, whereas kind="descriptions" is for an audio description. Of course the src calls the specific file and srclang sets the source language for the new HTML5 track element. In this case, en represents English. Unfortunately, no browsers currently support the new track element.

Lastly, we allow one last bit of fallback content in case the user can't use the new HTML5 video element or Flash when we give them something purely text based.

<p>Final fallback content</p>

Now, even if the user can't see an image, they'll at least have some descriptive content served to them.

Next, we'll create a container div to house our text-based captions. So no browser currently supports closed captioning for the new HTML5 audio or video element, we'll have to leave room to include our own:

<div id="captions"></div>

Lastly, we'll include Kroc's text prompts to download the HTML5 video in closed or open file formats:

<p><strong>Download Video:</strong>
Closed Format: <a href="__VIDEO__.MP4">"MP4"</a>
Open Format: <a href="__VIDEO__.OGV">"Ogg"</a>
</p>

 

HTML5 Multimedia Development Cookbook Recipes for practical, real-world HTML5 multimedia driven development.
Published: May 2011
eBook Price: $23.99
Book Price: $39.99
See more
Select your format and quantity:

 

        Read more about this book      

(For more resources on Multimedia development, see here.)

There's more...

In addition to the optional controls attribute for the new HTML5 audio and video elements, there's also the optional loop attribute. As you might guess, this example would allow the HTML5 video to keep playing over and over:

<video controls height="360" loop width="640">

Always consider accessibility

That final descriptive content we're defaulting to could be an alternate place to serve downloadable links to those using accessibility technologies. It would obfuscate the download ability from those who can see or hear, so you should determine if that approach works for you.

Browser support

The web browsers that have the best accessibility support for the new HTML5 audio and video elements include:

HTML5 Multimedia Development Cookbook

See more

You can keep track of HTML5 accessibility at http://html5accessibility.com. The site tracks what new HTML5 features like audio and video are available and in which browsers. You might be surprised to discover that as of this writing, Opera is the least accessibility friendly web browser, rating even below Microsoft Internet Explorer 9.

 

Crafting a slick audio player

Neutron Creations Principal and Co-Founder and front-end developer Ben Bodien created a customized HTML5 audio player for Tim Van Damme's The Box podcast at http://thebox.maxvoltar.com. Ben's creation is fast, intuitive, and slick. Let's take a deeper look at how he did it.

HTML5 Multimedia Development Cookbook

Ben's custom HTML5 audio player features an attractive photo of the person being interviewed (Shaun Inman, in this case), a play/pause button, a track indicating playback progress, and the ability to pop the HTML5 audio player itself out into a separate window if you so choose. That's it. Nothing more is needed. As an added touch, notice the detail of the slight transparency of the HTML5 audio player bar. Smooooth.

How to do it...

At first, Ben's markup seems deceptively simple:

<p class="player">
<span id="playtoggle" />
<span id="gutter">
<span id="loading" />
<span id="handle" class="ui-slider-handle" />
</span>
<span id="timeleft" />
</p>

Wait a minute, I hear you thinking, "Where's the HTML5 audio tag?!" Never fear. Ben's a clever guy and has a plan for this. But first let's examine what he's done so far.

<p class="player">

This one's simple so far. Ben creates a wrapping element (a <p> in this case) to put his player in. Could he have used a <div> instead? Perhaps. Do what makes the most sense for you and your project.

<span id="playtoggle" />

Ben then uses this self-closing (notice the trailing slash at the end) span for the play/pause toggle button.

<span id="gutter">
<span id="loading" />
<span id="handle" class="ui-slider-handle" />
</span>

Now, it gets interesting. Ben's "gutter" span houses the timeline track with a bar indicating the loading or buffering progress of the HTML5 audio file and the circular element indicating the playback head, which you can "scrub" back and forth if you choose.

<span id="timeleft" />

Finally, Ben uses another self-closing span to display the remaining time, in minutes and seconds.

The <span> element does the job, but it isn't very semantic, is it? Patrick H. Lauke was quick to point out that using focusable elements would go a long way toward making this approach accessible to those who rely on assistive technologies.

How it works...

Ben uses jQuery to detect support for HTML5 audio.

if(!!document.createElement('audio').canPlayType) {
var player = '<p class="player"> ... </p>\
<audio>\
<source src='//dgdsbygo8mp3h.cloudfront.net/sites/default/files/blank.gif' data-original="/path/to/episode1.ogg" type="audio/ogg"></source>\
<source src='//dgdsbygo8mp3h.cloudfront.net/sites/default/files/blank.gif' data-original="/path/to/episode1.mp3"
type="audio/mpeg"></source>\
<source src='//dgdsbygo8mp3h.cloudfront.net/sites/default/files/blank.gif' data-original="/path/to/episode1.wav" type="audio/
x-wav"></source>\
</audio>';
$(player).insertAfter("#listen .photo");
}

In that chunk of code, we can see how if the browser supports HTML5 audio, it gets served the full HTML5 <audio> tag, complete with fallbacks to .ogg, .mp3, and .wav, a file format we haven't worked with yet. Since the new HTML5 <audio> and <video> elements are file format agnostic, a .wav file should work just fine too.

Ben has created a simple bit of JavaScript to allow browsers to do what they feel most comfortable doing. Consider this approach if it makes sense for you and your project, but remember that you're relying on JavaScript to do the heavy lifting, as opposed to other approaches we've looked at that don't rely on it.

Note that if you use a <div> to contain the HTML5 video player, that JavaScript would have to be adjusted as well. Simply put, the <p class="player"> … </p> would be changed to <div class="player"> … </div>.

There's more...

So far, we've set the markup for the player and "sniffed" to see which file format any particular browser wants. Now, we need to add some functionality.

audio = $('.player audio').get(0);
loadingIndicator = $('.player #loading');
positionIndicator = $('.player #handle');
timeleft = $('.player #timeleft');
if ((audio.buffered != undefined) &&(audio.buffered.length != 0)){
$(audio).bind('progress', function() {
var loaded = parseInt(((audio.buffered.end(0) / audio.duration)
* 100), 10);
loadingIndicator.css({width: loaded + '%'});
});
}
else {
loadingIndicator.remove();
}

And then add a function to calculate the position of the playback head to determine the time remaining, being careful to include a leading zero if the time remaining calls for it.

$(audio).bind('timeupdate', function() {
var rem = parseInt(audio.duration - audio.currentTime, 10),
pos = (audio.currentTime / audio.duration) * 100,
mins = Math.floor(rem/60,10),
secs = rem - mins*60;
timeleft.text('-' + mins + ':' + (secs > 9 ? secs : '0' + secs));
if (!manualSeek) { positionIndicator.css({left: pos + '%'}); }
if (!loaded) {
loaded = true;
$('.player #gutter').slider({
value: 0,
step: 0.01,
orientation: "horizontal",
range: "min",
max: audio.duration,
animate: true,
slide: function() {
manualSeek = true;
},
stop:function(e,ui) {
manualSeek = false;
audio.currentTime = ui.value;
}
});
}
});

The only thing left to invoke is the play/pause button functionality.

$(audio).bind('play',function() {
$("#playtoggle").addClass('playing');
}).bind('pause ended', function() {
$("#playtoggle").removeClass('playing');
});
$("#playtoggle").click(function() {
if (audio.paused) { audio.play(); }
else { audio.pause(); }
});

Style and substance

After the simple markup and detailed JavaScript to create Ben's customized HTML5 audio player, the only thing left is to style it:

.player {
display: block;
height: 48px;
width: 400px;
position: absolute;
top: 349px;
left: -1px;
-webkit-box-shadow: 0 -1px 0 rgba(20, 30, 40, .75);
-moz-box-shadow: 0 -1px 0 rgba(20, 30, 40, .75);
-o-box-shadow: 0 -1px 0 rgba(20, 30, 40, .75);
box-shadow: 0 -1px 0 rgba(20, 30, 40, .75);
border-top: 1px solid #c2cbd4;
border-bottom: 1px solid #283541;
background: #939eaa;
background: -webkit-gradient(linear, 0% 0%, 0% 100%,
from(rgba(174, 185, 196, .9)), to(rgba(110, 124, 140, .9)),
color-stop(.5, rgba(152, 164, 176, .9)),
color-stop(.501, rgba(132, 145, 159, .9)));
background: -moz-linear-gradient(top, rgba(174, 185, 196, .9),
rgba(152, 164, 176, .9) 50%, rgba(132, 145, 159, .9) 50.1%,
rgba(110, 124, 140, .9));
background: linear-gradient(top, rgba(174, 185, 196, .9),
rgba(152, 164, 176, .9) 50%,
rgba(132, 145, 159, .9) 50.1%, rgba(110, 124, 140, .9));
cursor: default;
}
#playtoggle {
position: absolute;
top: 9px;
left: 10px;
width: 30px;
height: 30px;
background: url(../img/player.png) no-repeat -30px 0;
cursor: pointer;
}
#playtoggle.playing {background-position: 0 0;}
#playtoggle:active {top: 10px;}
#timeleft {
line-height: 48px;
position: absolute;
top: 0;
right: 0;
width: 50px;
text-align: center;
font-size: 11px;
font-weight: bold;
color: #fff;
text-shadow: 0 1px 0 #546374;
}
#wrapper #timeleft {right: 40px;}
#gutter {
position: absolute;
top: 19px;
left: 50px;
right: 50px;
height: 6px;
padding: 2px;
-webkit-border-radius: 5px;
-moz-border-radius: 5px;
-o-border-radius: 5px;
border-radius: 5px;
background: #546374;
background: -webkit-gradient(linear, 0% 0%, 0% 100%,
from(#242f3b), to(#516070));
background: -moz-linear-gradient(top, #242f3b, #516070);
background: linear-gradient(top, #242f3b, #516070);
-webkit-box-shadow: 0 1px 4px rgba(20, 30, 40, .75) inset,
0 1px 0 rgba(176, 187, 198, .5);
-moz-box-shadow: 0 1px 4px rgba(20, 30, 40, .75) inset,
0 1px 0 rgba(176, 187, 198, .5);
-o-box-shadow: 0 1px 4px rgba(20, 30, 40, .75) inset,
0 1px 0 rgba(176, 187, 198, .5);
box-shadow: 0 1px 4px rgba(20, 30, 40, .75) inset,
0 1px 0 rgba(176, 187, 198, .5);
}
#wrapper #gutter {right: 90px;}
#loading {
background: #fff;
background: #939eaa;
background: -webkit-gradient(linear, 0% 0%, 0% 100%,
from(#eaeef1), to(#c7cfd8));
background: -moz-linear-gradient(top, #eaeef1, #c7cfd8);
background: linear-gradient(top, #eaeef1, #c7cfd8);
-webkit-box-shadow: 0 1px 0 #fff inset, 0 1px 0 #141e28;
-moz-box-shadow: 0 1px 0 #fff inset, 0 1px 0 #141e28;
-o-box-shadow: 0 1px 0 #fff inset, 0 1px 0 #141e28;
box-shadow: 0 1px 0 #fff inset, 0 1px 0 #141e28;
-webkit-border-radius: 3px;
-moz-border-radius: 3px;
-o-border-radius: 3px;
border-radius: 3px;
display: block;
float: left;
min-width: 6px;
height: 6px;
}
#handle {
position: absolute;
top: -5px;
left: 0;
width: 20px;
height: 20px;
margin-left: -10px;
background: url(../img/player.png) no-repeat -65px -5px;
cursor: pointer;
}
.player a.popup {
position: absolute;
top: 9px;
right: 8px;
width: 32px;
height: 30px;
overflow: hidden;
text-indent: -999px;
background: url(../img/player.png) no-repeat -90px 0;
}
.player a.popup:active {background-position: -90px 1px;}
Content matters

It's a lot easier and more rewarding to spend the time to create something interesting when the content it's wrapping is compelling. The Box audio interviews are always a good listen—it's just too bad the author Tim Van Damme doesn't publish them more often. Here's hoping that changes in the future. Check it out at http://thebox.maxvoltar.com.

Being careful with details

This approach works beautifully when there's one new HTML5 audio or video element on a page at a time to deliver. If you require more than one, you're going to have to modify the JavaScript to tie into multiple "hooks" in the markup.

Summary

In this article we constructed our own player while still supporting accessibility.


Further resources on this subject:


HTML5 Multimedia Development Cookbook Recipes for practical, real-world HTML5 multimedia driven development.
Published: May 2011
eBook Price: $23.99
Book Price: $39.99
See more
Select your format and quantity:

About the Author :


Dale Cruse

Since 1995, Boston-area web developer Dale Cruse has been publishing websites for high-profile clients ranging from the U.S. Army to Bloomingdale's. He has been a guest lecturer at the Art Institute of New England & is currently pursuing speaking opportunities. Contact him at http://dalejcruse.com. He is also the author of the Champagne blog Drinks Are On Me at http://drinksareonme.net.

Books From Packt


MODx Web Development - Second Edition
MODx Web Development - Second Edition

Plone 3 Multimedia
Plone 3 Multimedia

Kentico CMS 5 Website Development: Beginner's Guide
Kentico CMS 5 Website Development: Beginner's Guide

Inkscape 0.48 Essentials for Web Designers
Inkscape 0.48 Essentials for Web Designers

Grok 1.0 Web Development
Grok 1.0 Web Development

Yii 1.1 Application Development Cookbook: RAW
Yii 1.1 Application Development Cookbook: RAW

Python 3 Web Development Beginner's Guide
Python 3 Web Development Beginner's Guide

Agile Web Application Development with Yii1.1 and PHP5
Agile Web Application Development with Yii1.1 and PHP5

Code Download and Errata
Packt Anytime, Anywhere
Register Books
Print Upgrades
eBook Downloads
Video Support
Contact Us
Awards Voting Nominations Previous Winners
Judges Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software
Resources
Open Source CMS Hall Of Fame CMS Most Promising Open Source Project Open Source E-Commerce Applications Open Source JavaScript Library Open Source Graphics Software