Audio & Video

Pulse Coded Modulation

PCM streams are made up of a sample rate: how frequently the amplitude is measured (ex. 320 Mbps), and bit depth: how many possible digital values can be represented by one sample.

Compression

The Digital to Audio Converter (DAC) on your computer's soundcard expects PCM (HD Audio or AC'97), but PCM is large and therefore expensive to store and transmit.

Lossy vs Lossless Compression

Lossless compression allows us to more efficiently represent the same uncompressed data. Lossy compression sacrifices fidelity for size. Lossy compression allows you to store thousands of songs on your portable media player. Lossless compression allows you to store and play media that sounds fantastic on your expensive audio playing equipment.

Codecs

Codecs are how the media data is represented, or encoded. Some codecs are lossy like MP3, some are lossless like FLAC. Most codecs are encumbered by patents. :(

Containers

Containers such as WAV, MP4, and Ogg represent the meta-data of a media file such as artist or duration, subtitles, etc. Containers, in addition to their meta-data, will contain streams of audio or video encoded via a specific codec.

File Extension

Sometimes used by OS to determine what program should open what file. Unreliabe; anyone can change the file extension of a given file, but that does not change the encoding (how the bits are arranged) or the container (how the meta-data and streams are packaged).

Playlists

Playlist files are used by media playing applications to play subsequent mediums. Browsers do not understand playlist files, but playlist files can easily be parsed in JavaScript.

Protocol

How are the bits transferred from one machine to another? On the Web, HTTP is the most familiar, but there are other protocols used when streaming media such as RTSP and ICY.

Know Your Terms

Codecs, containers, file extensions, playlist files, and protocols are not equivalent. Ex. Media A could be aac encoded, in an MP4 container, with a .m4a extension, listed in a m3u playlist file, served over ICY.

Hardware Acceleration

Media can be decoded to PCM in software or in hardware. Application Specific Integrated Circuits (ASICs) can be faster and more power efficient than General Purpose Processors. Mobile friendly. They can also be patent encumbered.

Source tags

Both <audio> and <video> tags support nested <source> tags of different encodings.


            <audio controls>
              <source src="foo.ogg" type="audio/ogg"/>
              <source src="foo.mp3" type="audio/mp3"/>
              Sorry, you're browser doesn't support ogg or mp3 codecs
              (or possibly audio tags).
            </audio>

Feature Detecting Codec Support


            function canPlayOgg () {
              var a = document.createElement('audio');
              return !!(a.canPlayType &&
                        a.canPlayType('audio/ogg; codecs="vorbis"')
                         .replace(/no/, ''));
            };

            canPlayOgg(); // true or false

More feature detection

Audio

            
              <audio controls>
                <source src="../media/lateralus_clip.ogg" type="audio/ogg"/>
                <source src="../media/lateralus_clip.mp3" type="audio/mp3"/>
              </audio>

Out of Control


            <audio id="noControls">
              <source src="../media/lateralus_clip.ogg" type="audio/ogg"/>
              <source src="../media/lateralus_clip.mp3" type="audio/mp3"/>
            </audio>
            <button id="play">Play/Pause</button>
            <button id="volUp">Volume+</button>
            <button id="volDown">Volume->/button>
            <input id="seek" type="range" value="" max=""/>
            <script>
              var audio = document.getElementById("noControls");
              var play = document.getElementById("play");
              var volUp = document.getElementById("volUp");
              var volDown = document.getElementById("volDown");
              var seek = document.getElementById("seek");
              var playing = false;

              play.addEventListener("click", function (e) {
                playing = !playing;
                if (playing) {
                  audio.play();
                } else {
                  audio.pause();
                }
              });

              function changeVol (up) {
                if (up) {
                  audio.volume += 0.1;
                } else {
                  audio.volume -= 0.1;
                }
              };

              volUp.addEventListener("click", function () {
                changeVol(true);
              });

              volDown.addEventListener("click", function () {
                changeVol(false);
              });

              audio.addEventListener("timeupdate", function (e) {
                seek.value = audio.currentTime;
              });

              seek.min = seek.value = audio.seekable.start(0);
              seek.max = audio.seekable.end(0);
              seek.addEventListener("change", function (e) {
                audio.currentTime = seek.value;
              });
            </script>

Events


            <audio id="audioA" controls>
              <source src="../media/lateralus_clip.ogg" type="audio/ogg"/>
              <source src="../media/lateralus_clip.mp3" type="audio/mp3"/>
            </audio>
            <br/>
            <audio id="audioB" controls>
              <source src="../media/lateralus_clip.ogg" type="audio/ogg"/>
              <source src="../media/lateralus_clip.mp3" type="audio/mp3"/>
            </audio>
            <script>
              var audioA = document.getElementById("audioA");
              var audioB = document.getElementById("audioB");

              audioA.addEventListener("ended", function () {
                audioB.currentTime = 0;
                audioB.play();
              });

              audioB.addEventListener("ended", function () {
                audioA.currentTime = 0;
                audioA.play();
              });
            </script>

More events at Are We Playing Yet?

Video


            <video controls>
              <source src="../media/bunny.webm" type="video/webm"/>
              <source src="../media/bunny.mp4" type="video/mp4"/>
            </video>

Mixing Video and Canvas 2D

Demo