shockwave-sound.com
View Cart | License | Blog | Contact
[ Home ][ Testimonials ][ Help/FAQ ][ Affiliate program ][ CD collections ][ My tags ][ My orders ][ Custom music ]

Interactive Music in Games

We look at the possibilities and the impact

By: West B. Latta


Whether you're a game developer, game player, or carry only passing interest, it is plain to see the growth and advancement of the video game industry over the past decade. Robust graphics systems, ample disc space, bountiful system memory, and dedicated DSP have all become increasingly common on today's game platforms.

While this continues to drive the look, feel, gameplay, and sound of games, it can be said that, to a large degree, high-profile, large budget games have increasingly looked to film as their benchmark for quality. Achieving a true 'cinematic' feel to a game seems to be the hallmark of what we now consider 'AAA' games.

As game technology progresses, it is useful to look at not only the ways in which the technological aspects have improved, but also how design and artistic approaches have changed in relation to changing technology. With regards to music, what is it, specifically, about cinematic music that works so well? In this brief article, we'll take a look at how changing technology has altered our perception and application of what music in games should be.

Where We've Been

In the early years of games, music was predominatly relegated to relatively short background loops, generated by on-board synthesizer chips and various systems of musical 'control data' that would trigger these pre-scripted musical sequences. While not unlike our use of MIDI today, these systems were typically proprietary, and learning the language and programming of these systems was no mean feat for a workaday composer.

And yet, these were the 'iconic' years for video game music - where the Super Mario jingle, the Zelda theme, and many other melody-heavy tunes were indellibly imprinted on the minds of a generation. The limitations of the sound systems in these consoles were, in themselves, a barrier to creating anything other than relatively simple, catchy tunes.

As we progressed into the mid and late 1990's, technologies afforded us higher quality sounds - with higher voice counts, FM synthesis and even sample playback through the use of wavetable soundcards. Though the sounds were often highly compressed, the playback of real, recorded audio was a leap forward for home consoles and computer games. PCs and even some consoles moved to more MIDI-based or tracker-based musical systems, and so were somewhat easier to compose for than their earlier predecessors. Even so, musical soundtracks didn't drastically advance beyond the simple, background loop modality for quite some time.

In the mid to late 1990's, however, we began to hear a shift in game soundtracks. While simple backgrounds were still the norm, there was a sort of "mass-exodus toward pre-recorded background music"(1). there were a few higher profile titles that were afforded a greater percentage of budget, disc space, and system resources. This all added up to a slow, but perceptible shift toward the elusive 'cinematic' feel of film. I still remember watching the opening cinematic for Metal Gear Solid 2: Sons of Liberty and thinking to myself, "This can't be a videogame!" The quality of the voice acting, the soundtrack - the entire game felt, to me, like a dramatic leap forward. This was but one example among many titles that set out to push the boundaries of audio in games.

During the past 10 years, we have seen rapid and dramatic changes in the technology, artistry, and application of music in video games. Disc-based game platforms came to the fore with the release of the Sony Playstation 2 and Nintendo Gamecube early in the decade, and higher powered consumer PCs became increasingly more affordable. As a result, we hear a definite shift in musical scores, with significantly longer runtime, more complexity, more robust instrumentation and arrangement, higher quality samples, and even CD-quality orchestral recordings.

Where Are We Now?

At present, we're steeped in the current generation of gaming systems. Xbox 360, PS3, Nintendo Wii, and PC gaming have grown to include full HD video resolution and high-quality 5.1 surround sound. Low fidelity, synthesized or sample-based soundtracks have given way to fully arranged and orchestrated scores, recorded by world-class symphonies. While they haven't yet become household names like Zimmer, Williams or Goldsmith, well-known game composers are highly sought after as developers continue to strive for a more cinematic feel to their games. Truly, some game soundtracks rival those of major motion pictures in quality, scope and performance. This trend has even given way to a small 'video game soundtrack' industry, with record labels devoted specifically to releasing and promoting game sountracks to the mass market via CD and digital download.

Moreover, the sound of classic and contemporary video games have increasingly gained mainstream popularity as the synthesizers of old platforms such as Gameboy, C64, and NES have made their way into popular music by some of today's biggest musical artists. Likewise, game soundtracks are increasingly being presented to the public in unique ways. Bands such as The 1-Ups, The Minibosses and Contraband present re-arranged versions of old game tunes on live instruments, while live orchestras perform soundtracks via events such as Video Games Live.

While it is undeniable that the quality and scope of game music have, in some cases, grown to match that of film, it simply isn't enough. Games are an interactive medium, and as such, the presentation of musical soundtracks must also be able to adapt to changing gameplay. To get a truly immersive experience, the music in games must change on-the-fly according to what is happening in the game, while still retaining a cinematic quality. Rigidly scripted musical background sequences can't impart the same level of depth as music that truly matches the moment by moment action.

Surprisingly, adaptive and interactive music schemes have been used in games for longer than we realize. Even the original Super Mario Brothers music changed tempo as the player's time was running out. Yet making highly interactive, high-quality, orchestral scores adds a layer of complexity seldom attempted by many game developers. Instead, many continue to rely on simple geographic and 'event' triggers for our accompaniment, rather than a truly adaptive music system.

While some developers have attempted to tackle this issue themselves, many of their solutions are proprietary. To go a bit deeper into interactive music, we will instead turn our attention to middleware developers. Firelight Technologies - makers of the FMOD Ex audio sytem, and Audiokinetic - makers of Wwise - the two premier audio middleware providers for today's most popular AAA titles.

FMOD Ex

Firelight has taken a unique approach to dealing with interactive or adaptive music. Their FMOD Designer system allows two distinctly different approaches. Through their Event system, the composer can utilize multichannel audio files, or 'stems'. This allows certain individual instruments or sections to be added or subtracted based on game states, or any other dynamic information fed into the FMOD engine such as player health, location, proximity to certain objects or enemies, etc. This technique was used to great effect in Splinter Cell:Chaos Theory, where, depending on the level of 'stealth and stress' of the player, different intensities of music would begin to brought in. This type of layering is often called a 'vertical' approach to music system design.

 

The second approach FMOD takes is through their Interactive Music system. This system takes a more 'logic-based' approach, and allows the designer to define various cues, segments and themes that transition to other cues, segments or themes based on any user-defined set of parameters. Moreover, this particular system allows for beat-matched transitions, and time-synchronized 'flourish' segments. In this way, a designer or composer might break down their various musical themes into groups of smaller components. From there, they would devise the logic that determines when a given theme, for example "explore" is allowed to transition to a "combat" theme. This segment and transition based approach is often referred to as a 'horizontal' approach.

A system of this kind was used in the successful Tomb Raider: Legend. For that particular project, composer Troels Folmann used a system which he devised called 'micro-scoring', crafting a vast number of small musical phrases and themes that were then strung together in a logical way based on the players actions throughout the course of the game. For example, the player may explore a jungle area with an ambient soundtrack playing. As they interact with an artifact or puzzle, a seamless transition is made to a micro-score that is specific to that game event.

Audiokinetic Wwise

Wwise is relatively new to game development, gaining popularity over the past several years with its first major debut in FASA Interactive's Shadowrun. Since that time, Audiokinetic has rapidly enhanced their system, and their interactive music functionality takes a 'best of both worlds' approach.
With Wwise, it is possible to have both multichannel stems as well as a logic-based approach to music design. A composer can create a series of themes with time-synchronized transitions based on game events or states, while simultaneously allowing other parameters to fade various stems in and out of the mix. This system incorporates both a horizontal and vertical approach to music design, and it has resulted in an incredibly powerful toolset for composers and audio designers.

What's next?

The term 'videogames' now seems to encompass an entire spectrum of interactive entertainment in all shapes and sizes: casual web-based games, mobile phone games, multiplayer online games, and all manner and scope of console and PC games. It seems impossible to predict the future of interactive music for such a variety of forms, and yet we have some clues and ideas about what might be next for those AAA titles.

First and foremost - we can be sure that the huge orchestras and big-name composers aren't going away any time soon. In fact, as more games use the 'film approach' to scoring, it seems more likely that it will continue to be the standard for what we consider blockbuster games. Fortunately, tools like FMOD and Wwise have given composers and audio designers robust tools to adapt and modify their scoring approach to be more truly interactive with the game environment. As this generation of consoles reaches maturity, we will yet see some of the finest and most robust implementations of interactive music, I'm sure.

Even so, pre-recorded orchestral music - however well designed - will still have some static elements that cannot be changed or made truly interactive. Once a trumpet solo is 'printed to tape', it cannot easily be changed. Yet the technological leaps of the next-generation of consoles may present another option. It isn't unreasonable to think that we may see a sort of return to a hybrid approach to composing, using samples and some form of MIDI-like control data. While at first this may seem like a step backward, consider this: with the increasing quality of commercial sample-libraries of all types, and the extremely refined file compression schemes used on today's consoles, it is possible to think that the next Xbox or Playstation could, in fact, yield enough RAM and CPU power to load a robust (and highly compressed) orchestral sample library. The composer, then, is hired to design a truly interactive music score in a format akin to MIDI - note data, controller data, as well as realtime DSP effects. This score, then, would not only adapt in the ways we've described above (fading individual tracks in and out, and logically transitioning to new musical segments) - but because we have separated the performance from the sample data, we would now have control over each individual note played. The possibilities are nearly endless - realtime pitch and tempo modulation, transference of musical themes to new instruments based on game events, and even aleatoric or generative composing, which assures that a musical piece conforms to a given set of musical rules, yet never plays the same theme twice.


Click for larger image

Indeed, thesse possibilities and more are surely coming, and it is an exciting time for composers, audio designers, and gamers alike. For now, we can enjoy a new level of attention and awareness on game music. We are treated to truly orchestral experiences, if not completely adaptive and interactive ones. And yet, in the coming years, interactive music technology will continue to mature, and we will assuredly hear more sophisticated implementations of these technologies across the full spectrum of games. I encourage you to listen closely to the games you or your friends play over the next few years. The tunes you hear today are helping to shape a musical revolution for tomorrow.

Footnote: 1 - Gamepro - Next Gen Audio Will Rely On Midi

About the author: West Latta has been making strange noises for over 30 years. He has spent the last several years developing his craft in the game industry as composer, sound designer, and integration specialist. He is currently a Sound Supervisor for Microsoft Game Studios, as well as a freelance writer, composer and audio designer.
Other articles you may find useful:
Asset Management: How to keep track of sound clips using metadata and cataloguing. Three ways to build a sound library: Record sounds yourself, or find another way. Timeline of classical composers: Get an overview of the lives and times of classical music maestros. Depth and space in the mix, part 1: How to use reverb, pre-delay, EQ and delay to make your mix better. Depth and space in the mix, part 2: Further tips on improving the sound of your productions. Maximizing composer agreements: How you as a composer for games and other media can get the best out of the contract.
YouTube and music use: How "fingerprinted music" is causing advertisements on your YouTube video. Using Reverb to enhance your production: John Radford on the use and abuse of Reverb in music. Do the work: Music composers' tips and strategies for overcoming procrastination and getting the job done. Sound effects in music composition: How you can use sound FX in music production for games, film, media. Mixing as part of the composing process - part 1: Planning your instrumentation and approach. Mixing as part of the composing process - part 2: Making your sounds and instruments work with your composition to best effect.
Sound for picture - Faking it: Some great tips on making your audience feel they are there. Royalty Free music in 24-bit: Why we are upgrading to High Definition music downloads. Choosing music for a short film project: We look at some options for obtaining your musical score. Choosing music for a Documentary: Help and tips for obtaining your film soundtrack. Tips and Curiosities from Computer Game Music, pt 1: Piotr Koczewski discusses video game music. Tips and Curiosities from Computer Game Music, pt 2: More talk about composing music for video games.
Copyrights in Classical music and Public Domain music: We try to explain why "public domain music" still has rights attached to it. 1 year Shockwave-Sound.com exclusive: Some of our best music can be found only at Shockwave-Sound.com first year. Browse royalty-free music super quick: With our free Demo DVD-ROM you can skim through tracks quickly on your own PC. Getting started with voiceover: Things you need to know if you would like to make a living as a professional voice talent. Sennheiser PXC 450 noise canceling headphones: Video review of these classy noise reducing headphones Surviving your first composing gig: How to handle your client when composing music for video games or film/TV.
Creating radio ads with music and voice: We discuss some good practices and neat tricks for a great sounding ad spot. Recording sound for perspective: Good sound recording practice for a realistic result. Making a long-playing sound or Audio-CD starting out from a short, looping sound file. Creative workflow in Sonar, part 1: Save time and frustration while working in Sonar music production. Surround music in video games: Rob Bridgett discusses the viability and aesthetics of 5.1 sound heaven Shockwave-Sound's sister site for sound-fx
We introduce our new site for listening to and buying sound effects.
Cue the Music, Part 1: Using copyrighted music in your project or presentation Cue the Music, Part 2: We look at Five ways to get music for your project without breaking copyrights. Cue the Music, Part 3: How to use Royalty Free Music to the best effect for your project. Working with audio in Sony Vegas, Part 1: Importing & Timestretching audio files Working with audio in Sony Vegas, Part 2: Adding FX, Mixing & Rendering Audio Files Samson Zoom H4 portable recorder: An in-depth product review of this handy sound recording unit.
How to get music on your web site: We explain how to Embed music on a web page and how to make a Flash that plays music. Music rights terms and expressions: Podcast safe music, Sync License, Royalty Free Music, Performance Rights... Confused yet? Royalty free music explained: What really lies behind this term? We talk a little music licensing history and look at this expression. How to build a music track from loops: Do this to get the "set of music loops" to play as a longer music track YouTube Safe Music: How to find music for your YouTube video and properly credit the composer and publisher. Common myths and misunderstandings about music rights: We try to clear up some of these.
Orchestral MIDI arrangement: A beginner's guide to the Orchestral MIDI Mockup. A guide to virtual pianos: We take a look (and a very close listen) to virtual piano plug-ins. Strengthen your 3D animation with audio: How to use royalty-free music and sound-fx with 3D animation Ideas for Effectively Using Sibelius and Pro Tools 8: We look at ways to streamline and optimize your composing work. Cleaning up noisy dialogue: Get rid of background noise and improve sound quality of voice recordings Interactive Music in Games: We look at ways to make videogame music react and respond to the players actions.
Writing music for games, part 1: Video games composer Kole on some issues to keep in mind. Writing music for games, part 2: Kole dissects another video game music project. Writing music for games, part 3: Finding a way to compose music for Facebook games and stay within the boundaries. Writing music for games, part 4: How to make the most of what little resources you have available.   Music Production hardware and software tips, part 1: Important issues you need to consider when selecting your tools, computer, CPU, RAM, software etc.
Music Production hardware and software tips, part 2: Useful tips for choosing your hardware for music production. Composing music for cellphones / mobile phones - Part 1: Tips and tricks of the trade. Composing music for mobile phones / cellphones - Part 2: More useful info for producers. Observations of Memorable Themes: We discuss music composition and how to make your melodies memorable. Developing musical ideas in video games - Part 1. Ways of making the music work for the game. Developing musical ideas in video games - Part 2. More thoughts on scoring music for video games.
Ideas for creating unique musical colours: Tips to try to make it sound and feel 'different' Choosing the right classical music, part 1: We recommend 10 pieces of bombastic, powerful, awe inspiring classical music. Choosing the right classical music, part 2: In this part we recommend 10 beautiful, soft, heavenly and emotional classical music tracks      

Would you like to contribute an article to Shockwave-Sound.com? We will pay you $150.00, and we will include your bio, a link to your web site, and if you wish, a quick plug of your product or service. Shockwave-Sound.com is used by almost 4,000 unique visitors every day. Contact us if you have an article idea/pitch for us that you feel is useful, relevant and well written. First, though, you may want to read this blog post about article requirements.

Copyright notice: This article and all other text on this web site is under Copyright to Shockwave-Sound.com. This text may not be copied, re-printed, re-published, in print or electronically, in whole or in part, without written permission from Shockwave-Sound.com.

[Switch to Classic Navigation]