By Hocheol Yang
When we pass by construction sites or when we see a crying baby we can easily identify where they are because we can easily locate the sources of sounds. Similarly, when we direct our head or body toward the sources of sounds then the sound should be louder and clearer. This is such a natural experience in our daily life to survive. Making similar virtual environments are comparatively simple on 3D modeled spaces and multi channel surround systems seem to enhance these natural aural behaviors in general. How about the multi-channel surround audio systems in virtual reality, especially for Oculus Rift?
Before we move on to the main topic, let’s look into the debates between binaural versus multi-channel sound systems. I believe you already know that the most of the music albums are not actually recorded to reflect the actual performance environment of artists. Instead, recording engineers virtually decide positions of instruments and vocals by adjusting pan, volume, and filters. The key factor for these recordings is even or clearer bandwidth distribution. For example, if base and drums, which use lower frequencies, are positioned in the same channel with the similar volume then the recorded sounds are not clearer because they interfere with each other. Some classical music and live concert albums use binaural recording techniques, but most recorded music use stereo channels to have a clearer bandwidth separation. This means the locations of the drums and guitars are not relevant to the actual stage of musicians. Although a clearer separation is also one of the advantages of multi channel surround system, it is more difficult to verify the advantage for delivering better staging than binaural systems.
You may want to compare the differences between wearing stereo headphones and multi channel speakers when you play games supporting a multi channel system. In fact, many reviewers have tested this before, but is is difficult to find a good comparison because the reviewers do not use identical speakers. Most results of these reviews are based on differences from different speakers, sound processing units, and volumes. This may be my next project.
Binaural audio supporters’ claim is that with proper binaural recordings or simulations and with quality stereo speakers we do not need multiple speakers to generate multi-dimensional audio space because it is rare to find someone with more than two ears. This is similar with what Oculus Rift does to our eyes. This method is simple but effective and the most widely used method for the current software available on the Oculus Rift. In order to set up 5.1 or 7.1 sound software developers need to set up 5 or 7 different points around users to capture sound from the virtual world. Players also need to have multiple speakers that correspond to each sound capturing point. Binaural audio supporters argue these extra costs are not necessary. They claim having a quality pair of speakers that have flat outputs across all bandwidth is more important. In fact, when you play multi-channel movies or games on consoles with good quality binaural surround emulation system known as Dolby surround (this surround sound is totally different with the 3D sound emulations that reconstruct 3D staging from two channel sources) you will definitely feel sound staging, which is similar with the sound when you have multi-channel speakers.
Multi-channel speakers supporters’ claim is that even if we have two ears, multiple speakers can provide clearer spatial separation than just using two speakers. In other words, only two channels do not convey enough surround sound information. In fact, when you play multi-channel movies or games on consoles with Dolby surround what you will experience is less clear than the sound when you have with physical multi-channel speakers. This is a sound claim when we see the examples of stereo recording, but the advantage for delivering right staging to users is not quite tested. In fact, the virtual staging for clearer separations are more artificial than real. It means that we can hear individual instruments’ sound more clearly, but they are not positioned to how they are played when artists perform.
Debates between binaural versus multi-channel sound systems are still ongoing issues since there have been no legitimate studies until today. You can read more about this on a BBC’s recent experiment. Either way, having interactive surround sound is one important factor to increase more sense of presence on VR devices.
I Surround sound on HMD with 3D graphics
3D gaming development tools usually offer ambient sound tools such as Ambient Sound Actor on Unreal Engine kit. These tools usually have distance algorithms and virtual microphones that capture sounds around player and transmit the corresponding speakers based upon the user’s movements and directions. For example, when there is a singing bird on the scene which is assigned as an individual sound actor the bird will have its own volume set by developers. The volume will be at its maximum when a virtual microphone, which will transformed to the users left or right ear’s speaker, and the bird have 0 distances. As the distance increase the volume will be attenuated. The distance is dependent on movements of both the user’s and the bird’s animation.
In this case, developers can put 2 virtual microphones for 2 or 2.1 audio systems, or 5 or 7 microphones for 5.1 or 7.1 audio systems. If there is no significant difference among 2.1, 5.1, and 7.1 audio systems then the binaural supporters’ claims will be supported especially when users use headphones.
I Surround sounds on HMD with camera recorded footage
This is a more complex problem than the previous environment. Since camera recorded footage should include multiple sound sets and video playback software should simulate multi-channel sounds based on the user’s behaviors, but this is currently not available on any of available content and software for capturing the physical world. This might be the reason why most 360 videos do not provide interactive audio based on user’s operations.
Photo: BINAURAL AUDIO RIG/LINK
As you see in this picture it looks like there is an attempt at making interactive 360 video with binaural sound recording.
Photo: BINAURAL AUDIO and Oculus Rift/LINK
Although, the attempt was innovative, it looks like matching the recorded sound with user’s movement is not easy or accessible. One of the main issues with it is the lack of a video player. There is no known interactive video player that can mix down multi channel recordings to match with users interaction. Conceptually it is simple, so there should be some research lab or patent, but it is not something we can easily access.
Still, it looks like most interactive videos are not supporting interactive surround sounds. It is still in a developmental phase and we will soon see the standards of it.
Updates on 2017
Now the spatial audio feature is enabled on YouTube and Facebook from 2017!