Interactive Binaural Audio
3D Audio with Binaural Signals
Binaural audio signals are signals intended for reproduction at the left and right ears of a listener. If they contain the same auditory cues that also occur in natural listening, this creates a very realistic spatial impression.
A simple way to create binaural signals is to place microphones at the ears of a person or an artificial head. These microphones capture a real acoustic scene in the same way that the person or artificial head hears it. Alternatively, binaural signals can also be created synthetically from spherical microphone array recordings or purely virtual acoustic scenes.
Binaural signals need to be reproduced at the ears of the listener for the spatial impression to work. The easiest way to achieve this is the use of headphones. Loudspeaker playback of binaural signals is possible using crosstalk cancellation.
Try it yourself: put on your headphones and listen to binaural recordings created by IKS on our YouTube channel.
Interactive Binaural Audio
Human listeners naturally move their heads while listening. In real listening situations, the sounds at the ear change with movements of the head and ears. For example, turning the head towards a sound source leads to the sound being heard from the front, while turning the head away makes the sound come from the side.
For realistic binaural reproduction, the binaural signals need to match the listener's head movements. These movements can be measured using head tracking devices. The possibilities for adding head-tracking to binaural reproduction depend on when and how the binaural signals are obtained:
- If the listener's head orientation is known when the binaural signals are obtained, it can be incorporated in the signal generation process. This is often the case for synthetically generated binaural signals and widely used in Acoustic Virtual Reality.
- If the binaural signals are obtained before the listener's head orientation known, the signals need to be retroactively modified. This can be achieved using the Binaural Cue Adaptation (BCA) algorithm developed at IKS.
Binaural Cue Adaptation (BCA)
The Binaural Cue Adaptation (BCA) method adapts binaural signals to listener head movements after the signals have been generated or recorded. It can therefore be applied
- to binaural recordings made with artificial heads or head-mounted microphones, which could not be made interactive otherwise
- in devices such as headphones, which only have access to the audio signals and cannot influence the signal generation in the media source.
Block Diagram
The algorithm works in the time-frequency domain. It changes the apparent source directions in the binaural signals by applying a signal-adaptive time-varying filter. The filter depends on the measured current head orientation of the listener as well as on the spatial properties of the signal.
Applications and Benefits
Possible applications of the Binaural Cue Adaptation method include
Benefits of BCA in hearables and headphones include
- improved naturalness and immersion, solution of the in-head perception problem
- compatibility with any headphone audio content
- implementation entirely in the headphones, independent of the media device / smartphone
- low head-tracking latency
Audio Examples
The following audio examples illustrate the subjective differences between variants of the Binaural Cue Adaptation (BCA) algorithm proposed in [Nagel20].
An input signal is created by mixing a speech signal with incoherent white noise at different Coherent-to-Incoherent power ratios (CIRs). Variants of the BCA algorithm are used to artificially rotate the scene by 60°. The result can be compared with the simulated reference signal.
References
[Nagel21]
Sebastian Nagel and Peter Jax
On the Use of Additional Microphones in Binaural Cue Adaptation
ITG-Fachtagung Sprachkommunikation, September 2021
[Nagel20]
Sebastian Nagel, Daniel Haupt, and Peter Jax
Coherence-Adaptive Binaural Cue Adaptation
AES Conference on Audio for Virtual and Augmented Reality (AVAR), August 2020
[Nagel18a]
Sebastian Nagel and Peter Jax
Dynamic Binaural Cue Adaptation
International Workshop on Acoustic Signal Enhancement (IWAENC), September 2018