3D/360° music production and multi-track recording using the ZYLIA microphone array

5/24/2018

Introduction

Typically 3D audio is reproduced using whether multi loudspeaker setups (5.1, 7.1, 22.2, e.g. Dolby Atmos) or headphones. Therefore, there are also two main recording setups related to 3D audio recording:

Multi-microphone setup – that is, each microphone is put in the same place where the target loudspeaker is placed. In that case, it is not easy to upmix or downmix the number of channels and usually it is necessary to use one microphone setup for one loudspeaker setup
Dummy head – in that case an artificial head is used where a pair of microphones is placed in the artificial ears. This type of recordings can be directly reproduced with headphones.

There is also a third approach, recording of sound objects or folleys and use them in game engine mixers or 3D audio renderers.

MUSIC AND AUDIO PRODUCTION PROCES

Now you can divide audio production process into four main stages. Pre-production stage is the most engaging. It depends on many factors such as the characteristics of sound sources you are going to record, their position in relation to each other, acoustic conditions or simply your budget for equipment. It requires selection of microphones and their right placement in recording landscape. In many cases, it is connected with multiple test recordings you need to make and microphone placement changes. The more experienced audio engineer you are the less time it should take and the better results you will obtain.

The second stage – recording. It should be straightforward. Start recording, repeat it, record a few takes and that’s it. But what if you had just one chance to record a scene? Or you didn’t notice that there was something wrong with your microphone placement? You go home and then it shows up that your recordings are poor, that some strange noises from a background are noticeable. What can you do? You can repeat a whole recording day or try to mask it in the mixing process. In both cases, it equals additional hours of your extra work.

You never know what happens during a field recording. Being able to correct a “bad” microphone placement after the recording would be a great option!

Felix Andriessens, Sound designer, BERLIN

How to save some time and reduce the risk of bad microphone placement or unwanted noises in your recordings? You can simplify the first two steps by using a ZYLIA microphone array. The system is capable of recording a full 360˚ music scene and representing it in 19-channel ZYLIA format or B-format (Wikipedia). By using one of those formats it is possible to save the whole sound field information in a single recording which can be further (in post-production) decomposed into any number of Virtual Microphones. In this case, you are using one device for recording the whole 360˚ audio scene. You simply capture everything with a high spatial resolution.

HOW IT WORKS? THE TECHNOLOGY BEHIND ZYLIA ZM-1

ZYLIA is a 3rd order Ambisonics microphone array. That is, it records 360˚ audio in the high spatial resolution that further on can be decomposed into different audio layers that represent individual sound sources, instruments or background noises.
Instead of using multiple spot microphones you put ZYLIA in the middle of the audio scene. The recorded audio tracks are processed by the dedicated software which allows creating so-called Virtual Microphones. They provide an adaptive isolation of sound sources. As a result, you receive separated tracks of the recorded sound sources. The software allows for steering a Virtual Microphone at any direction in the 3D space. In post-processing, you can now easily manipulate with recording and select exact sound sources, ambient or room reflections. What is more, you can identify unwanted sounds and remove them from your recording. Virtual Microphones operate in two modes:

The main advantage of this approach is that all those adjustments may be done without repeating the pre-recording and recording step because it is possible to change a spatial characteristic of Virtual Microphone in the post‑production phase

Beamforming – it simulates different types of spatial microphone characteristics, starting from super cardioid, up to an omnidirectional pattern.
Separation mode – additional post‑processing algorithms for simulating close microphone placement. The ambient sounds and background noises are reduced and the separation between the sound sources is even higher.

Fig. Sound source separation. ZYLIA – 3rd order Ambisonics Microphone.

Fig. The concept of Virtual Microphones in ZYLIA.

3D AUDIO IN A NEW WAY

ZYLIA ZM-1 microphone array can be used in many applications. Let us name a few: binaural recordings, object recording, Ambisonics 360˚ or multichannel 5.1, 7.1 and 22.2.

Binaural recordings

Binaural recording is a well-known method of recording sound scene with two microphones, creating a 3D stereo sound sensation for the listener, he can actually hear the effect of being in the middle of the sound scene. This effect is often created using a technique known as "dummy head recording" but you can also prepare a binaural recording with ZYLIA microphone array. There are two options to achieve that effect with ZYLIA:

set two Virtual Microphones to a binaural mode or
use 2 microphone capsules that are on the opposite side of the ZM-1 – dummy head emulation

https://www.youtube.com/watch?v=OCA2Leqpw8E
https://www.youtube.com/watch?v=7rlUfsLs-E0

Fig. Ambient and binaural recordings with ZYLIA ZM-1.

Object recording

Typical audio mix used for film or game production contains lots of sound objects that are further placed in the 3D space by using dedicated spatialization plugin. However, recording of each object separately is time-consuming and prone to errors such as capturing of unwanted noises.
By using ZYLIA ZM-1 it is possible to record a full 360˚ audio scene and then in the post‑production extract sound objects – instruments, dialogs, etc. Extraction process provides:

Raw audio extracted by a virtual microphone
3D object position and trajectory of moving sound source/object

The extracted sound object can be then used in the mixing process.

3rd order Ambisonics and 360 movies

Ambisonics audio (B-format) can be used directly in many audio and video players such as YouTube or Facebook 360˚. 360˚ players convert Ambisonics audio into binaural (two channel) signal. The higher the microphone order the better the spatial resolution (ability to extract and listen to point sources in the surrounding sound scene) and rotation effect during playback.

It is highly recommended to use the highest possible microphone array, e.g. 3rd order. The high spatial resolution provides a truly immersive experience where the listener is able to easily localize sound sources in the 3D space. On the other hand, using lower order microphones result in fuzzy sounds during binaural or loudspeaker playback.

360˚ audio comes usually with 360˚ videos. For that context, ZYLIA ZM-1 is often combined with a range of popular cameras e.g. Insta360.

3rd order Ambisonics plus objects

There are situations where Ambisonics sound is not enough to fully represents 3D audio space. Usually, it is a scene where there is a lot of sound sources which lead to problems with sound localisation. There are at least two solutions to solve this problem:

Spatial Encoding - Record sound with ZYLIA, extract dominant sound sources or directions and encode that sound objects into Ambisonics domain. You can use Facebook 360˚ encoder for this purpose. This approach gives a good control over spatial resolution. Here is an example of such mixing process with sample files -- Create your own spatial Ambisonics mix with ZYLIA!
Amplification of selected directions – in that context Ambisonics sound (b-format) is mixed together with sound objects. That latter can be the output of Virtual Microphone obtained from the same recording, however, focused on particular sound or direction. In that context, it is possible to improve localisation of perceptually (or artistic) most important sounds.

TIP: in order to achieve a good spatial effect in your 3D mixes use sound objects as the main hearing cue and Ambisonics sound for ambient sound representation. Sound objects can be achieved in one of the two methods explained above.

Surround multichannel

Cinematic audio and music production often requires recording a surround sound e.g. 5.1, 22.2 (Dolby Atmos). In that context, it is possible to use ZYLIA ZM-1 and ZYLIA Studio PRO plugin. The latter consists of surround preset demonstrated in the below picture.

Basically, virtual microphones point in the position where the loudspeakers are placed during playback. Such preset can be used live or in the post-production. It is important to mention that ZYLIA 19-channel format or B-format allows for immediate upmix or downmix number of surround channels. You only need to change a preset at the same recording. Such sound production is not possible using traditionally positioned microphones, because every surround setup requires a new microphone setup and recording.

Fig. ZYLIA Studio PRO.

A notable example of surround sound production is a cinematic trailer of First Aid Kit band recorded by Sony Music UK. During that production, Sony Music UK (Kris Winter) used a ZYLIA ZM-1 microphone attached to a standard filming equipment.

Both artists played and sang and moved around the camera in both directions. Sound was captured by ZYLIA in 19-channel format. In the post-production, a 5.1 preset was used in ZYLIA Studio PRO plugin. Synchronized audio and video was prepared to listen in the cinematic 5.1 environment.

If there will be a need in the future to upmix the number of channel, e.g. 22.2 then the only thing to do is select 22.2 preset in ZYLIA Studio PRO using the same recording.

Soon this trailer will be available in cinemas in UK.

1 Comment