by Eduardo Patricio

In general VR related workflows can be complex and everyone seems to be looking for standard solutions. Here, we will show you, step by step, how to prepare a 360 video with spatial audio in, possibly, the shortest way! 

Required gear:

• ZYLIA ZM-1 microphone array;
• A 360 camera (e.g. Insta 360 One X);
• A computer.

​After following steps A, B and C, you’ll have a video file with 1st order Ambisonics spatial audio that can be played on your computer with compatible video players (e.g. VLC) or uploaded to YouTube.

​Having said that, a small horizontal offset is not the end of the world 

With the gear in place, start recording both audio and video and clap in between the mic and the camera. The clap sound spike can be used to sync the footage later.

• Bring your files (the 19-channel audio from the ZM-1 and the video one from your camera) onto the DAW;
• Lower the video volume;
• Align the clap position;
• Trim the ends;
• Adjust the volume of the audio file;
• Add ZYLIA Ambisonics Converter Plugin to the audio track and adjust the settings;
• Change the master output to 4 channels;
• Render!

Here’s a video showing all the sub-steps in Reaper:

​Note
If you need to check how the recording sounds, add a binaural decoder plugin (e.g. IEM Binaural decoder) to the audio track, after ZYLIA Ambisonics Converter.
 

That’s it!
Now you can enjoy the spatial audio

• By playing it on your computer, using VLC player, for example (very few players will handle 360 video + Ambisonics correctly); or…
• By uploading it to YouTube. 

*Software tools used
​

Mounting:
https://www.amazon.com/Stage-MY550-Microphone-Extension-Attachment/dp/B0002ZO3LK/ref=sxbs_sxwds-stvp?keywords=microphone+clamp+arm&pd_rd_i=B0002ZO3LK&pd_rd_r=6860690f-2adc-4b00-a80e-de436939ed2b&pd_rd_w=GlE2J&pd_rd_wg=qrTTx&pf_rd_p=a6d018ad-f20b-46c9-8920-433972c7d9b7&pf_rd_r=GGS60M0DGQ5DQF44594V&qid=1575529629
 
https://www.amazon.com/Aluminum-Microphone-Swivel-Camera-Monitor/dp/B07Q2V6CBC/ref=sr_1_186?keywords=microphone+boom+clamp&qid=1575529342&sr=8-186

Amazon:
https://www.amazon.com/Neewer-Adjustable-Articulating-Mirrorless-Camcorders/dp/B07SV6NVDS/ref=sr_1_205?keywords=microphone+clamp+arm&qid=1575531393&sr=8-205
Allegro generic alternative for us to test: https://allegro.pl/oferta/ramie-przegubowe-11-magic-arm-do-kamery-8505530470
 

We are happy to announce the new release of ZYLIA Ambisonics Converter plugin v1.4.0. 

​What would happen if on a rainy and cloudy day, during a walk along a forest path, you could move into a completely different place thousands of kilometers away from you? Putting the goggles on would get you into a virtual reality world, you would find yourself on a sunny island in the Pacific Ocean, you would be on the beach, admiring the scenery and walking among the palm trees listening to the sound of waves and colorful parrots screeching over your head.
​
It sounds unrealistic, but such goals are determined by the latest trends in the development of Augmented / Virtual Reality technology (AR / VR). Technology and content for full VR or 6DoF (6 Degrees-of-Freedom) rendered in real time will give the user the opportunity to interact and navigate through virtual worlds. To experience the feeling of "full immersion" in the virtual world, realistic sound must also follow a high-level image. Therefore, only each individual sound source present in virtual audio landscape provided to the user as a single object signal can reliably reflect both the environment and the way the user interacts with it.

There are many possibilities of using a 6DoF VR technology. You can imagine exploring a movie plan in your own pace. You could stroll between the actors, look at the action from different sides, listen to any conversations and paying attention to what is interesting only for you. Such technology would provide really unique experiences.

​A wide spectrum of virtual reality applications drives the development of technology in the audio-visual industry. Until now, image-related technologies have been developing much faster, leaving the sound far behind. We have made the first attempts to show that 6DoF for sound is also achievable.

​It's extremely challenging to record high-quality sound from many sources present in the sound scene at the same time. We managed to do this using nine ZYLIA ZM-1 multi-track microphone arrays evenly spaced in the room.

In our experiment the sound field was captured using two different spatial arrangements of ZYLIA ZM-1 microphones placed within and around the recorded sound scenes. In the first arrangement, nine ZYLIA ZM-1 microphones were placed on a rectangular grid. Second configuration consisted of seven microphones placed on a grid composed of equilateral triangles.

​Microphone signals were captured using a personal computer running GNU/Linux operating system. Signals originating from individual ZM-1 arrays were recorded with the specially designed software. 

​We recorded a few takes of musical performance with instruments such as an Irish bouzouki (stringed instrument similar to the mandolin), a tabla (Indian drums), acoustic guitars and a cajon.

To present interesting possibilities of using audio recorded with multiple microphone arrays we have created a Unity project with 7 Ambisonics sources. In this simulated environment, you will find three sound sources (our musicians) represented by bonfires among whom you can move around. Experiencing fluent immersive audio becomes so natural that you can actually feel being inside of this scene.

​During exploration experiments, MPEG Standardization Committee has realized that there is a lack of test material for audio and video experiments into MPEG-I Immersive Video. Our recordings have been accepted as a reference material in MPEG Standardization Committee. Each of the experts involved in standardization works on MPEG-I can now use the Zylia recordings to develop and test audio technologies for VR/AR applications. 

By Eduardo Patricio

​In this post we present two videos in different formats, but edited from the same source material captured on the 20th of June 2018, at Barigui park (Curitiba, Brazil).

The audio was recorded with the ZYLIA ZM-1 3rd order Ambisonics spherical microphone array while the video was captured by a 360-degree camera (Gear 360).
 
Below, you can watch both videos and find some information on how to achieve the two different results, with focus on preparing the audio recorded with the ZM-1 microphone for each scenario.

Interactive, immersive video with full 3D sound 
​(media components: 360-degree video + Ambisonics audio) 

​Non-interactive video with fixed perspective 3D sound
(media components: Tiny planet” video + binaural audio)

The microphone and the camera were placed on a single camera stand with a small clamped extension arm (see picture below). Both devices were aligned vertically with a small horizontal offset. We made sure the microphone and the camera always had the same relative facing direction (front of the microphone aligned with the camera side where the recording button is found).

For scenario B, we used the video from Gear 360 in ‘tiny planet’ format and a binaural audio track.

Since, the source material is the same as the one from scenario A, we’ll list here only the steps that differ.

Scenario B steps:

• Process stereoscopic video from Gear 360 on Insta360 Studio to have the ‘tiny planet’ effect;
• Convert the raw 19-channel file from ZYLIA Studio to binaural, using ZYLIA Studio PRO running in REAPER.
• Edit 360-degree video and Ambisonics audio on Adobe Premiere.

Zylia: Tell us what is the story behind your project?
​
Yao: Last spring, I was 9 months away from graduating from Berklee College of Music, and the panic of post-graduation uncertainty was becoming unbearable. I was struggling to plan my career and I wanted to do something different. I spent a whole summer researching the ins and outs of spatial audio and decided to do my Senior Portfolio Project around my research. What I have found is that spatial audio is often found in VR games and films - recreating a 3D environment. It is rarely used as a tool for music composition and production. I saw my opportunity. 

With the help and hard work of my team (around 60 students involved), we succeeded in creating ‘Unraveled’, an immersive 360 audio and visual experience, where the audience would find themselves at the center of all elements, being surrounded by choir, strings, synths and imagery. My role was the project leader, composer, and executive producer. I found a most talented team of friends to work on this together: Gareth Wong and Deniz Turan as co-producers, Carlos Del Castillo as visual designer, Ben Knorr as music contractor, Paden Osburn as music contractor and conductor, Jeffrey Millonig as lead engineer and Sherry Li as lead vocalist and lyricist. Not to mention the wonderful musicians and choir members.​ I am truly grateful for their hard work, dedication and focus. 
​
‘Unraveled’ also officially kickstarts my company ICTUS, a company that provides music and sound design content specializing in spatial audio solutions. For immersive experiences such as VR, AR and MR, we are your one-stop audio shop for a soundscape that completes the reality. We provide music composition, sound design, 360 recording, mixing, mastering, post-production, spatialization and visualizing services tailored to your unique project. 
We are incredibly humbled that 'Unraveled' has been officially selected for the upcoming 44th Seattle International Film Festival, which runs May 17 to June 10, and to have been accepted for the Art and Technology Exhibition at the Boston Cyberarts Gallery, from Saturday May 26 to Sunday July 1.
​

Zylia: Recording so many people at once must have been challenging. How did you organize this?
Yao: I worked very closely with Paden Osburn, the conductor and music contractor, to schedule, revise, coordinate and plan the session. Paden is a dear to work with, basically allowing me to focus on the music while she coordinated with the rest of the amazing choir members. We had developed a great workflow.

I also had many meetings with the team of engineers as well as many professors to figure out the simplest, most efficient way to record. It was indeed very challenging and stressful to pull off, but it was also one the most magical night of my life. ​

Zylia: Tell us more about the technical part of this project.
Yao: On October 27, 2017, we had a recording session of the choir parts with 40 students from Berklee College of Music. The recording was done using three ambisonic microphones (Zylia, Ambeo, TetraMic). We tried forging a 320 piece choir by asking the 40 students to shift their positions around the microphones for every overdub. We also recorded 12 close mic-ed singers to have some freedom spatializing individual mono sources.

The spatialization was achieved through Facebook360 Spatial Workstation in REAPER. Many sound design elements were created in Ableton and REAPER. The visuals were done in Unity. We basically created a VR game and recorded a 360 video of the performance. Carlos Del Castillo did an outstanding job creating an abstract world that had many moments syncing with the musical cues.
​

​We are happy to announce a new version of ZYLIA Ambisonics Converter. We introduced a few changes based on your input and suggestions.

​We added a batch processing. Now, it is possible to process multiple 19 channels wave files within a single session.

There are also quality improvements and bug fixes for 2nd and 3rd order HOA. This update significantly increases the perceptual effect of rotation in HOA domain as well as corrects spatial resolution for 2nd and 3rd order. It is recommended to update to this new version.

By Jakub Zamojski & Lukasz Januszkiewicz

Recording and mixing surround sound becomes more and more popular. Among the popular multichannel representation of surround sound systems like 5.1, 7.1 or cinematic 22.2, especially worthy of note is an Ambisonics format, which is a full-sphere spatial audio technique allowing to get a real immersive experience of 3D sound. You can find more details about Ambisonics here (What is the Ambisonics format?).

Our previous blog post “2nd order Ambisonics Demo VR” described the process of combining audio and the corresponding 360 video into fine 360 movie on Facebook. Presented approach assumes using of 8-channel TBE signal from ZYLIA Ambisonics Converter and converts audio into the Ambisonics domain. As a result we get a nice 3D sound image which is rotating and adapting together with the virtual movement of our position. However, it is still not possible to adjust parameters (gain, EQ correction, etc.) or change the relative position of the individual sound sources present in the recorded sound scene.
In this tutorial we are going to introduce  another approach of using ZYLIA ZM-1 to create a 3D sound recording, which gives much more flexibility in sound source manipulation. It allows us not only to adjust the position of instruments in recorded 3D space around ZYLIA microphone, but also to control the gain or to apply any additional effects (EQ, Comp, etc.). In this way we are able to create a fancy spatial mix using only one microphone instead of several spot mics!

In the end of July 2017, using ZYLIA ZM-1 microphone we have recorded a band called “Trelotechnika”. All band members were located around ZM-1 microphone, 4 musicians and one additional sound source – drums (played from a loudspeaker). During the post-production process, we applied ZYLIA Studio PRO VST plug-in (within Reaper DAW) on recorded 19-channel audio track. This allowed us to separate the previously recorded instruments and transfer them into the individual tracks in the DAW. Those tracks were then directed to the FB360 plug-ins, where encoding to the Ambisonics domain was performed.

​4. Spatialiser track – receives 5 signals from tracks with separated instruments. Spatialiser allows to distribute sound sources in desired positions in the 3D space.
​      a) Click on FX and choose FB360 Spatialiser.

   d) Back to Spatialiser view. You will see an equirectangular picture and five circles with numbers. Each circle represents a sound source position in the space. By default, sources are located in the positions corresponding to the real positions of the instruments in the picture, but it is possible to adjust it by clicking on the circle and dragging it around the picture.

By Łukasz Januszkiewicz

• Microphone placement. ​​

​​Because of the size of a music scene, there were four mics used on the stage.
​The main ZM-1 microphone was positioned approximately 20 cm below the camera.
Backup ZM-1’s were positioned on the right and on the left side of the main ZM-1 (distance of 4-5 meters). There was also one mic just right in front of the choir. 
​ 

• The final setup 

​… and hiding behind the curtain :-)

​Captured video was stitched and processed with Kodak PIXPRO SP360 software. The recorded audio was manually synchronized with the processed video. Then we used our software to convert ZM-1 recorded tracks into 1st, 2nd and even 3rd order HOA (Higher Order Ambisonics) signals. Finally, we merged the video and the 1st order Ambisonics audio track into one MP4 file and uploaded it to YouTube 360 service*.
 
*At this moment YouTube 360 supports only 1st order Ambisonics audio.

We have recorded a great performance of orchestra and choir with a newest microphone array technology. Final effect is astonishing! Our 360 movie gives you a chance to participate in this beautiful event, and actually be in the middle of the stage.

​Video will be available soon.

How to turn a room into a recording studio?