In July 2021, Jordan Rudess (Dream Theater) and his friends Steve Horelick and Jerry Marotta gathered at the Dreamland Recording Studios, NY, for a spectacular and singular concert event, spanning classical to impressionistic piano genres to progressive, experimental rock with spacious synth soundscapes. The team at Zylia captured the performance with multiple 360° cameras and 3rd order Ambisonics microphones, allowing the creation of a unique, immersive 3D journey.         
 
Zylia’s 3D Audio recording solution gives artists a unique opportunity to connect intimately with the audience. Thanks to the Oculus VR application, the viewer can experience each uniquely developed element of the performance at home as vividly as it would be by a live audience member. Using Oculus VR goggles and headphones, viewers have the opportunity to:

• watch the concert from several different perspectives each with a 360° view,   
• listen to the music in 360° spatial audio at every vantage point,
• tilt and turn their head and witness how 3D audio follows their movement perfectly
• hear the unique acoustics of the venue,
  
• feel the fantastic atmosphere of the event.

To create a multi-point 360° audio-video experience, we set up a recording spots on the stage amongst the musicians – each equipped with a ZYLIA ZM-1S and a 360° camera. This allows a listener to choose from which point they watch the concert and experience it in immersive 360°. With every move, the sound projected to the listener’s ears changes, corresponding to the position of their head. To combine 360° video with 360° audio, we used a video editor (Adobe Premiere), a Digital Audio Workstation supporting 19-channel tracks (Reaper) as well as Unity3D and Wwise engines.

Recording was done using ZYLIA 6DoF Recording Application. Qoocam 8K cameras were used for video recording.

A RAW signal from microphones was synchronized using ZYLIA 6DoF Recording Application and converted to 3rd order Ambisonics format. Output Ambisonics signals were mixed in DearVR software.

The 360° videos from the Qoocam 8K were stitched and converted to an equirectangular video format.

For the purpose of distributing the concert, a VR application was created in Unity3D and Wwise engines for Oculus Quest VR goggles.

Multi-point 360° audio and video – the outcome

The outcome of this project is an Oculus application with the concert “Jordan Rudess & Friends in 3D Audio” – an immersive, deep, and emotionally engaging music experience.

For this workflow you will require:

• MacOS system (with connection to Ethernet + WiFi)
• Insta360 One R camera,
• Insta360 mobile app installed on your smartphone
• ZM-1 microphone
• OBS (https://obsproject.com/)
• ZYLIA Streaming application (https://www.zylia.co/zylia-streaming-application-download.html)
• MacOS RTMP Local host application (https://github.com/sallar/mac-local-rtmp-server)
• Facebook profile

This project consists of 3 main parts:
A. Receiving a live 360° video from the Insta360 to OBS
B. Receiving Ambisonics audio from the ZM-1microphone to OBS.
C. Configuring Facebook for live stream of 360° video with Ambisonics audio

​A. Receiving a live 360° video from the Insta360 to OBS
1. Turn On the Insta360 camera and connect your computer Wi-Fi to the corresponding Insta360 Wi-Fi network.

2. Simultaneously, connect your computer to your local ethernet using an ethernet cable. On your network preferences you should be connected with Ethernet and Wi-Fi.
On step number 4 you will need the IP address for the Insta360 Wi-Fi referred on your network preferences.

3. On your MacOS, open the application Local RTMP server (https://github.com/sallar/mac-local-rtmp-server).
This simple application will be used to receive a live stream from the Insta360 camera. It also provides you the local host address for OBS.

5. Open OBS and setup your project:

• On the video resolution you require a 2:1 format such as 1280x640
• Also, select your audio to 4.0 to receive 1st order Ambisonics audio later.

6. To receive the video stream from the Insta360 to OBS:

• On the Sources, click the plus sign and add “Media Source”
• Disable “Local File”
• On the input add the same address as the Insta360 camera, you may also use:
  • rtmp://localhost/live/abc

After adding the source, you should be receiving a 360° video from the Insta360 into OBS.

B Receiving Ambisonics audio from the ZM-1microphone to OBS.

7. Connect your ZM-1 microphone to your computer.
Open ZYLIA Streaming application.

8. In OBS, add source select ZYLIA Streaming application.

You are now receiving 1st Order Ambisonics audio (4 channels) and a 360° video into OBS.

​C Setup Facebook for livestreaming

​On the Settings – Stream, enable “Broadcast this as a spherical video” and also “This broadcast has Audio 360 (ambix4)”.

10. In OBS press “Start Streaming”. You are now live!

Some of you, who came across the term ‘3D sound’, probably are wondering what it actually is?
Intuitively, we may understand it by analogy to 3D space, which as we know is made up of three dimensions: width, depth, and height. In real life, any position in space is characterized by these dimensions.
​
Now imagine, that you are outside, for example in a park, full of various sounds. You can hear children laughing on the playground behind you; a dog is barking on your left, a couple of people are talking while sitting on a bench in front of you – you can hear their voices more and more clearly when you move towards them. We may say that you are in the 3-dimensional sound space. When you move, the sound you hear changes as well, corresponding to the position of your ears (its intensity, direction, height and even timbre).

Reproducing this natural human way of hearing in the recording isn’t easy, however, we already have the technology which allows us to do it – it is binaural and Ambisonics sound.

​The most common type of recordings – which you hear while listening to the music on your computer or watching TV –is stereo. You may also come across mono recordings but these have been outdated with the introduction of stereo mixing. Stereo sound basically allows you only to hear if the sound comes from left or right which is the main advantage over the mono-type of audio.
​
Bellow, you may find some great recordings that will allow you to hear the difference between these different formats. We have also summarized their characteristics in the bullet points, to present the information most clearly.

by Pedro Firmino

In this adaptation, we will show you how to create a 360 video with 3rd Order Ambisonics audio using:

• MacOS
• ZYLIA ZM-1 microphone
• ZYLIA Ambisonics Converter plugin
• IEM Binaural Decoder
• Reaper
• Modifed version of Google Spatial Media Metadata injector created by professor Angelo Farina.

​

This tutorial consists in 2 parts:
A: Preparing the 360 content with 16 channels
B: Injecting metadata using Spatial Media Injector version, modified by Angelo Farina.

At the moment, only HOAST library ( https://hoast.iem.at/ ) is the only platform which allows online video playback of 3rd Order Ambisonics and therefore the content created from this tutorial is meant to be watched locally using VLC player.

For this tutorial, basic Python knowledge is advised.

For preparing a 360 video with 1st order Ambisonics, visit the link:
https://www.zylia.co/blog/how-to-prepare-a-360-video-with-spatial-audio

1. As usual, start by recording your 360 video with the ZYLIA ZM-1 microphone and remember to have the front of the ZM-1 aligned with the front of the 360 camera.

2. After recording, import the 360 video and the 19 Multichannel audio file into Reaper.
Syncronize the audio and video.

​3. On the ZM-1 audio track, insert ZYLIA Ambisonics Converter and select 3rd Order Ambisonics. This will decode your 19 multichannel track into 16 channels (3rd Order Ambisonics).

​4. On the Master track, click on the Route button, On the track channels, select 16. Now you are receiving the signal from the 16 channels from the audio track.

​5. Once the video is ready for exporting, click File – Render.

As for the settings:
Sample rate: 48000
Channels: 16 (click on the space and manually type 16)
Output format: Video (ffmpeg/libav encoder)
Size: 3840 x 1920 (or Get width/height/framerate from current video item
Format: QT/Mov/MP4
Video Codec: H.264
Audio Codec: 24 bit PCM

Render the video.

After having the 360 video with 16 channels, it is necessary to inject metadata for Spatial Audio.

In order to do this, Python is required. Python is preinstalled in macOS but
you can download Python 2.7 version here: https://www.python.org/download/releases/2.7/

Afterward, download Angelo Farina’s modified version of Spatial Media Metadata Injector, located at:
http://www.angelofarina.it/Ambix+HL.htm

​The next part:

1. With the downloaded file located in your Desktop, run macOS Terminal application.
2. Using “cd” command, go to folder where you have Spatial Media Injector (eg. “cd ~/Desktop/spatial-media-2/”)

3. Run Python script “sudo python setup.py install”. Type your password.

​After the build is complete, type command: “cd build/lib/spatialmedia”

6. Enter python gui.py and the application should run.

​With the Spatial Media Metadata Injector opened, simply open the created 360 video file, and check the boxes for the 360 format and spatial audio. Inject metadata and your video will be ready for playback using 3rd Order Ambisonics audio.

In this tutorial we describe the process of converting 360 video and 3rd order Ambisonics to 2D video with binaural audio with linked rotation parameters.
This allows us to prepare a standard 2D video while keeping the focus on the action from the video and audio perspective.
It also allows us to control the video and audio rotation in real time using a single controller.

Reaper DAW was used to create automated rotation of 360 audio and video.
Audio recorded with ZYLIA ZM-1 microphone array.

Below you will find our video and text tutorial which demonstrate the setup process.
Thank you Red Bull Media House for providing us with the Ambisonics audio and 360 video for this project.

Ambisonics audio and 360 video is Copyrighted by Red Bull Media House Chief Innovation Office and Projekt Spielberg, contact: cino (@) redbull.com
Created by Zylia Inc. / sp. z o.o. https://www.zylia.co

Requirements for this tutorial:

•  Video and audio recorded with 360 video camera and ZM-1 microphone
•  ZYLIA Ambisonics Converter plugin
•  IEM Binaural Decoder
•  Reaper

We will use Reaper as a DAW and video editor, as it supports video and multichannel audio from the ZM-1 microphone.

Before recording the 360 video with the ZM-1 microphone make sure to have the front of the camera pointing the same direction as the front of the ZM-1 (red dot on the equator represents the front of the ZM-1 microphone) , this is to prevent future problems and to know in which direction to rotate the audio and video.

​Step 1 - Add your 360 video to a Reaper session. 

The video file format may be .mov .mp4 .avi or other.
From our experience we recommend to work on a compressed version of the video and replace this media file later for rendering (step 14).

To open the Video window click on View – VIDEO  or press Control + Shift + V to show the video.

Step 2 - Add the multichannel track recorded with the ZM-1 and sync the Video with the ZM-1 Audio track.

Import the 19 channel file from your ZM-1 and sync it with the video file.

​Step 3 – Disable or lower the volume of the Audio track from the video file.

Since we will not use the audio from the video track, we require to remove or put the volume from the audio track at minimum value. 
To do so, right click on the Video track – Item properties – move the volume slider to the minimum.

​Step 4 – Merge video and audio on the same track.

Select both the video and audio track and right click – Take – implode items across tracks into takes
This will merge video and audio to the same track but as different takes.

​Step 5 – Show both takes.
​
To show both takes, click on Options – Show all takes in lanes (when room) or press Ctrl + L

Step 6 – Change the number of channels to 20.

Click on the Route button and change the number of track channels from 2 to 20, this is required to utilize the 19 multichannel of the ZM-1.

​Step 7 - Play both takes simultaneously.

If we press play right now, it will only play the selected take, therefore we need to be able to play both takes simultaneously, therefore:

Right click on the track – Item settings – Play all takes.

​On the preset selection, choose Equirectangular/spherical 360 panner, this will flatten your 360 video allowing you to control the camera parameters such as field of view, yaw, pitch and roll.

Step 9 – As FX, add ZYLIA Ambisonics Converter plugin and IEM binaural Converter.

 On the FX window add as well:

• ZYLIA Ambisonics Converter plugin and set it to 3rd Ambisonics Order. Make sure to set the microphone orientation how you recorded it in the first place.
• IEM Binaural Decoder. Here you can choose headphone equalization at your liking.

You should now have the binaural audio which you can test by changing the rotation and elevation parameters in ZYLIA Ambisonics Converter plugin.

​Step 10 – Link the rotation of both audio and video.

The next steps will be dedicated to linking the Rotation of the ZYLIA Ambisonics Converter and the YAW parameter from the Video Processor.

On the main track, click on the Track Envelopes/Automation button and enable the UI for the YAW (in Equirectangular/spherical 360 panner) and Rotation (in ZYLIA Ambisonics Converter plugin).

​Step 11 – Control Video yaw with the ZYLIA Ambisonics Converter plugin.

On the same window, on the YAW parameters click on Mod…  (Parameter Modulation/Link for YAW) and check the box Link from MIDI or FX parameter.
Select ZYLIA Ambisonics plugin: Rotation

Step 12 – Align the position of the audio and video using the Offset control.

On the Parameter Modulation window you are able to fine-tune the rotation of the audio with the video.
Here we changed the ZYLIA Ambisonics plugin Rotation Offset to -50 % to allow the front of the video match the front of the ZM-1 microphone.

Step 13 – Change the Envelope mode to Write.

To record the automation of this rotation effect, right-click on the Rotation parameter and select Envelope to make the envelope visible.

​After, on the Rotation Envelope Arm button (green button), right click and change the mode to write.

​By pressing play you will record the automation of video and audio rotation in real time.

Step 14 – Prepare for Rendering

After writing the automation, change the envelope to Read mode instead of Write mode.

​Disable the parameter modulation from the YAW control:
Right click on Yaw and uncheck “Link from MIDI or FX parameter”

​OPTIONAL: Replace your video file with the uncompressed version.

If you have been working with a compressed video file, this is the time to replace it with the original media file. To do this, right click on the video track and select item properties.

Scroll to the next page and click Choose new file.
Then select your original uncompressed video file.

​Step 15 – Render!

You should now have your project ready for Rendering.
Click on File – Render and set Channels to Stereo.
On the Output format choose your preferred Video format.
We exported our clip in .mov file with video codec H.264 and 24bit PCM for the Audio Codec.

by Eduardo Patricio

In general VR related workflows can be complex and everyone seems to be looking for standard solutions. Here, we will show you, step by step, how to prepare a 360 video with spatial audio in, possibly, the shortest way! 

Required gear:

• ZYLIA ZM-1 microphone array;
• A 360 camera (e.g. Insta 360 One X);
• A computer.

​After following steps A, B and C, you’ll have a video file with 1st order Ambisonics spatial audio that can be played on your computer with compatible video players (e.g. VLC) or uploaded to YouTube.

​Having said that, a small horizontal offset is not the end of the world 

With the gear in place, start recording both audio and video and clap in between the mic and the camera. The clap sound spike can be used to sync the footage later.

• Bring your files (the 19-channel audio from the ZM-1 and the video one from your camera) onto the DAW;
• Lower the video volume;
• Align the clap position;
• Trim the ends;
• Adjust the volume of the audio file;
• Add ZYLIA Ambisonics Converter Plugin to the audio track and adjust the settings;
• Change the master output to 4 channels;
• Render!

Here’s a video showing all the sub-steps in Reaper:

​Note
If you need to check how the recording sounds, add a binaural decoder plugin (e.g. IEM Binaural decoder) to the audio track, after ZYLIA Ambisonics Converter.
 

That’s it!
Now you can enjoy the spatial audio

• By playing it on your computer, using VLC player, for example (very few players will handle 360 video + Ambisonics correctly); or…
• By uploading it to YouTube. 

*Software tools used
​

Mounting:
https://www.amazon.com/Stage-MY550-Microphone-Extension-Attachment/dp/B0002ZO3LK/ref=sxbs_sxwds-stvp?keywords=microphone+clamp+arm&pd_rd_i=B0002ZO3LK&pd_rd_r=6860690f-2adc-4b00-a80e-de436939ed2b&pd_rd_w=GlE2J&pd_rd_wg=qrTTx&pf_rd_p=a6d018ad-f20b-46c9-8920-433972c7d9b7&pf_rd_r=GGS60M0DGQ5DQF44594V&qid=1575529629
 
https://www.amazon.com/Aluminum-Microphone-Swivel-Camera-Monitor/dp/B07Q2V6CBC/ref=sr_1_186?keywords=microphone+boom+clamp&qid=1575529342&sr=8-186

Amazon:
https://www.amazon.com/Neewer-Adjustable-Articulating-Mirrorless-Camcorders/dp/B07SV6NVDS/ref=sr_1_205?keywords=microphone+clamp+arm&qid=1575531393&sr=8-205
Allegro generic alternative for us to test: https://allegro.pl/oferta/ramie-przegubowe-11-magic-arm-do-kamery-8505530470
 

We are happy to announce the new release of ZYLIA Ambisonics Converter plugin v1.4.0. 

​What would happen if on a rainy and cloudy day, during a walk along a forest path, you could move into a completely different place thousands of kilometers away from you? Putting the goggles on would get you into a virtual reality world, you would find yourself on a sunny island in the Pacific Ocean, you would be on the beach, admiring the scenery and walking among the palm trees listening to the sound of waves and colorful parrots screeching over your head.
​
It sounds unrealistic, but such goals are determined by the latest trends in the development of Augmented / Virtual Reality technology (AR / VR). Technology and content for full VR or 6DoF (6 Degrees-of-Freedom) rendered in real time will give the user the opportunity to interact and navigate through virtual worlds. To experience the feeling of "full immersion" in the virtual world, realistic sound must also follow a high-level image. Therefore, only each individual sound source present in virtual audio landscape provided to the user as a single object signal can reliably reflect both the environment and the way the user interacts with it.

There are many possibilities of using a 6DoF VR technology. You can imagine exploring a movie plan in your own pace. You could stroll between the actors, look at the action from different sides, listen to any conversations and paying attention to what is interesting only for you. Such technology would provide really unique experiences.

​A wide spectrum of virtual reality applications drives the development of technology in the audio-visual industry. Until now, image-related technologies have been developing much faster, leaving the sound far behind. We have made the first attempts to show that 6DoF for sound is also achievable.

​It's extremely challenging to record high-quality sound from many sources present in the sound scene at the same time. We managed to do this using nine ZYLIA ZM-1 multi-track microphone arrays evenly spaced in the room.

In our experiment the sound field was captured using two different spatial arrangements of ZYLIA ZM-1 microphones placed within and around the recorded sound scenes. In the first arrangement, nine ZYLIA ZM-1 microphones were placed on a rectangular grid. Second configuration consisted of seven microphones placed on a grid composed of equilateral triangles.

​Microphone signals were captured using a personal computer running GNU/Linux operating system. Signals originating from individual ZM-1 arrays were recorded with the specially designed software. 

​We recorded a few takes of musical performance with instruments such as an Irish bouzouki (stringed instrument similar to the mandolin), a tabla (Indian drums), acoustic guitars and a cajon.

To present interesting possibilities of using audio recorded with multiple microphone arrays we have created a Unity project with 7 Ambisonics sources. In this simulated environment, you will find three sound sources (our musicians) represented by bonfires among whom you can move around. Experiencing fluent immersive audio becomes so natural that you can actually feel being inside of this scene.

​During exploration experiments, MPEG Standardization Committee has realized that there is a lack of test material for audio and video experiments into MPEG-I Immersive Video. Our recordings have been accepted as a reference material in MPEG Standardization Committee. Each of the experts involved in standardization works on MPEG-I can now use the Zylia recordings to develop and test audio technologies for VR/AR applications. 

By Eduardo Patricio

​In this post we present two videos in different formats, but edited from the same source material captured on the 20th of June 2018, at Barigui park (Curitiba, Brazil).

The audio was recorded with the ZYLIA ZM-1 3rd order Ambisonics spherical microphone array while the video was captured by a 360-degree camera (Gear 360).
 
Below, you can watch both videos and find some information on how to achieve the two different results, with focus on preparing the audio recorded with the ZM-1 microphone for each scenario.

Interactive, immersive video with full 3D sound 
​(media components: 360-degree video + Ambisonics audio) 

​Non-interactive video with fixed perspective 3D sound
(media components: Tiny planet” video + binaural audio)

The microphone and the camera were placed on a single camera stand with a small clamped extension arm (see picture below). Both devices were aligned vertically with a small horizontal offset. We made sure the microphone and the camera always had the same relative facing direction (front of the microphone aligned with the camera side where the recording button is found).

For scenario B, we used the video from Gear 360 in ‘tiny planet’ format and a binaural audio track.

Since, the source material is the same as the one from scenario A, we’ll list here only the steps that differ.

Scenario B steps:

• Process stereoscopic video from Gear 360 on Insta360 Studio to have the ‘tiny planet’ effect;
• Convert the raw 19-channel file from ZYLIA Studio to binaural, using ZYLIA Studio PRO running in REAPER.
• Edit 360-degree video and Ambisonics audio on Adobe Premiere.