In July 2021, Jordan Rudess (Dream Theater) and his friends Steve Horelick and Jerry Marotta gathered at the Dreamland Recording Studios, NY, for a spectacular and singular concert event, spanning classical to impressionistic piano genres to progressive, experimental rock with spacious synth soundscapes. The team at Zylia captured the performance with multiple 360° cameras and 3rd order Ambisonics microphones, allowing the creation of a unique, immersive 3D journey.         
 
Zylia’s 3D Audio recording solution gives artists a unique opportunity to connect intimately with the audience. Thanks to the Oculus VR application, the viewer can experience each uniquely developed element of the performance at home as vividly as it would be by a live audience member. Using Oculus VR goggles and headphones, viewers have the opportunity to:

• watch the concert from several different perspectives each with a 360° view,   
• listen to the music in 360° spatial audio at every vantage point,
• tilt and turn their head and witness how 3D audio follows their movement perfectly
• hear the unique acoustics of the venue,
  
• feel the fantastic atmosphere of the event.

To create a multi-point 360° audio-video experience, we set up a recording spots on the stage amongst the musicians – each equipped with a ZYLIA ZM-1S and a 360° camera. This allows a listener to choose from which point they watch the concert and experience it in immersive 360°. With every move, the sound projected to the listener’s ears changes, corresponding to the position of their head. To combine 360° video with 360° audio, we used a video editor (Adobe Premiere), a Digital Audio Workstation supporting 19-channel tracks (Reaper) as well as Unity3D and Wwise engines.

Recording was done using ZYLIA 6DoF Recording Application. Qoocam 8K cameras were used for video recording.

A RAW signal from microphones was synchronized using ZYLIA 6DoF Recording Application and converted to 3rd order Ambisonics format. Output Ambisonics signals were mixed in DearVR software.

The 360° videos from the Qoocam 8K were stitched and converted to an equirectangular video format.

For the purpose of distributing the concert, a VR application was created in Unity3D and Wwise engines for Oculus Quest VR goggles.

Multi-point 360° audio and video – the outcome

The outcome of this project is an Oculus application with the concert “Jordan Rudess & Friends in 3D Audio” – an immersive, deep, and emotionally engaging music experience.

​We are happy to announce the new release of the ZYLIA Studio in version 2.1.1 for Windows and macOS. With this version, we introduce the Chinese language of the application.
​
China is a huge market full of great musicians and sound engineers. Especially for them, we have prepared the Chinese language version of our hit software. ZYLIA Studio is an application dedicated to working with ZM-1 3rd order Ambisonics microphone array which provides an easy workflow of one-mic multi-track recording, AUTOMIXING, and manual mixing of tracks to prepare a balanced recording.

​Additionally, few small bugs were fixed connected with the stability of the energy map.

Tiger Woods once said that no matter how good you get, you can always get better, and that's the exciting part. Here at Zylia, we couldn’t agree more, therefore, we are thrilled to present you a new, and more importantly, improved version of our ZYLIA 6DoF Recording Application.
 
Intuitiveness, efficiency, and a great look that’s how in a few words we can describe the ZYLIA 6DoF Recording Application, which we released just recently. Let's take a closer look at it together!
 
For those, a little less familiar with our products – ZYLIA 6DoF Recording Application is a part of the ZYLIA 6 Degrees of Freedom Navigable Audio solution, which is the most advanced technology on the market for recording and post-processing multipoint 3D audio. The solution is based on multiple Higher Order Ambisonics microphones which capture large sound-scenes in high spatial resolution and a set of software for recording, synchronizing signals, converting audio to B-Format, and rendering HOA files.
 
This new ZYLIA 6DoF Recording Application has replaced the command-line toolkit for recording and synchronization. Thanks to a clear and comfortable graphical user interface, working with the app is fast and easy. We’ve also added a bunch of new features, which significantly improved the user experience.   

ZYLIA 6DoF Recording Application was designed as a tool for capturing 3D audio content from multiple ZYLIA ZM-1S microphones (up to 3000 channels on a single computer). To make the entire process smoother, we’ve introduced a set of very useful options:
 

• manual or automatic microphones position configuration,
• import/export of microphones configurations,
• ZM-1S identification via LED lights,
• LED lights control (it is possible to turn on/off the LED on all or on part of the microphones),
• sample by sample synchronization of recorded material from any number of recordings (it gives the high sampling precision necessary for full VR / 6DoF content production).

​Although the command-line toolkit did its job very well, our drive to excellence and constantly paying attention to the opinions of our customers resulted in developing a new, better version of the Recording Application. During the optimization, not only we kept the old, proven features but also added new ones and took care of the visual aspect of the interface. Here are all the great improvements we introduced:
 

• Automatic synchronization

In the command-line version of the app a user before starting recording had to choose the destination folder where the files will be saved. This folder had to be also manually indicated on the stage of synchronization. In the new version of the ZYLIA 6DoF Recording Application, the synchronization is automatic and does not require a source folder to be specified manually.

​

• Clippings under control

In the new version of the 6DoF Recording App we have made the Volume Unit meters of each microphone visible on the scene. This allows to monitor the intensity of the sound on every microphone and control unwanted clippings.

​​

• Protection of the recording

It may happen that during the session, one of the microphones will disconnect from the net. In such a case, the recording automatically stops and saves the files to the specified directory. 

​

• Marking recording with notes

A very useful new feature that has been added to the Recording Application is a possibility of taking notes during the recording. All the comments on the given take together with their timestamps are being saved together with the audio. Such notes can be of great help during the post-production process.

​While working on the new ZYLIA 6DoF Recording Application, we have not forgotten to make the interface aesthetically pleasing and familiar to our users, granting great usability and a joyful experience. See for yourself: 

OBA
 
The most popular method of producing a soundtrack for games is known as Object-based audio. In this technique, the entire audio consists of individual sound assets with metadata describing their relationships and associations. Rendering these sound assets on the user's device means assembling these objects (sound + metadata) to create an overall user experience. The rendering of objects is flexible and responsive to the user, environmental, and platform-specific factors [ref.].​

​In practice, if an audio designer wants to create an ambient for an adventure in a jungle, he or she needs to use several individual sound objects, for example, the wind rustling through the trees, sounds of wild animals, the sound of a waterfall, the buzzing of mosquitoes, etc. The complexity associated with Object-based renderings increases with the number of sound objects. This means that the more individual objects there are (the more complex the audio scene is) the higher is the usage of the CPU (and hence power consumption) which can be problematic in the case of mobile devices or limitations of the bandwidth during data transmission.  

VSBA
 
A complementary approach for games is Volumetric Scene-based audio, especially if the goal is to achieve natural behavior of the sound (reflections, diffraction). VSBA is a set of 3D sound technologies based on Higher-Order Ambisonics (HOA), a format for the modeling of 3D audio fields defined on the surface of a sphere. It allows for accurate capturing, efficient delivery, and compelling reproduction of 3D sound fields on any device (headphones, loudspeakers, etc.). VSBA and HOA are deeply interrelated; therefore, these two terms are often used interchangeably. Higher-Order Ambisonics is an ideal format for productions that involve large numbers of audio sources, typically held in many stems. While transmitting all these sources plus meta-information may be prohibitive as OBA, the Volumetric Scene-based approach limits the number of PCM (Pulse-Code Modulation) channels transmitted to the end-user as compact HOA signals [ref.].

To summarizing, the advantages of the Volumetric Scene-based audio approach affecting the CPU and power consumption are:
 

• Rendering complexity independent of the number of objects in the scene, decreasing the CPU usage from 100 to 10 000 times in comparison to OBA

The source signals are converted to a fixed number of HOA signals, uniquely dependent on the HOA order, and not on the number of objects present in the scene (even if the audio scene is created from thousands of audio signals).

• Possibility of rendering HOA signals to any reproduction device

A single format of HOA signals (with the addition of a few audio objects, if needed) can be reproduced on any device including headphones, soundbars, mobile etc.

• Lower number of PCM channels sent to the end user

HOA signals can be “spatially compressed” while maintaining high levels of spatial accuracy.

• Personalized interaction with immersive audio

HOA can be reproduced through headphones via the HOA to Binaural rendering and paired to Head-mounted displays to allow real-time adaptation of the sound field to head rotations in VR productions. It also facilitates 360° video applications where the sound field rotation can typically be controlled by moving a finger on the touch screen of a mobile device.
​

• Lower production costs for loudspeaker layout

The production (mixing and mastering) is performed in a single HOA format which can be rendered to any channel-based audio format for any loudspeaker layout (e.g., 2.0, 5.1, Dolby Atmos, …).  There is no need to produce separate mixes for each target loudspeaker layout which reduces production costs and efforts.

One of the most innovative and efficient tools for producing Volumetric Scene-based audio is ZYLIA 6 Degrees of Freedom Navigable Audio solution. It is based on several Higher Order Ambisonics microphones which capture large sound-scenes in high resolution, and a set of software for recording, synchronizing signals, converting audio to B-Format, and rendering HOA files. The Renderer can be also used independently from the 6DoF hardware – to create navigable 3D assets for audio game design.

ZYLIA 6 DoF HOA Renderer is a MAX/MSP plugin available for MAC OS and Windows. It allows processing and rendering ZYLIA Navigable 3D Audio content. With this plugin users can playback the synchronized Ambisonics files, change the listener’s position, and interpolate multiple Ambisonics spheres. The plugin is also available for Wwise, allowing developers to use ZYLIA Navigable Audio technology in various game engines.

Volumetric Scene-based audio and Higher Order Ambisonics can be used for many different purposes, not only for creating soundtracks for games. This format is very efficient when producing audio for:
 

• Virtual, Augmented, and Mixed Realities
• Facebook 360, YouTube Spatial Audio, Google Resonance
• 360° videos

#zylia #gameaudio #6dof #objectbased #scenebased #audio #volumetric #gamedevelopment #GameDevelopersConference #GDC2021 #GDC

We are happy to announce the new release of the ZYLIA 6DoF Recording Application in version 1.0.0 for Linux and macOS. This application is a part of the ZYLIA 6DoF Navigable Audio solution (ZYLIA 6DoF VR/AR set). It replaces the command line toolkit for the recording and synchronization process. For your comfort, this application has a graphical user interface, so there is no need to use the command line anymore.  

This application offers all features of the ZYLIA 6DoF Recording Toolkit such as:​

• Recording of 6DoF content from an unlimited number of ZYLIA ZM-1S microphones. Or up to 3000 channels on a single computer.
• ZYLIA ZM-1S microphones identification via LED.
• Manual or automatic microphones position configuration.
• Import or export of microphone configuration.
• LED control on all connected ZYLIA ZM-1S microphones. It is possible to turn on/off the LED on all of part of the microphones.
• Sample by sample synchronization of recorded material from any number of recordings. It gives the high sampling precision necessary for full VR / 6DoF content production.

Additionally, there are added few new features.

• During recording, the VU meters of each microphone is shown on the recording screen.
• Additionally, during recording, you can take notes about the recorded take, which will be saved in a certain directory together with audio.
• Automatic management of the location of the raw and synchronized signal.
• Disconnecting the microphone during recording no longer causes a crash of the recording process.

For this workflow you will require:

• MacOS system (with connection to Ethernet + WiFi)
• Insta360 One R camera,
• Insta360 mobile app installed on your smartphone
• ZM-1 microphone
• OBS (https://obsproject.com/)
• ZYLIA Streaming application (https://www.zylia.co/zylia-streaming-application-download.html)
• MacOS RTMP Local host application (https://github.com/sallar/mac-local-rtmp-server)
• Facebook profile

This project consists of 3 main parts:
A. Receiving a live 360° video from the Insta360 to OBS
B. Receiving Ambisonics audio from the ZM-1microphone to OBS.
C. Configuring Facebook for live stream of 360° video with Ambisonics audio

​A. Receiving a live 360° video from the Insta360 to OBS
1. Turn On the Insta360 camera and connect your computer Wi-Fi to the corresponding Insta360 Wi-Fi network.

2. Simultaneously, connect your computer to your local ethernet using an ethernet cable. On your network preferences you should be connected with Ethernet and Wi-Fi.
On step number 4 you will need the IP address for the Insta360 Wi-Fi referred on your network preferences.

3. On your MacOS, open the application Local RTMP server (https://github.com/sallar/mac-local-rtmp-server).
This simple application will be used to receive a live stream from the Insta360 camera. It also provides you the local host address for OBS.

5. Open OBS and setup your project:

• On the video resolution you require a 2:1 format such as 1280x640
• Also, select your audio to 4.0 to receive 1st order Ambisonics audio later.

6. To receive the video stream from the Insta360 to OBS:

• On the Sources, click the plus sign and add “Media Source”
• Disable “Local File”
• On the input add the same address as the Insta360 camera, you may also use:
  • rtmp://localhost/live/abc

After adding the source, you should be receiving a 360° video from the Insta360 into OBS.

B Receiving Ambisonics audio from the ZM-1microphone to OBS.

7. Connect your ZM-1 microphone to your computer.
Open ZYLIA Streaming application.

8. In OBS, add source select ZYLIA Streaming application.

You are now receiving 1st Order Ambisonics audio (4 channels) and a 360° video into OBS.

​C Setup Facebook for livestreaming

​On the Settings – Stream, enable “Broadcast this as a spherical video” and also “This broadcast has Audio 360 (ambix4)”.

10. In OBS press “Start Streaming”. You are now live!