Zylia announces the first release of ZYLIA Ambisonics Converter plugin.
ZYLIA Ambisonics Converter Plugin is a VST/AU plugin which enables conversion to Ambisonics directly within a Digital Audio Workstation (DAW). Thanks to ZYLIA Ambisonics Converter Plugin the whole recording system becomes even more powerful enabling new features for 3D audio production and surround sound recording.
ZYLIA Ambisonics Converter Plugin transforms a single 19 channel ZM-1 [http://www.zylia.co/zylia-zm-1- microphone.html] recording into Higher Order Ambisonics file that is compatible with top class tools for VR and 360 video production. It enables you to prepare 3D audio recordings for playback on the ‘Facebook 360’ and ‘Youtube 360’ platforms.
• B-format conversion (1st, 2nd and 3rd order of HOA)
• FuMa or ACN channel ordering & SN3D or N3D normalization
• ambiX and TBE formats
• Support for YouTube360 and Facebook360
• Correction of microphone position in the post-production stage
• Available as VST, VST3 and AU plugins
• macOS and Windows versions
• Compatible with DAW that supports at least 19 channels per track
Available in ZYLIA PRO Set.
What would happen if on a rainy and cloudy day, during a walk along a forest path, you could move into a completely different place thousands of kilometers away from you? Putting the goggles on would get you into a virtual reality world, you would find yourself on a sunny island in the Pacific Ocean, you would be on the beach, admiring the scenery and walking among the palm trees listening to the sound of waves and colorful parrots screeching over your head.
It sounds unrealistic, but such goals are determined by the latest trends in the development of Augmented / Virtual Reality technology (AR / VR). Technology and content for full VR or 6DoF (6 Degrees-of-Freedom) rendered in real time will give the user the opportunity to interact and navigate through virtual worlds. To experience the feeling of "full immersion" in the virtual world, realistic sound must also follow a high-level image. Therefore, only each individual sound source present in virtual audio landscape provided to the user as a single object signal can reliably reflect both the environment and the way the user interacts with it.
What are Six Degrees of Freedom (6DOF)
"Six degrees of freedom" is a specific parameter count for the number of degrees of freedom an object has in three-dimensional space, such as the real world. It means that there are six parameters or ways that the object can move.
There are many possibilities of using a 6DoF VR technology. You can imagine exploring a movie plan in your own pace. You could stroll between the actors, look at the action from different sides, listen to any conversations and paying attention to what is interesting only for you. Such technology would provide really unique experiences.
A wide spectrum of virtual reality applications drives the development of technology in the audio-visual industry. Until now, image-related technologies have been developing much faster, leaving the sound far behind. We have made the first attempts to show that 6DoF for sound is also achievable.
How to record audio in 6DoF?
It's extremely challenging to record high-quality sound from many sources present in the sound scene at the same time. We managed to do this using nine ZYLIA ZM-1 multi-track microphone arrays evenly spaced in the room.
In our experiment the sound field was captured using two different spatial arrangements of ZYLIA ZM-1 microphones placed within and around the recorded sound scenes. In the first arrangement, nine ZYLIA ZM-1 microphones were placed on a rectangular grid. Second configuration consisted of seven microphones placed on a grid composed of equilateral triangles.
Fig. Setup of 9 and 7 ZYLIA ZM-1 microphone arrays
Microphone signals were captured using a personal computer running GNU/Linux operating system. Signals originating from individual ZM-1 arrays were recorded with the specially designed software.
We recorded a few takes of musical performance with instruments such as an Irish bouzouki (stringed instrument similar to the mandolin), a tabla (Indian drums), acoustic guitars and a cajon.
Unity and 3D audio
To present interesting possibilities of using audio recorded with multiple microphone arrays we have created a Unity project with 7 Ambisonics sources. In this simulated environment, you will find three sound sources (our musicians) represented by bonfires among whom you can move around. Experiencing fluent immersive audio becomes so natural that you can actually feel being inside of this scene.
MPEG Standardization Committee
For scenario A, we used the regular stitched video from the Gear 360 and a 1st order Ambisonics audio file.
Scenario A - Basic steps taken:
Here are the detailed steps taken for the conversion to Ambisonics:
* Standard currently (August 2018) used on YouTube.
Since, the source material is the same as the one from scenario A, we’ll list here only the steps that differ.
Scenario B steps:
- Process stereoscopic video from Gear 360 on Insta360 Studio to have the ‘tiny planet’ effect;
- Convert the raw 19-channel file from ZYLIA Studio to binaural, using ZYLIA Studio PRO running in REAPER.
- Edit 360-degree video and Ambisonics audio on Adobe Premiere.
The most significant change is support for a new model of ZYLIA ZM-1. We added tailored spatial filters for a 3E microphone model to ZYLIA Studio and ZYLIA Studio PRO.
Source localization function is a new feature which will help you to find sound sources in the space. Thanks to that you will add a new virtual microphone in the right 3D position immediately.
To enable this function, press on the "Source localization" button in the left-upper corner of ZYLIA Studio PRO window. The localization algorithm works in real time. During playback of your 19-channel recording, estimated positions of the sound sources will appear on the VST/AU radar screen (blue icons). To add a virtual microphone in this position, simply click on the sound source icon.
ZYLIA Control Panel now is bundled with ZYLIA ZM-1 driver for macOS operating systems. Thanks to this application you will be able to manipulate LED brightness of your ZYLIA ZM-1 device.
To dim LED of plugged ZYLIA ZM-1 microphone, open ZYLIA Control Panel application and by using the slider decrease brightness to the desired level (you can turn the LED off as well). After closing ZYLIA Control Panel you'll be ready to start your recording session. ZYLIA Control Panel saves the last used LED brightness level and restores it right after application start.
But how to achieve a great rehearsal recording without spending too much time on technical issues? First, you need to know is what happens with the audio signal that goes to the recording device. The terms 'conversion' and 'converter' are crucial here.
Conversion of the audio into a digital form
Depending on the sampling frequency set, these snapshots are taken at different time intervals. For example, at 44.1kHz the analog-to-digital converter takes one sample every 1 / 44,100th of a second. It mins that it takes 44,100 samples representing the voltage every one second and converts them into appropriate numbers which can be stored in the digital form.
The audio signal recorded in this form, in order to be reproduced, must go ‘way back’ – so it needs to be converted into the analog signal, which will again be a variable electrical voltage. This process takes place in the digital-to-analog converter.
Audio bit depth
As it was already mentioned, in the 44.1kHz system, the audio-to-digital converter takes 44,100 samples per second. So, in the 48kHz system it is going to be 48,000, and so on. Increasing this value allows capturing more information about the sound which equals higher sound frequencies. This simply means that high-frequency instrument components can be better reproduced.
On mastering process, it is significant to operate on full band samples where low and high-frequency components are well heard. It is easier to mask some instruments by others in the same acoustic scene. Taking care of high sample frequency, you can guarantee your listeners’ natural timbre and feelings.
High sample frequency and bit resolution recordings are strongly recommended in compression, equalization and many additional audio effects. Standard configurations are 16 bits depth and 44,1k Hz sample frequency called CD standard. More advanced audio processing may require higher parameters.
In audio mastering process, the quantization noise prevents engineer to operate on well-recorded sound frequencies. Low bit depth frequency components are much distorted than the high bit. When mastering operator mix rehearsals with noise, the final record contains strengthened noise, which may be onerous for the listener and difficult to accept in professional audio production.
- Multi-microphone setup – that is, each microphone is put in the same place where the target loudspeaker is placed. In that case, it is not easy to upmix or downmix the number of channels and usually it is necessary to use one microphone setup for one loudspeaker setup
- Dummy head – in that case an artificial head is used where a pair of microphones is placed in the artificial ears. This type of recordings can be directly reproduced with headphones.
There is also a third approach, recording of sound objects or folleys and use them in game engine mixers or 3D audio renderers.
MUSIC AND AUDIO PRODUCTION PROCES
Felix Andriessens, Sound designer, BERLIN
HOW IT WORKS? THE TECHNOLOGY BEHIND ZYLIA ZM-1
Instead of using multiple spot microphones you put ZYLIA in the middle of the audio scene. The recorded audio tracks are processed by the dedicated software which allows creating so-called Virtual Microphones. They provide an adaptive isolation of sound sources. As a result, you receive separated tracks of the recorded sound sources. The software allows for steering a Virtual Microphone at any direction in the 3D space. In post-processing, you can now easily manipulate with recording and select exact sound sources, ambient or room reflections. What is more, you can identify unwanted sounds and remove them from your recording. Virtual Microphones operate in two modes:
- Beamforming – it simulates different types of spatial microphone characteristics, starting from super cardioid, up to an omnidirectional pattern.
- Separation mode – additional post‑processing algorithms for simulating close microphone placement. The ambient sounds and background noises are reduced and the separation between the sound sources is even higher.
3D AUDIO IN A NEW WAY
Binaural recording is a well-known method of recording sound scene with two microphones, creating a 3D stereo sound sensation for the listener, he can actually hear the effect of being in the middle of the sound scene. This effect is often created using a technique known as "dummy head recording" but you can also prepare a binaural recording with ZYLIA microphone array. There are two options to achieve that effect with ZYLIA:
- set two Virtual Microphones to a binaural mode or
- use 2 microphone capsules that are on the opposite side of the ZM-1 – dummy head emulation
Typical audio mix used for film or game production contains lots of sound objects that are further placed in the 3D space by using dedicated spatialization plugin. However, recording of each object separately is time-consuming and prone to errors such as capturing of unwanted noises.
By using ZYLIA ZM-1 it is possible to record a full 360˚ audio scene and then in the post‑production extract sound objects – instruments, dialogs, etc. Extraction process provides:
- Raw audio extracted by a virtual microphone
- 3D object position and trajectory of moving sound source/object
Ambisonics audio (B-format) can be used directly in many audio and video players such as YouTube or Facebook 360˚. 360˚ players convert Ambisonics audio into binaural (two channel) signal. The higher the microphone order the better the spatial resolution (ability to extract and listen to point sources in the surrounding sound scene) and rotation effect during playback.
It is highly recommended to use the highest possible microphone array, e.g. 3rd order. The high spatial resolution provides a truly immersive experience where the listener is able to easily localize sound sources in the 3D space. On the other hand, using lower order microphones result in fuzzy sounds during binaural or loudspeaker playback.
360˚ audio comes usually with 360˚ videos. For that context, ZYLIA ZM-1 is often combined with a range of popular cameras e.g. Insta360.
There are situations where Ambisonics sound is not enough to fully represents 3D audio space. Usually, it is a scene where there is a lot of sound sources which lead to problems with sound localisation. There are at least two solutions to solve this problem:
- Spatial Encoding - Record sound with ZYLIA, extract dominant sound sources or directions and encode that sound objects into Ambisonics domain. You can use Facebook 360˚ encoder for this purpose. This approach gives a good control over spatial resolution. Here is an example of such mixing process with sample files -- Create your own spatial Ambisonics mix with ZYLIA!
- Amplification of selected directions – in that context Ambisonics sound (b-format) is mixed together with sound objects. That latter can be the output of Virtual Microphone obtained from the same recording, however, focused on particular sound or direction. In that context, it is possible to improve localisation of perceptually (or artistic) most important sounds.
Cinematic audio and music production often requires recording a surround sound e.g. 5.1, 22.2 (Dolby Atmos). In that context, it is possible to use ZYLIA ZM-1 and ZYLIA Studio PRO plugin. The latter consists of surround preset demonstrated in the below picture.
Basically, virtual microphones point in the position where the loudspeakers are placed during playback. Such preset can be used live or in the post-production. It is important to mention that ZYLIA 19-channel format or B-format allows for immediate upmix or downmix number of surround channels. You only need to change a preset at the same recording. Such sound production is not possible using traditionally positioned microphones, because every surround setup requires a new microphone setup and recording.
Both artists played and sang and moved around the camera in both directions. Sound was captured by ZYLIA in 19-channel format. In the post-production, a 5.1 preset was used in ZYLIA Studio PRO plugin. Synchronized audio and video was prepared to listen in the cinematic 5.1 environment.
If there will be a need in the future to upmix the number of channel, e.g. 22.2 then the only thing to do is select 22.2 preset in ZYLIA Studio PRO using the same recording.
Soon this trailer will be available in cinemas in UK.
Basia: Music is the only language in the universe that we can all speak and understand. Even when you don’t necessarily play an instrument or sing, you can feel it. Music makes people emotional, helps go through happy and sad times and that is why I feel like it is the most powerful weapon we have. It is very important to me because it brings hope that humans can unite and create no matter the differences.
When I was about a year old, my grandma started singing to me and playing me different recordings and albums. I would repeat those melodies after her. When she passed away, my mom and grandpa took over. Grandfather played me jazz in the car - lots of Ella Fitzgerald and Louis Armstrong; I still remember the cassettes he used! :-) My mom showed me Michael Jackson, Basia Trzetrzelewska, Kayah and classical music such as Bach and Chopin.
That lead to piano lessons at age of 5 and continues until day on :-)
Basia: I heard about ZYLIA from my mentor and friend – Mikolaj Stroinski. It was during the summer of 2017. He called me and said: Basia, you need to meet these incredible guys, They created a mic... and he just kept going and going :-)
Zylia: What do you think about ZYLIA ZM-1 mic after testing it and using for casual recording?
Basia: ZYLIA mic is absolutely revolutionary. It sounds great, looks fantastic so it captures attention, easy to use. I remember when I flew to LA to sing during the NAMM festival, me and my guitar player - Lucas did a few recording in the super loud convention center using ZYLIA ZM-1 mic. We were afraid that the outside noise would make the recording muddy and just not the best quality. Oh how wrong we were!! :-)
The sound was great for the conditions we found ourselves in.
Basia Galaj and Lucas Fonseca at NAMM Show 2018 in Anaheim.
Basia: I think and I really hope it will become a necessity for every musician or music passionate that would like to easily record themselves in a great quality. It has so much potential in many areas! It can only grow in the VR world and that is becoming a next step in the music industry. It makes life of bands much easier since it is using only one cable, not 19 for every mic capsule inside the mic.
Basia: I am the leader / organizer of the very first TED talk at Berklee that will be happening next semester – Fall 2018. I am hoping to grow my blog and YouTube channel; I have lots of new content coming up very soon! Over the summer I am going to be doing interviews with Polish World War II veterans, asking them about their experiences and memories and I am going to write music to their confessions and stories. Hopefully, I will be able to use ZYLIA mic to record the interviews :-)
Some extra thing that I do is a project I called – Throwback Thursday! Every Thursday I do a video featuring my friends, we play a song from our childhood or just something we have memories of. I put those videos on my YouTube channel and Facebook fan page :-)
Zylia: We definitely need to see them:-) Thank you for your time and good luck with new projects!
Currently Basia lives in Boston, MA and attends the prestigious Berklee College of Music where she is pursuing a degree in Contemporary Writing and Production with a minor in Creative Entrepreneurship. On June 19th 2017, Basia released her debut EP called "Day By Day".
She is currently performing around East Coast with her band, playing her original music.
This edition is a minor update to the previous release and it brings a lot of small changes and bug fixes.
The new icon of ZYLIA Studio
Auto-mixing by default
After you separate your recording into separate instrument tracks the auto-mixing feature will engage when you visit the mixer screen for the first time. Thanks to that you’ll immediately receive a balanced recording. You can always reset the adjustments using the “Reset adjustments” option in the mixer’s menu. You can also use the “Mix automatically” button to repeat the auto-mixing procedure.
Bug fixes and other improvements
We had a great pleasure to meet Yao Wang during our visit at Berklee College of Music. A few days ago Yao published her project 'Unraveled' - it is a phenomenal immersive 360 audio and visual experience. You as a listener find yourself at the center of all elements, you are surrounded by choir, strings, synths, and imagery. You can experience being in the middle of the music scene.
Get to know more about this project and read an interview with Yao Wang.
Art work by @cdelcastillo.art
Yao: Last spring, I was 9 months away from graduating from Berklee College of Music, and the panic of post-graduation uncertainty was becoming unbearable. I was struggling to plan my career and I wanted to do something different. I spent a whole summer researching the ins and outs of spatial audio and decided to do my Senior Portfolio Project around my research. What I have found is that spatial audio is often found in VR games and films - recreating a 3D environment. It is rarely used as a tool for music composition and production. I saw my opportunity.
With the help and hard work of my team (around 60 students involved), we succeeded in creating ‘Unraveled’, an immersive 360 audio and visual experience, where the audience would find themselves at the center of all elements, being surrounded by choir, strings, synths and imagery. My role was the project leader, composer, and executive producer. I found a most talented team of friends to work on this together: Gareth Wong and Deniz Turan as co-producers, Carlos Del Castillo as visual designer, Ben Knorr as music contractor, Paden Osburn as music contractor and conductor, Jeffrey Millonig as lead engineer and Sherry Li as lead vocalist and lyricist. Not to mention the wonderful musicians and choir members. I am truly grateful for their hard work, dedication and focus.
‘Unraveled’ also officially kickstarts my company ICTUS, a company that provides music and sound design content specializing in spatial audio solutions. For immersive experiences such as VR, AR and MR, we are your one-stop audio shop for a soundscape that completes the reality. We provide music composition, sound design, 360 recording, mixing, mastering, post-production, spatialization and visualizing services tailored to your unique project.
We are incredibly humbled that 'Unraveled' has been officially selected for the upcoming 44th Seattle International Film Festival, which runs May 17 to June 10, and to have been accepted for the Art and Technology Exhibition at the Boston Cyberarts Gallery, from Saturday May 26 to Sunday July 1.
‘Unraveled’ has been officially selected for the upcoming 44th Seattle International Film Festival, which runs May 17 - June 10, with more than 400 films from 80 countries, running 25 days, and with over 155,000 attendees!
Yao: I worked very closely with Paden Osburn, the conductor and music contractor, to schedule, revise, coordinate and plan the session. Paden is a dear to work with, basically allowing me to focus on the music while she coordinated with the rest of the amazing choir members. We had developed a great workflow.
I also had many meetings with the team of engineers as well as many professors to figure out the simplest, most efficient way to record. It was indeed very challenging and stressful to pull off, but it was also one the most magical night of my life.
Yao: On October 27, 2017, we had a recording session of the choir parts with 40 students from Berklee College of Music. The recording was done using three ambisonic microphones (Zylia, Ambeo, TetraMic). We tried forging a 320 piece choir by asking the 40 students to shift their positions around the microphones for every overdub. We also recorded 12 close mic-ed singers to have some freedom spatializing individual mono sources.
The spatialization was achieved through Facebook360 Spatial Workstation in REAPER. Many sound design elements were created in Ableton and REAPER. The visuals were done in Unity. We basically created a VR game and recorded a 360 video of the performance. Carlos Del Castillo did an outstanding job creating an abstract world that had many moments syncing with the musical cues.
Zylia: What do you think about using ZYLIA ZM-1 mic for recording an immersive 360 audio?
Yao: I clearly remember meeting Tomasz Zernicki on the Sunday prior to our choir session. The Zylia team came to Berklee and demonstrated the capabilities of their awesome microphone, and I thought I had nothing to lose, so I asked for a potential (and super last minute!) collaboration that has proven to be fruitful. This has also brought me great friendship with Edward C. Wersocki who operated the microphone at our session. Unfortunately, he couldn't stay for the whole session, so only partial lines were recorded with the ZYLIA. He also guided me with the A to B format conversion which was extremely easy and user-friendly. I loved the collaboration and will only keep pursuing and exploring more possibilities with spatial audio. Hopefully, this will be the first of many collaborations.
Behind the scene, ZYLIA ZM-1, photo by @jamiexu0528.
Yao: My long-term goal would be to establish my company ICTUS as one of the leading experts in the field of spatial audio. We are currently working on an interactive VR music experience called ‘Flow’ with an ethnic ensemble, GAIA, and the visuals are influenced by Chinese water paintings. The organic nature of this project will be a nice contrast to ‘Unraveled’s futuristic space vibe.
Another segment of the company is focused on creating high quality, cinematic spatial audio for VR films and games. We are producing a 3D audio series featuring short horror/thriller stories with music, descriptive narration, dialogues, SFX and soundscapes. Empathy is truly at the heart of this project, some of our stories will have a humanitarian purpose and we will be associated with many organizations that are fighting to end domestic abuse, human trafficking, rape, abuse and other violent crimes. We hope to bring more awareness and traffic to these causes with our art. Spatial audio is incredibly powerful, it really allows you to be in the shoe of the victims and without the visuals, I swear your imagination will go crazy!
Yao is a composer, sound designer, producer and artist. She recently graduated from Berklee College of Music with a Bachelor of Music Degree in Electronic Production & Design and Film Scoring. Passionate about immersive worlds and storytelling, Yao has made it her mission to pursue a career combining her love for music, sound and technology. With this mission in mind, she is now the CEO and founder of ICTUS, a company that provides spatial audio solutions for multimedia.