Zylia: Tell us what is the story behind your project?
Yao: Last spring, I was 9 months away from graduating from Berklee College of Music, and the panic of post-graduation uncertainty was becoming unbearable. I was struggling to plan my career and I wanted to do something different. I spent a whole summer researching the ins and outs of spatial audio and decided to do my Senior Portfolio Project around my research. What I have found is that spatial audio is often found in VR games and films - recreating a 3D environment. It is rarely used as a tool for music composition and production. I saw my opportunity.
With the help and hard work of my team (around 60 students involved), we succeeded in creating ‘Unraveled’, an immersive 360 audio and visual experience, where the audience would find themselves at the center of all elements, being surrounded by choir, strings, synths and imagery. My role was the project leader, composer, and executive producer. I found a most talented team of friends to work on this together: Gareth Wong and Deniz Turan as co-producers, Carlos Del Castillo as visual designer, Ben Knorr as music contractor, Paden Osburn as music contractor and conductor, Jeffrey Millonig as lead engineer and Sherry Li as lead vocalist and lyricist. Not to mention the wonderful musicians and choir members. I am truly grateful for their hard work, dedication and focus.
‘Unraveled’ also officially kickstarts my company ICTUS, a company that provides music and sound design content specializing in spatial audio solutions. For immersive experiences such as VR, AR and MR, we are your one-stop audio shop for a soundscape that completes the reality. We provide music composition, sound design, 360 recording, mixing, mastering, post-production, spatialization and visualizing services tailored to your unique project.
We are incredibly humbled that 'Unraveled' has been officially selected for the upcoming 44th Seattle International Film Festival, which runs May 17 to June 10, and to have been accepted for the Art and Technology Exhibition at the Boston Cyberarts Gallery, from Saturday May 26 to Sunday July 1.
"Get a pair of headphones. Cave in somewhere quiet. Alone. Empty your thoughts and… Allow yourself to dive into something new. A pristine place that will truly disconnect you from the daily frenzy of life."
‘Unraveled’ has been officially selected for the upcoming 44th Seattle International Film Festival, which runs May 17 - June 10, with more than 400 films from 80 countries, running 25 days, and with over 155,000 attendees!
Zylia: Recording so many people at once must have been challenging. How did you organize this?
Yao: I worked very closely with Paden Osburn, the conductor and music contractor, to schedule, revise, coordinate and plan the session. Paden is a dear to work with, basically allowing me to focus on the music while she coordinated with the rest of the amazing choir members. We had developed a great workflow.
I also had many meetings with the team of engineers as well as many professors to figure out the simplest, most efficient way to record. It was indeed very challenging and stressful to pull off, but it was also one the most magical night of my life.
Behind the scene, photo by @jamiexu0528.
Zylia: Tell us more about the technical part of this project.
Yao: On October 27, 2017, we had a recording session of the choir parts with 40 students from Berklee College of Music. The recording was done using three ambisonic microphones (Zylia, Ambeo, TetraMic). We tried forging a 320 piece choir by asking the 40 students to shift their positions around the microphones for every overdub. We also recorded 12 close mic-ed singers to have some freedom spatializing individual mono sources.
The spatialization was achieved through Facebook360 Spatial Workstation in REAPER. Many sound design elements were created in Ableton and REAPER. The visuals were done in Unity. We basically created a VR game and recorded a 360 video of the performance. Carlos Del Castillo did an outstanding job creating an abstract world that had many moments syncing with the musical cues.
Zylia: What are your plans for future? Any interesting projects on your mind?
Yao: My long-term goal would be to establish my company ICTUS as one of the leading experts in the field of spatial audio. We are currently working on an interactive VR music experience called ‘Flow’ with an ethnic ensemble, GAIA, and the visuals are influenced by Chinese water paintings. The organic nature of this project will be a nice contrast to ‘Unraveled’s futuristic space vibe.
Another segment of the company is focused on creating high quality, cinematic spatial audio for VR films and games. We are producing a 3D audio series featuring short horror/thriller stories with music, descriptive narration, dialogues, SFX and soundscapes. Empathy is truly at the heart of this project, some of our stories will have a humanitarian purpose and we will be associated with many organizations that are fighting to end domestic abuse, human trafficking, rape, abuse and other violent crimes. We hope to bring more awareness and traffic to these causes with our art. Spatial audio is incredibly powerful, it really allows you to be in the shoe of the victims and without the visuals, I swear your imagination will go crazy!
In this short interview our Brand Ambassador Gene Torres describes his experiences with ZYLIA microphone and provides insights on recording his gigs and live performances.
Zylia: Gene, this is our second interview. Last time we talk you were testing a prototype of ZYLIA mic (the first interview). Now you are the owner of your own ZYLIA ZM-1 mic. Do you like it?
Gene: Hello, nice to chat with you again!
ZM-1, as far as I’m concerned, is a game changer in terms of recording rehearsals & gigs. I do like it, the possibilities are endless.
Zylia: How do you use ZYLIA recording studio?
Gene: I mainly use it for gigs & rehearsals. Jorge Sylvester a sax player and composer wants me to record a solo bunch of tunes that he wants to release!
>>Jorge Sylvester web page
Matter of fact, he just did a performance based on Eric Dolphy’s music. If you scroll down the above link, you can hear some of the show.
Performance of Gene Torres, Jorge Sylvester and Nora McCarthy doing The Re-Imagining Eric Dolphy show.
Zylia: What is so interesting for you in this system? Does it change the way you record your music?
Gene: Yes, it has changed the way I record not just my music but all the gigs that I do!!
I think that very noticeable is the fact that recording set-up time was cut by at least 50 minutes, definitely less set-up time is required. Within 10 minutes I’m up and running. When I had my two 8-channel computer interfaces there was at least 16 input cables that had to be plugged in. Not any more!
Recorded live performance. Rhonda Denet and the Silver Fox Trio. Mixed Company.
Zylia: You play with many NY musicians, what is their opinion?
Gene: Once, just out of curiosity, I counted the number of groups I was going to work with, either 6 months back or 6 months forward to a specific date... I counted up to 30 groups/bands!
Their opinion is they freak out cause the ZM-1 has the either Red or Blue led ring around it that glows, so they asked what the heck is that! :-) Once I explain what the ZM-1 is and what it does, they usually ask a million questions!!
Gene Torres, Andy Friedberg and Mike Campenni
Zylia: We are very happy that you decided to be our Ambassador and share with other musicians your knowledge and experiences using ZYLIA.
Gene: Once I received the ZM-1 I felt that the world of NYC needs to know what Zylia came out with and how good it is!!
Zylia: Do you have any plans for future? Any interesting gigs are coming?
Gene: I am going to record a solo recording of Saxophonist Jorge Sylvester when we both can work our schedules out!
I’m trying to finally release a CD as a band-leader. One of the things I want to record is a bunch of tunes with minimal direction but strictly improve. So the CD will have 1/3 originals, 1/3 covers but arranged, and 1/3 improve tunes!
Be on the lookout for it!
To route signal from ZM-1 open Audio Connection Manager (Window → Audio Connections or Alt+P). As Sources pick Hardware tab and as Destinations Ardour Tracks tab and connect 19-channel track with ZM-1 (green dots).
Now you can arm your track and record.
Input monitoring is possible for macOS users – create an aggregate device and choose it as your audio device.
Audio Device setup
The first step to get ZYLIA Studio PRO plugin to work is audio device configuration.
Open REAPER Preferences and choose Audio → Device (1). Next, set up your audio system in Audio System section (2).
IMPORTANT: ZYLIA Studio PRO requires 48000 Hz sample rate. You can set it in section 3.
ASIO users can set it in their driver’s control panel
(if not use Request sample rate option).
Audio Device setup
Plug in ZM-1, open REAPER Preferences Audio → Devices (1).
If you are using Windows – choose ASIO as your Audio system and as ASIO driver pick ZYLIA ZM-1 (2). It’s important to enable Request sample rate option and set 48000 Hz.
Number of last enabled input must correspond with “Zylia Studio Pro” track’s number of channels.
MacOS users can pick both ZYLIA ZM-1 or Aggregate Device (more in Appendix) as their Audio System. Main advantage of second solution is possibility to listen to output of ZYLIA virtual microphones and raw 19-channel track (input monitoring).
In Audio MIDI Setup you have to choose 20-channels or more in Format section (4).
Number of ZYLIA ZM-1 channels must correspond with “Zylia Studio Pro” track’s number of channels.
ZYLIA Studio PRO Reaper Template
- ZYLIA_Studio_PRO.rpp file contains blank Reaper session prepared for six sources. We suggest to save this file as Reaper project template – you’ll be able to make multiple projects based on this configuration (File→ Project Templates → Save project as template).
This works for live recordings (directly from the microphone) and for pre-recorded sessions.
Updated Surround Presets and more Virtual Microphones!
Please, note that the presets are not available in the VST3 plugin version, but you can use the *.zspro files bundled with the installers.
Changes to sources management
One-button automixing in ZYLIA Studio
The automixing feature automatically adjusts gains and pans of each instrument in the mix to enhance its sound. It is designed for a typical band setup. We take into account each instrument’s type and analyze its sound. Then we take into account what instruments were used during the recording and adjust the mixer properties accordingly. Check it out for yourself by using the demo recordings!
Wavpack to WAV export
So go ahead and download your copy of ZYLIA Studio v1.6 today.
Our previous blog post “2nd order Ambisonics Demo VR” described the process of combining audio and the corresponding 360 video into fine 360 movie on Facebook. Presented approach assumes using of 8-channel TBE signal from ZYLIA Ambisonics Converter and converts audio into the Ambisonics domain. As a result we get a nice 3D sound image which is rotating and adapting together with the virtual movement of our position. However, it is still not possible to adjust parameters (gain, EQ correction, etc.) or change the relative position of the individual sound sources present in the recorded sound scene.
In this tutorial we are going to introduce another approach of using ZYLIA ZM-1 to create a 3D sound recording, which gives much more flexibility in sound source manipulation. It allows us not only to adjust the position of instruments in recorded 3D space around ZYLIA microphone, but also to control the gain or to apply any additional effects (EQ, Comp, etc.). In this way we are able to create a fancy spatial mix using only one microphone instead of several spot mics!
Spatial Encoding of Sound Sources – Tutorial
“Spatial Encoding of Sound Sources” - a step-by-step description
- Download and install REAPER software. The evaluation version is fully functional and perfect to run with our demo.
- Download and install Facebook 360 Spatial Workstation software. It is a free bundle of spatial audio VST plug‑ins. In the session we used version 3.1 beta1.
- Download and install ZYLIA Studio Pro (VST3 Mac, Windows). It’s possible to run the demo in trial mode. In the session we used VST3 plug-in.
- Download the REAPER session prepared by Zylia Team. It is already configured with our audio tracks and all required effects. Unzip it.
- Download 360 movie – two versions are available: high quality [3840 x 1920] and medium quality [1920 x 960]. High quality version sometimes tends to pause in FB360 Video Player on slower CPUs.
- Run REAPER and open the session (ZYLIA-Ambisonics-sources-enc.rpp).
After opening the session, you will see several tracks:
1. Very important! Please, ensure that Reaper is working with sample rate of 48 kHz.
2. ZS_pro track – contains 19-channel WAVE file recorded with ZYLIA ZM-1. Click on FX button located on the ZS_pro track. If everything is correct, you will see ZYLIA Studio PRO VST plug-in. By default, there will be 5 virtual microphones – each one already assigned to one of the instruments: bass, drums, guitar, synth and pad. By clicking on a specific virtual microphone, you can adjust azimuth, elevation, width, and separation mode. Master send in routing panel for track should be unchecked.
a) Click on FX and choose FB360 Spatialiser.
b) Click on Load button placed on the video grid. Choose Slave mode and load the provided video clip. You will see a message box “H264 is not a recommended codec” - click X.
c) Set the video format to Mono and the display mode to Desktop. In Connect to DAW you should be able to choose your computer’s name. If not, try to restart Reaper and repeat the steps. You are ready to click the Open button. Video box will appear.
5. Control track – receives a multichannel signal from Spatialiser. Control provides connection with the Video Player, rotates the audio scene and applies binauralization.
a) Click on FX and choose FB360 Control.
b) Ensure that Listener Roll, Listener Pitch and Listener Yaw are properly received from video – controls should be darkened. Open a video box and try to rotate the image - Pitch and Yaw sliders should follow the image to rotate.
c) JS: Master Limiter boosts the volume and protects from clipping/distortions.
d) Master send in routing panel for track should be checked.
7. A good practice is to play video from the beginning of file to keep the synchronization. In some cases, it is necessary to close the VideoClient + VideoPlayer and load 360 video again to recover the synchronization.
8. Now you are able to rotate video across the pitch and yaw axis. Your demo is ready to run.
One of ZYLIA Studio PRO requirements is support for multichannel input for the plugin, which not many DAWs offer. Fortunately, there are ways to combine best of both worlds, and use ZYLIA’s technology with a typical multitrack workflow designed for dedicated microphone recording.
Such connection can be achieved thanks to ReWire, which is a software protocol allowing applications to exchange control messages and audio data. ReWire works both on macOS and Windows and requires no prior installation or configuration. Find out more at Wikipedia or Propellerhead website.
- Make sure to run your main DAW in the first place. In case of this tutorial, we will use Ableton Live. It will act as a ReWire master (mixer).
- When your DAW is up and running go to its audio settings. Select a proper device, sample rate, and buffer size. Keep in mind that ZS PRO will work only with a sample rate of 48 kHz.
- Start REAPER, which will be a host to ZS PRO plugin and act as a ReWire slave (device). If it is configured correctly, you should see “ReWire Slave Audio” in the upper right corner of REAPER window.
If you don’t see this it means that REAPER is not running as a ReWire slave. To fix this, go to “Edit” menu and select “Preferences”. In the list on the left find “Plug-ins” section and select “ReWire/DX”. Enable “Check for ReWire mixer/device on startup” option and restart REAPER.
To route audio form virtual microphone out of REAPER, you need to select “hardware” output for a track. These are the very same channels that will be available in Ableton. Click on “Add new hardware output...” menu and select one of the outputs from the list. It is advised to start from “ReWire Output 3” (for Z1 virtual microphone) since first and second outputs are usually associated with a master track
5. Go back to Ableton and create a new track. You will need “In/Out Section” of the mixer visible – if it’s not, use round “IO” button just right to the master track to turn it on. In “Audio From” section select “REAPER” as an input device, then select “REAPER Output 3” as an input channel. When you press “play” in one of the applications the signal from REAPER (processed by ZYLIA Studio PRO) should appear in Ableton. Set “Monitor” option to “In” to hear the sound produced by REAPER. The signal from virtual microphone now may be recorded and processed in the same way as it would be from a normal microphone connected to your audio interface.