Efficient Volumetric Scene-based audio with ZYLIA 6 Degrees of Freedom solutionWhat is the difference between Object-based audio (OBA) and Volumetric Scene-based audio (VSBA)?OBA The most popular method of producing a soundtrack for games is known as Object-based audio. In this technique, the entire audio consists of individual sound assets with metadata describing their relationships and associations. Rendering these sound assets on the user's device means assembling these objects (sound + metadata) to create an overall user experience. The rendering of objects is flexible and responsive to the user, environmental, and platform-specific factors [ref.].  In practice, if an audio designer wants to create an ambient for an adventure in a jungle, he or she needs to use several individual sound objects, for example, the wind rustling through the trees, sounds of wild animals, the sound of a waterfall, the buzzing of mosquitoes, etc. The complexity associated with Object-based renderings increases with the number of sound objects. This means that the more individual objects there are (the more complex the audio scene is) the higher is the usage of the CPU (and hence power consumption) which can be problematic in the case of mobile devices or limitations of the bandwidth during data transmission. VSBA A complementary approach for games is Volumetric Scene-based audio, especially if the goal is to achieve natural behavior of the sound (reflections, diffraction). VSBA is a set of 3D sound technologies based on Higher-Order Ambisonics (HOA), a format for the modeling of 3D audio fields defined on the surface of a sphere. It allows for accurate capturing, efficient delivery, and compelling reproduction of 3D sound fields on any device (headphones, loudspeakers, etc.). VSBA and HOA are deeply interrelated; therefore, these two terms are often used interchangeably. Higher-Order Ambisonics is an ideal format for productions that involve large numbers of audio sources, typically held in many stems. While transmitting all these sources plus meta-information may be prohibitive as OBA, the Volumetric Scene-based approach limits the number of PCM (Pulse-Code Modulation) channels transmitted to the end-user as compact HOA signals [ref.].  ZYLIAs interpolation algorithm for 6DoF 3D audio Creating a sound ambience for an adventure in a jungle through Volumetric Scene-based audio, can be as simple as taking multiple HOA microphones to the natural environment that produces the desired soundscape and record an entire 360° audio-sphere around devices. The main advantage of this approach is that the complexity of the VSBA rendering will not increase with the number of objects. This is because the source signals are converted to a fixed number of HOA signals, uniquely dependent on the HOA order, and not on the number of objects present in the scene. This is in contrast with OBA, where rendering complexity increases as the number of objects increases. Note that Object-based audio scenes can profit from this advantage by converting them to HOA signals i.e., Volumetric Scene-based audio assets. To summarizing, the advantages of the Volumetric Scene-based audio approach affecting the CPU and power consumption are:
Zylia 6 Degrees of Freedom Navigable Audio One of the most innovative and efficient tools for producing Volumetric Scene-based audio is ZYLIA 6 Degrees of Freedom Navigable Audio solution. It is based on several Higher Order Ambisonics microphones which capture large sound-scenes in high resolution, and a set of software for recording, synchronizing signals, converting audio to B-Format, and rendering HOA files. The Renderer can be also used independently from the 6DoF hardware – to create navigable 3D assets for audio game design.  ZYLIA 6 DoF HOA Renderer is a MAX/MSP plugin available for MAC OS and Windows. It allows processing and rendering ZYLIA Navigable 3D Audio content. With this plugin users can playback the synchronized Ambisonics files, change the listener’s position, and interpolate multiple Ambisonics spheres. The plugin is also available for Wwise, allowing developers to use ZYLIA Navigable Audio technology in various game engines. Watch the comparison between Object-based audio and Volumetric Scene-based audio produced with Zylia 6 Degrees of Freedom Navigable Audio solution. Notice how the 6DoF approach reduces the CPU during sound rendering. Volumetric Scene-based audio and Higher Order Ambisonics can be used for many different purposes, not only for creating soundtracks for games. This format is very efficient when producing audio for:
#zylia #gameaudio #6dof #objectbased #scenebased #audio #volumetric #gamedevelopment #GameDevelopersConference #GDC2021 #GDC
0 Comments
In this tutorial we describe the process of converting 360 video and 3rd order Ambisonics to 2D video with binaural audio with linked rotation parameters. This allows us to prepare a standard 2D video while keeping the focus on the action from the video and audio perspective. It also allows us to control the video and audio rotation in real time using a single controller. Reaper DAW was used to create automated rotation of 360 audio and video. Audio recorded with ZYLIA ZM-1 microphone array. Below you will find our video and text tutorial which demonstrate the setup process. Thank you Red Bull Media House for providing us with the Ambisonics audio and 360 video for this project. Ambisonics audio and 360 video is Copyrighted by Red Bull Media House Chief Innovation Office and Projekt Spielberg, contact: cino (@) redbull.com Created by Zylia Inc. / sp. z o.o. https://www.zylia.co Requirements for this tutorial:
We will use Reaper as a DAW and video editor, as it supports video and multichannel audio from the ZM-1 microphone. Before recording the 360 video with the ZM-1 microphone make sure to have the front of the camera pointing the same direction as the front of the ZM-1 (red dot on the equator represents the front of the ZM-1 microphone) , this is to prevent future problems and to know in which direction to rotate the audio and video. Step 1 - Add your 360 video to a Reaper session. The video file format may be .mov .mp4 .avi or other. From our experience we recommend to work on a compressed version of the video and replace this media file later for rendering (step 14). To open the Video window click on View – VIDEO or press Control + Shift + V to show the video.  Step 2 - Add the multichannel track recorded with the ZM-1 and sync the Video with the ZM-1 Audio track. Import the 19 channel file from your ZM-1 and sync it with the video file.  Step 3 – Disable or lower the volume of the Audio track from the video file. Since we will not use the audio from the video track, we require to remove or put the volume from the audio track at minimum value. To do so, right click on the Video track – Item properties – move the volume slider to the minimum.   Step 4 – Merge video and audio on the same track. Select both the video and audio track and right click – Take – implode items across tracks into takes This will merge video and audio to the same track but as different takes.  Step 5 – Show both takes. To show both takes, click on Options – Show all takes in lanes (when room) or press Ctrl + L  Step 6 – Change the number of channels to 20. Click on the Route button and change the number of track channels from 2 to 20, this is required to utilize the 19 multichannel of the ZM-1.  Step 7 - Play both takes simultaneously. If we press play right now, it will only play the selected take, therefore we need to be able to play both takes simultaneously, therefore: Right click on the track – Item settings – Play all takes.  Step 8 – Change 360 video to standard video. Next we will need to convert the 360 video to equirectangular video to visualize and control the rotation of the camera. To do so, open the FX window on our main track and search for Video processor.  On the preset selection, choose Equirectangular/spherical 360 panner, this will flatten your 360 video allowing you to control the camera parameters such as field of view, yaw, pitch and roll.  Step 9 – As FX, add ZYLIA Ambisonics Converter plugin and IEM binaural Converter. On the FX window add as well:
  You should now have the binaural audio which you can test by changing the rotation and elevation parameters in ZYLIA Ambisonics Converter plugin. Step 10 – Link the rotation of both audio and video. The next steps will be dedicated to linking the Rotation of the ZYLIA Ambisonics Converter and the YAW parameter from the Video Processor. On the main track, click on the Track Envelopes/Automation button and enable the UI for the YAW (in Equirectangular/spherical 360 panner) and Rotation (in ZYLIA Ambisonics Converter plugin).  Step 11 – Control Video yaw with the ZYLIA Ambisonics Converter plugin. On the same window, on the YAW parameters click on Mod… (Parameter Modulation/Link for YAW) and check the box Link from MIDI or FX parameter. Select ZYLIA Ambisonics plugin: Rotation  Step 12 – Align the position of the audio and video using the Offset control. On the Parameter Modulation window you are able to fine-tune the rotation of the audio with the video. Here we changed the ZYLIA Ambisonics plugin Rotation Offset to -50 % to allow the front of the video match the front of the ZM-1 microphone.  Step 13 – Change the Envelope mode to Write. To record the automation of this rotation effect, right-click on the Rotation parameter and select Envelope to make the envelope visible.  After, on the Rotation Envelope Arm button (green button), right click and change the mode to write.  By pressing play you will record the automation of video and audio rotation in real time. Step 14 – Prepare for Rendering After writing the automation, change the envelope to Read mode instead of Write mode.  Disable the parameter modulation from the YAW control: Right click on Yaw and uncheck “Link from MIDI or FX parameter”  OPTIONAL: Replace your video file with the uncompressed version. If you have been working with a compressed video file, this is the time to replace it with the original media file. To do this, right click on the video track and select item properties.  Scroll to the next page and click Choose new file. Then select your original uncompressed video file.  Step 15 – Render! You should now have your project ready for Rendering. Click on File – Render and set Channels to Stereo. On the Output format choose your preferred Video format. We exported our clip in .mov file with video codec H.264 and 24bit PCM for the Audio Codec.  Thank you for reading and don’t hesitate to contact us with any feedback, questions or your results from following this guide.
by Pedro Firmino In continuation to our previous blog post “How to stream 3D audio in Binaural format with ZYLIA ZM-1 and ZYLIA Ambisonics Converter plugin” ... Here is how you are able to accomplish the same effect in a Windows system using a DAW with ReaStream and ZYLIA Ambisonics converter plugin. What you require: - ZYLIA ZM-1 microphone array - ZYLIA Ambisonics Converter plugin - BinauralDecoder plugin by IEM ( https://plugins.iem.at/ ) - ReaStream plugin (https://www.reaper.fm/reaplugs/ ) - DAW of your choice. - OBS (as a streaming application) Step 1: Receive input of the ZM1 into your DAW As the first step we need to receive the input from the 19 channels of the ZM-1 to a track in the DAW, therefore connect the ZM-1 microphone, open your DAW and select the ZM-1 as your Audio device.  Afterwards, create a track, change the number of channels from 2 to 20 and add the ZM-1 as a 20 channel input source.   By arming the track to record, you should now be receiving the 19 multichannel signal from the ZM-1 into your DAW. Step 2: Achieving Binaural sound using ZYLIA Ambisonics Converter plugin and IEM Binaural Decoder On the FX chain of the ZM-1 input track you will have to add the following plugins in this specific order:
 On the BinauralDecoder, you may choose to add some headphone equalization if you believe it’s necessary for the streamed audio.  For last, in the ReaStream plugin, remember to have your track armed and enable live monitoring. To send the output for your streaming application, enable “send audio/MIDI” in the ReaStream plugin and select “local broadcast” from the dropdown list.  Step 3: Receive the signal in OBS On the right side of OBS, open the Settings and click the Audio settings. In the Desktop Audio choose your output device. In this case Speakers. Confirm with OK. With the Desktop Audio added, click on the cog Icon and select Filters.  In the + Icon add a new VST 2x Plug-in: Choose ReaJs (it is included in the ReaPlugs vst pack). Click the Open plugin interface, click Load – Utility – Volume. Set the volume to the lowest.  On the Filters window, also add ReaStream-standalone plugin. Open the Plug-in Interface and select to receive. Make sure the identifier is the same as in the Reaper session.
You will now be using the ZM-1 as an input audio device and are ready to start streaming!
by Eduardo Patricio
Let’s talk about recording music!
This tutorial will address key points to help you improve your sound and get to a great demo with your ZYLIA MUSIC set. All you need is the ZM-1 mic, ZYLIA Studio application and your music.
Before going any further, the general tips / rules apply:
Ok, have said that, as usual, let’s break it down into steps: 
A - Rehearsal room / Setup
You can record in rooms with great or not-so-great acoustics.
Key things are: A1 - Place the ZM-1 mic in a central position in the room; A2 - Avoid standing near walls. This will help specially if your room doesn’t have great acoustics. 
B - Recording
C – Mix and export
Once you have your recording you can adjust the volume of each instrument to get a better balance. You can also apply equalization to enhance desired or hide unwanted characteristics of each instrument.
EQ example
If you record 2 guitars, say, with different functions (one focused on doing chords and the other more high-pitched patterns or solos), you can enhance the low end of the 1st and the high end of the 2nd one.
When you’re done with your mix, just export your demo and that’s it! 
To illustrate, here are 2 ZM-1 examples of ZM-1 recordings in completely different spaces, a martial arts practice room and in a church. Have a listen!
Recording done in a room with, err…, not-so-great acoustics
Recording done in a room with beautiful and style-matching acoustics
If you’d like to go further with mixing, you can either:
Here are some related useful resources for you:
Tutorial: Recording an acoustic guitar with the ZM-1 mic Tutorial: How to record classical music ZYLIA Studio PRO DAW session templates ZYLIA Studio PRO basics
Let’s have a look at an easy way to get a music track done, from recording to mastering. It’s quick tutorial time!
Alright, the main steps to take are: (A) recording, (B) mixing and (C) mastering.
Let’s breakdown main steps A and B in ZYLIA Studio (5 sub-steps):
Note
In step 4 (Mix), for each instrument, you can define:
Once you’re happy with you mix, it’s time to export the resulting stereo file: 
The 6th sub-step (the last one!) is submitting your track to be mastered by the folks at LANDR.com
by Eduardo Patricio In general VR related workflows can be complex and everyone seems to be looking for standard solutions. Here, we will show you, step by step, how to prepare a 360 video with spatial audio in, possibly, the shortest way! Required gear:
After following steps A, B and C, you’ll have a video file with 1st order Ambisonics spatial audio that can be played on your computer with compatible video players (e.g. VLC) or uploaded to YouTube.  OK, let’s have a close look at each step.
 Having said that, a small horizontal offset is not the end of the world
With the gear in place, start recording both audio and video and clap in between the mic and the camera. The clap sound spike can be used to sync the footage later.
Here’s a video showing all the sub-steps in Reaper: Note If you need to check how the recording sounds, add a binaural decoder plugin (e.g. IEM Binaural decoder) to the audio track, after ZYLIA Ambisonics Converter.
That’s it! Now you can enjoy the spatial audio
*Software tools used
Mounting:
https://www.amazon.com/Stage-MY550-Microphone-Extension-Attachment/dp/B0002ZO3LK/ref=sxbs_sxwds-stvp?keywords=microphone+clamp+arm&pd_rd_i=B0002ZO3LK&pd_rd_r=6860690f-2adc-4b00-a80e-de436939ed2b&pd_rd_w=GlE2J&pd_rd_wg=qrTTx&pf_rd_p=a6d018ad-f20b-46c9-8920-433972c7d9b7&pf_rd_r=GGS60M0DGQ5DQF44594V&qid=1575529629 https://www.amazon.com/Aluminum-Microphone-Swivel-Camera-Monitor/dp/B07Q2V6CBC/ref=sr_1_186?keywords=microphone+boom+clamp&qid=1575529342&sr=8-186 Amazon: https://www.amazon.com/Neewer-Adjustable-Articulating-Mirrorless-Camcorders/dp/B07SV6NVDS/ref=sr_1_205?keywords=microphone+clamp+arm&qid=1575531393&sr=8-205 Allegro generic alternative for us to test: https://allegro.pl/oferta/ramie-przegubowe-11-magic-arm-do-kamery-8505530470 by Eduardo Patricio Earlier this year (2019), during Easter, we recorded a string quintet concert* with 2 ZM-1 microphones. One in front of the ensemble and 1 behind it. Very simple and portable setup with powerful postproduction possibilities!
Here’s a quick example of the multi-track possibilities we have from the recordings: Equipment used
 How we did it Using ZM-1 microphones basically allows you to have multi-track recordings from single mics. Here’s what we did for this project in 5 steps. 1. We recorded the 19 channels from each ZM-1 directly onto DAW tracks on each laptop. The result was 2 uncompressed, 24-bit, 48KHz wave files.  Resulting multi-channel audio file being played back in Reaper. 2. After the recording, we loaded both multi-channel files onto a single DAW session, onto separate tracks. We synchronized the files manually using a clear transient sound (the classic "clap to sync").  Two 19-channel audio files on 2 DAW tracks. 3. We added ZYLIA Studio PRO plugin to the tracks containing the multi-channel files (one instance of the plugin for each). The plugin converts the recorded information into perfect sound spheres, from which you can extract sounds from any direction.  ZYLIA Studio PRO plugin interface before adding any virtual microphones. 4. We added virtual microphones, by clicking on the plus sign in the middle of the circle and defined their direction (in the horizontal and vertical planes) and polar patterns (width). Obs.: That means we basically set up a complex microphone arrangement after the recording, and one that can be tweaked or changed at any moment (!). NOTE: If you don’t know yet how to work with virtual microphones and ZYLIA Studio PRO, you can check the following additional resources: 4A. From the main microphone recordings, 7 virtual microphones were used as seen in the following screen capture:  ZYLIA Studio PRO with 7 virtual microphones in place. The virtual microphones Z1 – Z5 were narrow (shotgun-like) ones, pointing at each instrument and using S2 separation mode. Z6 and Z7 form a wide stereo pair, pointing 45˚ up, to capture some of the overall sound and a great deal of reverberant sound. 4B. From the rear microphone recordings, 5 virtual microphones were used as seen in the following screen capture:  ZYLIA Studio PRO with 5 virtual microphones in place. Detailed configuration of each virtual microphone: Z1: 0˚ azimuth, 33˚ elevation, 60˚ width; Z2 and Z5 form a stereo pair with a sum of 90˚ azimuth, pointing up (-45˚ elevation), and with 60˚ width; Z3 and Z4 form a stereo pair with a sum of 80˚ azimuth, pointing down (-51˚ elevation), and with 60˚ width. 5. We directed each virtual microphone to individual tracks in our DAW and carried on with a regular basic mixing strategy (leveling, panning and EQing).  Section of DAW showing the tracks receiving signals from virtual microphones Z1-Z5 (Main ZM-1 microphone). To sum up
by Pedro Firmino In this article, we will give you an alternative way of recording and coloring your guitar tone using the ZYLIA ZM-1 microphone by taking advantage of multiple channel recording.  Please, remember that these are suggestions based on our own experience and might not be suitable for your specific situation. While trying our methods, make sure to listen and adapt to your instrument, environment, and musician. When miking a guitar it’s important to understand that this instrument is not designed to be experienced at close distances making it more challenging to find a balanced positioning for recording, especially if you plan to use the ZM-1 to obtain several channels for posterior mixing. After all, it’s one of the ZM-1 strengths. Here we will guide you to make the most out of the ZM-1 recording abilities. Step 1 Position the ZM-1 between the neck and the sound-hole.
At 25 cm distance, the results were quite flexible to work with, however, if you see clipping happening on the VU meter of ZYLIA Studio, reduce the gain on ZYLIA Control Panel or move the ZM-1 further from the sound source.
Record your take and take some time to listen to the results. If you’re happy with the tone of the guitar you may proceed to mixing stage, if not here are some options for you: - increasing distance of the ZM-1 and the guitar for more room tone and less presence. - positioning ZM-1 closer to neck, giving you more treble and percussive sounds. - positioning the ZM-1 closer to the soundhole, provides more lower end and less treble. - using another channel of the ZM-1 recordings to add brighter tone to the mix. This last option shows one of the great advantages of recording with the ZM-1, which it will be mentioned next. Step 3 Separate the take and Mix your tone. Using ZYLIA Studio or ZYLIA Studio PRO, you are able to get different tones from your instrument with a single recording due to the multiple channels. It can also be useful if you want to add some reflections from the environment. After Separating the take, try using the Auto-mix feature. The tone resulting of this feature might surprise you, but if that’s not the case you are always able to mix the tone by controlling the different channel parameters in the MIXER window.  Are you looking for more low end in your tone? Simply adjust the level of the channel correspondent to the microphone closer to the sound-hole of the guitar (In this example Musician 2). Or perhaps you want more room tone and a wider stereo image of your recording? Increase the levels of the back microphones and apply panning (Musician 4 and 5).
|
Categories
All
Archives
July 2021
AuthorsZylia Team |
RSS Feed
|
|
Connect with usSubscribe to ZYLIA Newsletter
|
Our brands:
Get ZYLIA:
|
|
© Zylia Sp. z o.o., copyright 2018. ALL RIGHTS RESERVED.
|