Unity realtime device audio / input device plugin

Hey everyone,
I’m looking to implement what may turn out to be quite tricky… But looks like JUCE is pretty on top of everything so I’m happy to try any possibilities.

I am developing a Unity application that processes live audio, from device input to output real-time. This is a piece of cake in JUCE but Unity does not seem to have any native audio input device system. I’m curious as to what everyone recommends as a solution.

First off, I found a third-party plugin capable of adding a device input to unity as a plugin. Looks good enough, but the lesson I’ve learned so far is to stick to JUCE for as much as possible. Plus, looks like configuration may be needed anyways to manage this plugin, as well as to connect it with further JUCE plugins, so I may end up writing a JUCE version of the above plugin.

Would it work to configure an audio plugin to take in/generate audio device input? I’m currently trying to add an AudioDeviceManager to a plugin, making it a “device plugin”. How would I go about with the callback functions, connecting it up to the internals of the plugin in order to output AudioIODevice data?

From what I can tell, the latter could allow for a purely JUCE audio backend, simply “managed” by Unity UI. Unlike using a third party library which I fear may introduce latency and other complications. This seems like the best solution to me but feel free to suggest otherwise!

Hi! Sorry not an answer to your question, I’m looking to make a quite similar application with strong accuracy audio input in Unity. Have you managed to find a good solution to your problem ? Is it possible to create a Juce plugin that manage the audio devices ?

What I can add to the mix is that I’ve tested making a standalone JUCE dynamic library and then I’ve used C# interop in Unity to invoke the library and pass stuff back and forth between the two worlds. Instead of using Unity’s audio engine, I let JUCE handle audio entirely and just used interop/pinvoke with a C# wrapper to the JUCE library.

I didn’t end up going super deep with it, but it definitely worked, and this should allow you to access audio input/output, etc. with much more possibilities and probably lower latency options than Unity’s internal audio engine will allow, and just keep interaction between to two high level.

If you want to actually pass audio buffers between JUCE and Unity’s audio engine things will probably get a bit more perilous. There should be ways to get the audio data out of JUCE into Unity, but probably only safely for things like visualization within Unity, I wouldn’t try to do any further processing or playback in Unity with any data buffers you pass from the unmanaged to managed world, but it sounds like this isn’t your use case.

If you don’t need an integration of audio buffers between JUCE and Unity’s audio engine, then I would definitely recommend going this route. However, you should know that if you haven’t done any C# managed to C++ interop/written any C# wrappers to dynamic C++ libraries, be prepared to invest a good bit of time reading up and tracking down information here, there is a lot to know, and a lot of potential pitfalls.

The other thing is, for me, I got this working fine on Windows, but for some reason on macOS my invokes to the dynamic lib were failing in Unity and I could not figure out why. It should be possible because I had other examples that showed as much, but specifically with a dynamic JUCE lib and Unity on macOS there was some final issue I just hadn’t nailed down yet. So be prepared to invest sometime in a happy JUCE/Unity cross platform integration.

Since Unity uses Mono, you’ll want to start here if you’re not already familiar with pinvoke:

1 Like

Exactly, I could have written this myself heh.

I have an audio engine working in Unity, tried a bunch of different ways but finally settled on the external library/pinvoke as well.

I did get the same engine working with Unity’s callback using pivoke as well but I had sample rate issues.

The third and final way was directly calling Unity’s API, had an example;

As yet another way to do it without JUCE and this hooks into the signal flow of Unity’s mixer.

Haven’t tried it with JUCE yet, it seemed like this functionality in JUCE didn’t gain traction? Originally 2018 POC stuff IIRC.