General Audio Visualiser Implementation

General JUCE discussion
1

I’m relatively new to audio coding, and was looking for some clarification about how to correctly implement any kind of audio visualizer.

Currently, this is how I am doing it:

Step 1: Audio Output

I’m building a basic AudioAppComponent (not plugin, just stand-alone application).

I have a synthesized AudioSource that is being played by an AudioSourcePlayer.
The AudioSourcePlayer has been added to the AudioAppComponent's DeviceManager as an audio callback to output audio.

So basically, the synthesized sound is getting sent through the application’s main audio callback to the audio output.

Now the next step is to introduce the visualisers to the chain.

Step 2: Visualisers get Access to Audio Output Callback

I’ve created a class to encapsulate multiple audio visualiser objects into a single object called “Visualisers”. This class is an AudioIODeviceCallback, giving it the ability to access the same audio that is being sent to the speaker output.

An instance of the “Visualisers” object is added to my application. This “Visualisers” object is added to the AudioAppComponent's DeviceManager as an audio callback to receive any audio that is sent to the output.

Now the “Visualisers” have access to a callback of audio data. When this audio callback is called, the audio output data is saved into storage buffers for each individual visualiser object within the encapsulating “Visualisers” object.

The reason I created this Visualisers grouping class was because I thought it would be more efficient to have only one audio AudioIODeviceCallback for multiple visualiser objects to receive at once, instead of a AudioIODeviceCallback for each individual visualiser object. I do not know if this is actually more efficient.

Step 3: Individual Visualiser’s Process Audio Data & Update Display

This is where I feel the most unsure about what I am doing.
When thinking about visualiser implementations, my first thought is that I should NOT use any locks or JUCE CriticalSections. I watched an audio talk where @timur talked about how all audio programming should be lock-free.

So I think I need some kind of lock-free circular ring buffer that the audio callback should copy samples into, and then a GUI thread should read from.

This is where I feel the most confused. Should I be using a Timer to constantly update GUI data and call repaint()?
I’ve looked at some open source visualiser examples where someone used a TimeSliceThread to update data structures used by the GUI, and then used a Timer to call repaint() repeatedly.
Should I even be trying to mess with threads for visualisers?

If anyone can confirm or deny any of the steps I am taking, that would be very helpful.

Thanks!

4 Likes
2

Hey, I went through most of your steps. My visualizer is open source and you can check it out here:


In general, your functional / behavioural goals will determine your architechture and constraints. I strived (and still work on) doing it “the correct way” all the way through (for instance, completely lockfree as you said yourself), use no audio cpu as well as embrace the visualization and go all in on opengl and fast, smooth and steady update rates - this is not easily done using timers - YMMV.

It makes for a pretty complex system with 4 threads interleaving each other (gui, audio, opengl, worker thread), but in the end it’s the only way to realize my goals.
As for your last point; what I did was push samples in the audio callback to a wait free buffer fifo. On each push, it signals a semaphore. The semaphore blocks the worker thread until there is work to be done. The worker thread is responsible for pulling all the data in the wait free fifo and push it into a circular fifo, while calling registrered process() functions for all samples pushed through. This system ensures no cpu time is wasted, has minimal latency and there is complete control and chronological, monotonic history over all samples processed, with the added benefit of being able to lock this buffer while calculating/processing something with no problems.

If you only use one buffer, you either lock it (bad) or you will get screen tearing while visualizing it. You will also have problems with visualizations that require a any synchronization or chronology (filters).

Notice this is from an audio plugin perspective, things change if you dont have to make space for a lot of other things (ie., the main thread). Another thing to realize is that software rendering will only get you so far in terms of efficiency.

5 Likes
3

Wow, this is very helpful, thank you!

What you mentioned the method of using two different buffers (one for initial storage, the next for processing), that was very eye opening.
At the end, when you said “software rendering will only get you so far…”, are you referring to DSP hardware?

Do you have recommendations about how to learn more about these techniques?
Should I look for books on Digital Signal Processing?

Thanks again!

4

No problem. No, I was referring to rendering graphics on the CPU, ie. the standard way of doing this. Using hardware acceleration can offload some tasks like rendering huge amounts of lines to the GPU. Note that this can be achieved automatically to some degree by using the OpenGLContext class.

The specifics of building audio plugin infrastructures is not something I’ve found litterature on, and has been a trial and error process. If you’re interested in getting deeper in the DSP and mathemathics, there are loads of free litterature on the subject, for instance:
http://www.dspguide.com/pdfbook.htm
https://ccrma.stanford.edu/~jos/ (specifically “Introduction to digital filters”)
http://www.ece.rutgers.edu/~orfanidi/intro2sp/orfanidis-i2sp.pdf

These are general and cover fundamentals in detail.

2 Likes
Book recommendations for Sound Engineering, Math
5

Awesome, thank you so much for the help!
Your visualizers are ridiculously cool by the way. :grinning:

6

Hey, thanks a lot and good luck on your journey.