Automated DSP Testing


#1

Hi JUCE community :slight_smile:

I’m working on some plugins & native apps that need to attain & retain a high standard of DSP processing. Thus I need to design a system that can load VST/AU/AAX plugins and C++ functions/classes on macOS/Windows and run a bunch of tests (which are defined by the math wizard / DSP guru in my company). This system will be integrated into my build pipeline so the tests can be run easily after a new build is kicked off. Some of the requirements I’m looking in such a system:
• Command line compatible (ie a GUI and human-clicking is not required)
• Spectral Feature processing/analyzing
• General DSP metering algorithms (calculating THD, phase response, frequency response, various amplitude/loudness measurements, etc.)
• Framework for organizing/running tests. Must be able to load/handle audio file formats
• Plotting/graphing capabilities for interfacing with humans when needed
• Something that 1 full-time developer can implement/handle

Right now I’m looking at using Boost.Python to write a Python module that would wrap JUCE’s AudioPluginInstance class & plugin loading code so that I could load all the different formats of our plugins and tweak their parameters and test different presets, etc. I’m thinking of using Python to write the tests because:
• Quick implementation
• Lots of great existing libraries: SciPy, MatPlotLib, NumPy, etc.
• Plotting capabilities a la MatPlotLib (useful for interfacing with humans, eg say a test fails and I want to report to our DSP guru)
• Great open-source community with lots of other modules available (maybe making use of some of the machine learning libs would be useful in the near future :open_mouth:)

I was wondering if anybody else has any stories/tips they could share if they’ve tried to build a similar testing system. Has anybody had success with existing audio plugin wrapper Python libs?

Does anybody know of some amazing open source C++ libraries that could offer similar functionality?

If anybody else is interested in the JUCE AudioPluginInstance wrapper for Python I mentioned above, I’d be very down to collaborate on an open source project.


#2

Your almost on track, but instead of Boost.Python, use pybind11, far better and up to date with C++ standards.
This is what I’m now using for Audio ToolKit which seems to be close to what you want to do.
Still, I would advise to wrap the components inside of your instances, not just the instances themselves. This would only be functional tests when you also require unit tests.

Testing DSP code with a high standard doesn’t require VST/AU/AAX testing, they are orthogonal. Especially given what you want to test (I’m testing the DSP pipelines built in Python even if they are also in C++ in my plugins).


#3

NIce! Thanks Matthieu! It looks like a much leaner alternative as well which is a big plus :slight_smile:

I’m planning on doing both small units tests to individual functions as well as testing the overall product that the end user interfaces with (ie testing the DSP modules connected together). So having elegant C++/Python interoperability will be really nice :slight_smile:


#4

I agree. I always try to design my plugins with my Python layers, it’s really easy to check results, display them, profile independently of the host…

And yes, leaner, you don’t need to precompile something. The inconvenience is when you have inheritance across Python modules, but still manageable.