Automated DSP Testing

Hi JUCE community :slight_smile:

I’m working on some plugins & native apps that need to attain & retain a high standard of DSP processing. Thus I need to design a system that can load VST/AU/AAX plugins and C++ functions/classes on macOS/Windows and run a bunch of tests (which are defined by the math wizard / DSP guru in my company). This system will be integrated into my build pipeline so the tests can be run easily after a new build is kicked off. Some of the requirements I’m looking in such a system:
• Command line compatible (ie a GUI and human-clicking is not required)
• Spectral Feature processing/analyzing
• General DSP metering algorithms (calculating THD, phase response, frequency response, various amplitude/loudness measurements, etc.)
• Framework for organizing/running tests. Must be able to load/handle audio file formats
• Plotting/graphing capabilities for interfacing with humans when needed
• Something that 1 full-time developer can implement/handle

Right now I’m looking at using Boost.Python to write a Python module that would wrap JUCE’s AudioPluginInstance class & plugin loading code so that I could load all the different formats of our plugins and tweak their parameters and test different presets, etc. I’m thinking of using Python to write the tests because:
• Quick implementation
• Lots of great existing libraries: SciPy, MatPlotLib, NumPy, etc.
• Plotting capabilities a la MatPlotLib (useful for interfacing with humans, eg say a test fails and I want to report to our DSP guru)
• Great open-source community with lots of other modules available (maybe making use of some of the machine learning libs would be useful in the near future :open_mouth:)

I was wondering if anybody else has any stories/tips they could share if they’ve tried to build a similar testing system. Has anybody had success with existing audio plugin wrapper Python libs?

Does anybody know of some amazing open source C++ libraries that could offer similar functionality?

If anybody else is interested in the JUCE AudioPluginInstance wrapper for Python I mentioned above, I’d be very down to collaborate on an open source project.

1 Like

Your almost on track, but instead of Boost.Python, use pybind11, far better and up to date with C++ standards.
This is what I’m now using for Audio ToolKit which seems to be close to what you want to do.
Still, I would advise to wrap the components inside of your instances, not just the instances themselves. This would only be functional tests when you also require unit tests.

Testing DSP code with a high standard doesn’t require VST/AU/AAX testing, they are orthogonal. Especially given what you want to test (I’m testing the DSP pipelines built in Python even if they are also in C++ in my plugins).

2 Likes

NIce! Thanks Matthieu! It looks like a much leaner alternative as well which is a big plus :slight_smile:

I’m planning on doing both small units tests to individual functions as well as testing the overall product that the end user interfaces with (ie testing the DSP modules connected together). So having elegant C++/Python interoperability will be really nice :slight_smile:

I agree. I always try to design my plugins with my Python layers, it’s really easy to check results, display them, profile independently of the host…

And yes, leaner, you don’t need to precompile something. The inconvenience is when you have inheritance across Python modules, but still manageable.

Hey :wave:

I’m also trying to automate my DSP testing with python/juce and I’m currently using pybind11, building my python module with cmake.
This works fine when using plain C++.
I’m running into issues though when I try to introduce juce into the process and I’m wondering weather some of you guys got it going in the meantime and could help me out?

I’m including juce with the following CMake lines:

target_include_directories (my_module PUBLIC
        "../../JuceLibraryCode"
        "../../../JUCE/modules"
        )

Which does compile my module in release mode.
Unfortunately, when I’m trying to include my module from a python script I get the following error:

__ZN4juce78this_will_fail_to_link_if_some_of_your_compile_units_are_built_in_release_modeC1Ev

Curiously, when I’m compiling in debug mode, I get:

__ZN4juce76this_will_fail_to_link_if_some_of_your_compile_units_are_built_in_debug_modeC1Ev

Heres my CMake.txt in debug mode:

cmake_minimum_required (VERSION 3.0.0)

set(CMAKE_VERBOSE_MAKEFILE ON)

project(my_module)
add_subdirectory(pybind11)
pybind11_add_module(my_module ../../Source/Main.cpp)

target_include_directories (my_module PUBLIC
        "../../JuceLibraryCode"
        "../../../JUCE/modules"
        )

target_compile_definitions(my_module PRIVATE
        "-D_NDEBUG=1"
        "-DNDEBUG=1"
        )

and my Main.cpp:

#include "../JuceLibraryCode/JuceHeader.h"
#include "../Builds/CLion/pybind11/include/pybind11/pybind11.h"


PYBIND11_MODULE(my_module, m) {
    m.def("add", [](int a, int b) {return a + b;});
}

int main(int argc, char* argv[])
{
    return 0;
}

Any help would be very much appreciated!

Best,
Simon

Hello :wave:

That’s very courageous of you to try to integrate JUCE into a CMake project! I have quite some experience with it, so I think I can help.

First of all, where is the JUCE code being compiled or linked into your CMake project? The errors you are getting look like linker errors, which means that there are no objects files or static libraries that contain the compiled JUCE code.

Then NDEBUG and _NDEBUG should be defined only when the configuration is not Debug. On the other hand, DEBUG and _DEBUG should be defined only in Debug configuration. You can achieve that using generator expressions:

target_compile_definitions(my_module PUBLIC
  $<$<CONFIG:Debug>:_DEBUG>
  $<$<CONFIG:Debug>:DEBUG>
  $<$<NOT:$<CONFIG:Debug>>:_NDEBUG>
  $<$<NOT:$<CONFIG:Debug>>:NDEBUG>
) 

I hope this helps. Feel free to ask me more questions (here or by direct message).

1 Like

Hey McMartin,

Thank you very much for replying this quickly! I’m happy to have reached you, as I’m quite new to CMake.
I’m answering here if that’s ok, maybe someone will profit from this conversation being public one day :slight_smile:

First of all, where is the JUCE code being compiled or linked into your CMake project?

Yes I should have documented that. I launched the Projucer once and let it generate the CMake.txt file for me using the CLion beta support.
In the settings of the projucer I specified to build a dynamic library and linked my Python installation.
At first CMake didn’t find a whole lot of Python Symbols which are included from pybind11. I could resolve this by adding the undefined dynamic_lookup flag to set (CMAKE_CXX_FLAGS).
After that Cmake could successfully build the dynamic library with the pybind11 include in my Main.cpp.
I realize now that I have to somehow link the my_module target to the compiled juce modules?
Honestly, I don’t know how to do that exactly.

thanks for the automatic debug/release expression, which is already very helpful!

Heres my project structure, with the compiled juce modules and Main file:

I ended up going back to a regular Xcode project and integrated pybind11, which was a pretty straightforward header-only library include and works great!
As it’s only a small project it wasn’t that important to bring CMake into the picture.

Thank you very much for your help!

cheers,
Simon