Anouncing "pluginval" an open-source cross-platform plugin validation tool


Added support for VST3 now:


I’d love one. Show me a spec and I’ll adhere to it. One of the biggest problems with audio plugins, both from the plugin and host side is that this is largely un-speced. Things have improved a bit in recent years with VST3 and AU validation tools but as far as I’m aware there is no concrete written spec about the order in which you can call things or on which threads.

All hosts are littered with hacks and work arounds for specific plugins that behave slightly out of the ordinary. Sometimes due to bad coding, but most often due to simply not having a reference for how something should behave.

Passing a validation is not the same as conforming to a spec. It simply means you pass the tests thrown at you. This may be fine if you’re the host that wrote the test i.e. you’ll only ever do the same things as the validator app (which is possible with Steinberg and the VST3 validator), but how does another host know exactly what it can and can’t do?

This tool isn’t designed to replace existing validation tools or specs, indeed it’s there to enforce them and provide a quick means of testing them or pushing the boundaries. My ultimate goal with this project is to create a set of tests (on level 10) so exhaustive and multi-threaded that it should be possible to write a near-bulletproof plugin.

Running this over your plugin from the start should help anyone catch design problems before you’ve launched and then a user tries your plugin in an obscure host you’ve not heard of before, let alone tested in…

As you mention though, this tool is also designed to be a “host testing” framework. Maybe not in the traditional sense but with this I can scan my entire plugin list (some several hundred) and run the validation tests on all of these plugins. This can be done periodically on a CI machine too.

With this data, we can very quickly see what plugins have issues where.
If all plugins fail a specific test, it’s probably a good idea to fix that behaviour in the host and remove the test for plugins.

Alternatively, if only a small amount of plugins fail a specific test, we can use these logs to inform the developers and maybe work around this small number in our hosts.

The main aim here is to speed up this data gathering process that’s usually only done out in the wild where it’s often too late to make changes.

I hope that helps explain things a little more and how I can see this being used!


I think the key thing is here that both host and plugin developers need to be contacting each other when there are issues to avoid these hacks where possible, all to often it’s easier to just add the hack and be done with it. That’s not a dig at you or anyone specifically but to all of us in the industry in general.


Yes, I completely agree. The main problem is that a lot of plugin developers are either too small to have the time to communicate with host devs and then fix things or they’re too big and making fixes to accommodate a single host is outside their dev cycle.

I’ve experienced both sides of the coin over many years spent writing a host and plugins. It’s a fantastic feeling when you help someone make their product better by reporting a bug that they are willing and able to fix, both plugin and host side. Unfortunately I’ve also wasted a lot of time registering with sites, downloading products, installing, scanning, authorising, creating sessions to test specific behaviour, getting crash traces, contacting developers, explaining in detail exactly why the software is crashing and why, all to hear back nothing or have no action taken on it. That’s disheartening.

One of the main aims of this tool is to make it less host/plugin specific and more general. Hopefully that way it can become a pseudo standard and basically help host and plugin devs spot problems early. This is one of the reasons it needs contributions from both sides of the land. We need to work together to make the whole audio software experience a good one for users.


I couldn’t agree more, it’s pretty much the same reason I was writing a plugin validator too. I haven’t taken a good look at this yet but one feature that would be useful (and more likely to be adopted by bigger players) is to have the ability to change the output data so for example it could output a JSON or XML file in a particular format that conforms to a unit test standard that can be parsed by CI tools.


Yes, it does and that’s intentional. Essentially this acts in two ways:

  1. For plugin developers to ensure compatibility with JUCE based hosts (and also as a general testing/validation tool, used in addition to the format-specific tools)
  2. For JUCE based host developers (possibly including JUCE itself :wink:) to test compatibility with plugins from all formats and built with all frameworks

It might be easiest if I run over some problem areas/workflows which we will be putting in to practice at Tracktion:

• For Host Developer QA Teams:

Testing plugin compatibility with your host can be an extremely tedious process. Simply getting hold of, updating, registering etc. plugins is extremely time consuming. At Tracktion, we have a contracted QA company (which I can highly recommend if anyone’s interested) so they have a huge database of plugins etc. already installed. I saw an opportunity here for them to periodically run pluginval on their installed plugins, across multiple formats on multiple OSes all essentially automated. They can then give us the reports and let us know if any plugins fail tests which likely mean there will be compatibility problems with Tracktion/Waveform.
We can then get them to do targeted testing with the actual DAW.

Originally, I had this plan to automate the DAW in this way but it seemed better to do it as an external tool and open source it so it can be used by everyone.

• For Host Developer CI:

Similarly to above, if you have a CI setup, you can add a job to periodically run pluginval on any installed plugins and find out compatibility problems early. This will most likely be used to spot regression testing.
I’ve tried very hard to make pluginval as simple to clone or download binaries for precisely for these kind of CI workflows.

The main advantage of having QA run tests like these is that they will already have to spend time updating plugins etc. The set on the build server is likely to be much reduced simply due to time constraints/expiring licences (iLok anyone?) etc. The larger the company, the more likely you are to have a dedicated person to manage this. Unfortunately at Tracktion we’re not quite there yet so we’ll use a combination.

• For Plugin Developer CI:

If you’re a plugin developer, you’ll likely know that testing with the dozens of hosts out there is astoundingly time consuming. pluginval aims to speed up that process by offering an automated way to check for silly mistakes, regressions and even things that are stricter than most hosts will require.

This should lead to a better overall level of software development and increased visibility in the space. Remember, with an open source tool we can add comments and log messages to explain to developers (particularly those new to the domain) why tests are failing and point them in the right direction.

I imagine fuzz testing here will come in extremely useful, particularly for things such as race conditions that can happen rarely and are very difficult to reproduce. Run some fuzzing for a few hours over your plugins and see what happens…

The other reason I like this CI-tooled approach is that developers (myself included) often respond better to machines telling them they’ve done something wrong. Getting a Jenkins email telling you you’ve broken the build due to some failing tests seems easier to take than a person telling you you’ve broken something. (Jenkins then nagging you every time you commit reinforces this message). You can then tidy it up, push it and no one else may even notice…

Of course pluginval should be used in conjunction with existing tools, particularly on automated CI as they don’t take any extra development time to run.

• For Plugin Developers:

Again similarly to above but having an open source project that you can actually run with a debugger attached to see where tests are failing is super useful.

The other side to this which I’ve not discussed yet is essentially some kind of “recognised” or “compatible with” acknowledgement status. At Tracktion, we love to promote plugin developers making cool software and particularly if you take the time to make sure your plugins work with our DAW. We want to add a compatibility table to our own website in order to do this (our user base can then look at this to hopefully find interesting new plugins) but we know testing with hosts is time consuming. If you can send us a passing pluginval report, we’re 95% sure your plugin will be compatible with us and we can add you to this list.

Who knows, in the future we might even add a GitHub badge!

• For Host End Users:

For users of our host, we wanted to provide a tool that they can run on problematic plugins to generate a log file which they can then send to us and plugin developers in order to give us a head start on fixing the problem.

We could even automate the collection of these so we can find the most popular, problematic plugins.

• For Plugin End Users:

Again, similarly to above but end users can use this tool to test their own installed plugins and generate reports to send to the plugin devs. This can save a lot of time and there are many users out there willing to do this providing the process is easy and quick enough.

I think that covers the main ways I see this being used but I’m sure there are many more which I’m excited to hear about!


Yep, I like this idea, I’ve just not really come across them yet.
If you’ve got any links to formats that CI tools use let me know and I’ll see what we can do!


I think JUnit is the most widely used


Thanks, that looks straightforward enough.
Maybe a goo approach would be to simply construct the XML format from the juce::UnitTestRunner TestResults?

You can get these after the test have run using: const TestResult* getResult (int index)
I think TestResult has everything needed to populate JUnit?


I suspect you’re probably right, when I get a moment I’ll take a look.


Hi Dave,

I’ve just came across this post. This looks awesome and thanks for making it GPL. In the ideal case this could evolve to a standard common denominator that both host and plugin developers can agree on. So when you’re plugin validates here, you can safely forward all tech support messages to the host that’s causing the problem :slight_smile:

Without digging too much in the internals I have a feature suggestion that would make it a killer tool for synth development. Basically you would load a plugin state, render a MIDI file to an audio file and compare this to earlier results. If the audio buffers don’t match (within reasonable tolerance), you know there’s something wrong. You can also check how it copes with different sample rates, buffer sizes etc.

In order to make this work, you would need objects with this structure:

struct PluginTestResult
    ValueTree testConditions; // contains information like samplerate, buffer size, etc.
    MemoryBlock pluginState; // contains the plugin's state as saved by the host
    MidiMessageSequence midiSequence; // contains arbitrary MIDI data that create sound
    AudioSampleBuffer renderedBuffer; // contains the rendered audio data

You would create a few of these objects for the 1.0.0 version (or whatever), save them to an external file, and then render and compare every of these cases whenever you want to run the test suite.

For audio effects you can omit the MidiMessageSequence, and use a input AudioSampleBuffer containing noise or other signals.

Is this something within the scope of this project, and if yes, where would you start implementing this? I’ve been toying with the idea of making a CI test app for synths for quite some time and would be happy to contribute.


Hi Christoph, sorry for the delayed reply, I’ve been thinking about this for the past day. This is indeed a workflow I would like to encourage so would be massively grateful if you would like to contribute.

I think the best approach of this would be to create some kind of XML format that a test can understand and then link it to a test by an ID. (Adding IDs to tests is something I’ve been meaning to add). You could then pass a path to the test runner which could then examine the file for any information it needs to run that test.

In your example, say you had a:

struct PluginRenderTest : public PluginTest
    int64 getID() override { return 0x42; }
    void setTestData (const ValueTree& dataToUse) override;
    void runTest (UnitTest& ut, AudioPluginInstance& instance) override
        // Run test with dataToUse

Where data is parsed from the file and extracted from the matching ID. The format could be similar to:

    <TESTSETUP id="42" sampleRate="44100" bufferSize="512" base64:pluginState="..." base64: midiSequence ="..." base64:renderedBuffer=".." />

Basically, all the properties apart from the id are simply passed to the test and interpreted accordingly. If the test needs this to run, (i.e. should be skipped if there is no entry) then it can simply pass or return some kind of “skipped” code.

This should also make it possible to run the same test with multiple sets of data, just have several TESTSETUPS.

The idea in pluginval is that you can write tests to do whatever you need, they just have to be general enough to be useful to everyone.

Does this sound like a sensible and flexible enough approach?


Yes that might be enough. I could write a compliment app that creates these XML states (let’s call them plugin snapshots) by loading a plugin, load a state, record some MIDI input and render the audio.

Ideally, the XML file also contains the plugin ID, so you can run all tests in a directory with something like this:

pluginval --test_plugin_snapshots directory_with_xml_files


Yes that sounds sensible and the kind of thing I had in mind.


Thanks @dave96! I’ll defenitly get this to our testing grid! It’s really awesome.

One thing that I think would REALLY simplify things for development would be to use JS Scripting utilitizing JUCE’s already existing engine. (instead of re-compiling each time you’d like a new test).

In this thread you’ve chimed with great suggestion of mimicking some hosts workflow.

With simple scripting any methodology could be easily added without re-compiling. So you’ll end up with some scripts for common workflows.

Imagine how easy it would be to change callback order.

     var p = PluginInstance();
     for (i = 0; i < 16; i++)
         p.prepareToPlay(44100.0, 64*i);
         for (y = 0; y < 200; y++)

With simple API that corresponds to common callbacks (state load/save, prepareToPlay, processBlock, processBlockBypassed, releaseResources, reset).

While scripting won’t be useful for most. they would benefit as it would be very simple to create simple scenarios very fast.
such as:
SpoofCubaseVST3, SpoofLiveVST, SpoofLiveAU etc…

Another thing that would make such utility great is diffing.
Since we’re in the audio-biz. I’d really like to be able and load 2 different VSTs and compare output for same/similar settings. so I could output audio and diff it.

Hope this feedback is worthy.
From here I guess pull-requests are the way to go :slight_smile:


Hi Dave,

Does pluginval test load a folder of plug-ins in different configurations and in different load order (Plug-in A before B and then B before A, etc…)?




Yes, this is exactly the kind of thing that would be extremely useful. I was thinking of ways to get users to easily write tests and was considering some kind of simple dll type plugin format but the overhead of that would probably be too much for anyone to bother.

Scripting bindings would mean people can simply write pure textual tests, dump them in a folder and run them, no need to even recompile.

After having just launched our new plugin a few days ago it and suffered some inevitable host compatibility this use case is rapidly climbing the ladder.

The only annoying thing with scripting is creating the bindings for everything. Oh how I wish we had reflection in C++.
I guess the AudioProcessor class won’t be too tricky to bind as there aren’t that many classes involved. It would be problematic if we go in to workflows similar to what @chrisboy2000 was mentioning earlier as we’d ned ValueTree bindings etc. too.

I’ve also been meaning to try out ChaiScript as it’s got a more complete Javascript syntax and creating bindings might be a bit simpler. This could be a good opportunity to do this…


Hi Rail, no, at the moment, each plugin is tested in its won process sandbox to avoid previous plugins corrupting the heap for subsequent plugins.

The current architecture doesn’t really lend itself to this workflow but I’m sure something could be worked around. Maybe with the file based ValueTree setup we were discussing above with @chrisboy2000?


Any scripting and you’ve got a happy camper.
The reason I’ve suggested JUCE since it’s already included. I guess no one expect scripting to provide EVERY possible method reflection. just enough for trying different flows.


Yeah, I agree JUCE would also be a good candidate. The main reason I mentioned ChaiScript is that I’ve always liked the look of the project but never had a real chance to test it out and this seemed like a good fit.

You’re right, a subset of workable tests would be good. There’s also probably a bunch of helper methods that might also be useful so bind such as those I’ve already used for generating and testing AudioBuffers.