Anouncing "pluginval" an open-source cross-platform plugin validation tool


#19

I couldn’t agree more, it’s pretty much the same reason I was writing a plugin validator too. I haven’t taken a good look at this yet but one feature that would be useful (and more likely to be adopted by bigger players) is to have the ability to change the output data so for example it could output a JSON or XML file in a particular format that conforms to a unit test standard that can be parsed by CI tools.


#20

Yes, it does and that’s intentional. Essentially this acts in two ways:

  1. For plugin developers to ensure compatibility with JUCE based hosts (and also as a general testing/validation tool, used in addition to the format-specific tools)
  2. For JUCE based host developers (possibly including JUCE itself :wink:) to test compatibility with plugins from all formats and built with all frameworks

It might be easiest if I run over some problem areas/workflows which we will be putting in to practice at Tracktion:


• For Host Developer QA Teams:

Testing plugin compatibility with your host can be an extremely tedious process. Simply getting hold of, updating, registering etc. plugins is extremely time consuming. At Tracktion, we have a contracted QA company (which I can highly recommend if anyone’s interested) so they have a huge database of plugins etc. already installed. I saw an opportunity here for them to periodically run pluginval on their installed plugins, across multiple formats on multiple OSes all essentially automated. They can then give us the reports and let us know if any plugins fail tests which likely mean there will be compatibility problems with Tracktion/Waveform.
We can then get them to do targeted testing with the actual DAW.

Originally, I had this plan to automate the DAW in this way but it seemed better to do it as an external tool and open source it so it can be used by everyone.


• For Host Developer CI:

Similarly to above, if you have a CI setup, you can add a job to periodically run pluginval on any installed plugins and find out compatibility problems early. This will most likely be used to spot regression testing.
I’ve tried very hard to make pluginval as simple to clone or download binaries for precisely for these kind of CI workflows.

The main advantage of having QA run tests like these is that they will already have to spend time updating plugins etc. The set on the build server is likely to be much reduced simply due to time constraints/expiring licences (iLok anyone?) etc. The larger the company, the more likely you are to have a dedicated person to manage this. Unfortunately at Tracktion we’re not quite there yet so we’ll use a combination.


• For Plugin Developer CI:

If you’re a plugin developer, you’ll likely know that testing with the dozens of hosts out there is astoundingly time consuming. pluginval aims to speed up that process by offering an automated way to check for silly mistakes, regressions and even things that are stricter than most hosts will require.

This should lead to a better overall level of software development and increased visibility in the space. Remember, with an open source tool we can add comments and log messages to explain to developers (particularly those new to the domain) why tests are failing and point them in the right direction.

I imagine fuzz testing here will come in extremely useful, particularly for things such as race conditions that can happen rarely and are very difficult to reproduce. Run some fuzzing for a few hours over your plugins and see what happens…

The other reason I like this CI-tooled approach is that developers (myself included) often respond better to machines telling them they’ve done something wrong. Getting a Jenkins email telling you you’ve broken the build due to some failing tests seems easier to take than a person telling you you’ve broken something. (Jenkins then nagging you every time you commit reinforces this message). You can then tidy it up, push it and no one else may even notice…

Of course pluginval should be used in conjunction with existing tools, particularly on automated CI as they don’t take any extra development time to run.


• For Plugin Developers:

Again similarly to above but having an open source project that you can actually run with a debugger attached to see where tests are failing is super useful.

The other side to this which I’ve not discussed yet is essentially some kind of “recognised” or “compatible with” acknowledgement status. At Tracktion, we love to promote plugin developers making cool software and particularly if you take the time to make sure your plugins work with our DAW. We want to add a compatibility table to our own website in order to do this (our user base can then look at this to hopefully find interesting new plugins) but we know testing with hosts is time consuming. If you can send us a passing pluginval report, we’re 95% sure your plugin will be compatible with us and we can add you to this list.

Who knows, in the future we might even add a GitHub badge!


• For Host End Users:

For users of our host, we wanted to provide a tool that they can run on problematic plugins to generate a log file which they can then send to us and plugin developers in order to give us a head start on fixing the problem.

We could even automate the collection of these so we can find the most popular, problematic plugins.


• For Plugin End Users:

Again, similarly to above but end users can use this tool to test their own installed plugins and generate reports to send to the plugin devs. This can save a lot of time and there are many users out there willing to do this providing the process is easy and quick enough.


I think that covers the main ways I see this being used but I’m sure there are many more which I’m excited to hear about!


#21

Yep, I like this idea, I’ve just not really come across them yet.
If you’ve got any links to formats that CI tools use let me know and I’ll see what we can do!


#22

I think JUnit is the most widely used


#23

Thanks, that looks straightforward enough.
Maybe a goo approach would be to simply construct the XML format from the juce::UnitTestRunner TestResults?

You can get these after the test have run using: const TestResult* getResult (int index)
I think TestResult has everything needed to populate JUnit?


#24

I suspect you’re probably right, when I get a moment I’ll take a look.


#25

Hi Dave,

I’ve just came across this post. This looks awesome and thanks for making it GPL. In the ideal case this could evolve to a standard common denominator that both host and plugin developers can agree on. So when you’re plugin validates here, you can safely forward all tech support messages to the host that’s causing the problem :slight_smile:

Without digging too much in the internals I have a feature suggestion that would make it a killer tool for synth development. Basically you would load a plugin state, render a MIDI file to an audio file and compare this to earlier results. If the audio buffers don’t match (within reasonable tolerance), you know there’s something wrong. You can also check how it copes with different sample rates, buffer sizes etc.

In order to make this work, you would need objects with this structure:

struct PluginTestResult
{
    ValueTree testConditions; // contains information like samplerate, buffer size, etc.
    MemoryBlock pluginState; // contains the plugin's state as saved by the host
    MidiMessageSequence midiSequence; // contains arbitrary MIDI data that create sound
    AudioSampleBuffer renderedBuffer; // contains the rendered audio data
};

You would create a few of these objects for the 1.0.0 version (or whatever), save them to an external file, and then render and compare every of these cases whenever you want to run the test suite.

For audio effects you can omit the MidiMessageSequence, and use a input AudioSampleBuffer containing noise or other signals.

Is this something within the scope of this project, and if yes, where would you start implementing this? I’ve been toying with the idea of making a CI test app for synths for quite some time and would be happy to contribute.


#26

Hi Christoph, sorry for the delayed reply, I’ve been thinking about this for the past day. This is indeed a workflow I would like to encourage so would be massively grateful if you would like to contribute.

I think the best approach of this would be to create some kind of XML format that a test can understand and then link it to a test by an ID. (Adding IDs to tests is something I’ve been meaning to add). You could then pass a path to the test runner which could then examine the file for any information it needs to run that test.

In your example, say you had a:

struct PluginRenderTest : public PluginTest
{
    PluginRenderTest()
    int64 getID() override { return 0x42; }
    void setTestData (const ValueTree& dataToUse) override;
    void runTest (UnitTest& ut, AudioPluginInstance& instance) override
    {
        // Run test with dataToUse
    }

Where data is parsed from the file and extracted from the matching ID. The format could be similar to:

<TESTSETUPS>
    <TESTSETUP id="42" sampleRate="44100" bufferSize="512" base64:pluginState="..." base64: midiSequence ="..." base64:renderedBuffer=".." />
</TESTSETUPS>

Basically, all the properties apart from the id are simply passed to the test and interpreted accordingly. If the test needs this to run, (i.e. should be skipped if there is no entry) then it can simply pass or return some kind of “skipped” code.

This should also make it possible to run the same test with multiple sets of data, just have several TESTSETUPS.

The idea in pluginval is that you can write tests to do whatever you need, they just have to be general enough to be useful to everyone.

Does this sound like a sensible and flexible enough approach?


#27

Yes that might be enough. I could write a compliment app that creates these XML states (let’s call them plugin snapshots) by loading a plugin, load a state, record some MIDI input and render the audio.

Ideally, the XML file also contains the plugin ID, so you can run all tests in a directory with something like this:

pluginval --test_plugin_snapshots directory_with_xml_files

#28

Yes that sounds sensible and the kind of thing I had in mind.


#29

Thanks @dave96! I’ll defenitly get this to our testing grid! It’s really awesome.

One thing that I think would REALLY simplify things for development would be to use JS Scripting utilitizing JUCE’s already existing engine. (instead of re-compiling each time you’d like a new test).

In this thread you’ve chimed with great suggestion of mimicking some hosts workflow.

With simple scripting any methodology could be easily added without re-compiling. So you’ll end up with some scripts for common workflows.

Imagine how easy it would be to change callback order.

     var p = PluginInstance();
     for (i = 0; i < 16; i++)
     {
         p.prepareToPlay(44100.0, 64*i);
         for (y = 0; y < 200; y++)
         {
            p.processBlock();
         }
     }

With simple API that corresponds to common callbacks (state load/save, prepareToPlay, processBlock, processBlockBypassed, releaseResources, reset).

While scripting won’t be useful for most. they would benefit as it would be very simple to create simple scenarios very fast.
such as:
SpoofCubaseVST3, SpoofLiveVST, SpoofLiveAU etc…

Another thing that would make such utility great is diffing.
Since we’re in the audio-biz. I’d really like to be able and load 2 different VSTs and compare output for same/similar settings. so I could output audio and diff it.

Hope this feedback is worthy.
From here I guess pull-requests are the way to go :slight_smile:


#30

Hi Dave,

Does pluginval test load a folder of plug-ins in different configurations and in different load order (Plug-in A before B and then B before A, etc…)?

Cheers,

Rail


#31

Yes, this is exactly the kind of thing that would be extremely useful. I was thinking of ways to get users to easily write tests and was considering some kind of simple dll type plugin format but the overhead of that would probably be too much for anyone to bother.

Scripting bindings would mean people can simply write pure textual tests, dump them in a folder and run them, no need to even recompile.

After having just launched our new plugin a few days ago it and suffered some inevitable host compatibility this use case is rapidly climbing the ladder.

The only annoying thing with scripting is creating the bindings for everything. Oh how I wish we had reflection in C++.
I guess the AudioProcessor class won’t be too tricky to bind as there aren’t that many classes involved. It would be problematic if we go in to workflows similar to what @chrisboy2000 was mentioning earlier as we’d ned ValueTree bindings etc. too.

I’ve also been meaning to try out ChaiScript as it’s got a more complete Javascript syntax and creating bindings might be a bit simpler. This could be a good opportunity to do this…


#32

Hi Rail, no, at the moment, each plugin is tested in its won process sandbox to avoid previous plugins corrupting the heap for subsequent plugins.

The current architecture doesn’t really lend itself to this workflow but I’m sure something could be worked around. Maybe with the file based ValueTree setup we were discussing above with @chrisboy2000?


#33

Any scripting and you’ve got a happy camper.
The reason I’ve suggested JUCE since it’s already included. I guess no one expect scripting to provide EVERY possible method reflection. just enough for trying different flows.


#34

Yeah, I agree JUCE would also be a good candidate. The main reason I mentioned ChaiScript is that I’ve always liked the look of the project but never had a real chance to test it out and this seemed like a good fit.

You’re right, a subset of workable tests would be good. There’s also probably a bunch of helper methods that might also be useful so bind such as those I’ve already used for generating and testing AudioBuffers.


#35

The plug-in is Superchord.
I’ve built the app on macOS and it validates fine event at strictness = 10 (AU, VST, VST3).

I suspect the problem is something related to the platform specific dll loading code,
as many plug-ins I tried on Windows produce any kind of console output at level 5 (AAS, Waves, Steinberg, SSL, mine, etc.). The light stays green for a few seconds and process explorer shows very little cpu usage for the launched subprocess.

NI, u-he, Plugin Alliance plug-in’s do work though.

FYI I’m running a Windows 10 x64 April 2018 rig, built pluginval using latest Juce 5.3.2 pulled from the submodule.


#36

Thanks for the heads up. I think I’ve also hit this issue now in but only if the pluginval is run not under the debugger.

It seems under some circumstances the child process is failing to ping back to the master, it’s timing out and being killed. Taking a look now.


#37

Update: the VST3 versions of my plug-ins validate fine on Windows here now, it’s only the VST version that don’t. Steinberg Halion Sonic SE VST3 doesn’t produce any log still.


#38

I’ve been looking at this for ages now and narrowed down the cause to the place dispatching the incoming requests. The child process is definitely set up and receiving the request message, it’s just not passing it on to the validation stage for some reason.

The problem is, now that I’ve added a load of logging to a file, it runs perfectly every time!
Can you grab the latest tip and build it from source and if it fails, send me the log file? You can find the log file by pressing the new “Options” button in the UI and “Show settings folder”.

Additionally, I’ve added an option to validate in the main process which could be useful for debugging. Next step is to add this to the command line tools (this doesn’t currently work as it blocks the message thread).