Anouncing "pluginval" an open-source cross-platform plugin validation tool


#21

Yep, I like this idea, I’ve just not really come across them yet.
If you’ve got any links to formats that CI tools use let me know and I’ll see what we can do!


#22

I think JUnit is the most widely used


#23

Thanks, that looks straightforward enough.
Maybe a goo approach would be to simply construct the XML format from the juce::UnitTestRunner TestResults?

You can get these after the test have run using: const TestResult* getResult (int index)
I think TestResult has everything needed to populate JUnit?


#24

I suspect you’re probably right, when I get a moment I’ll take a look.


#25

Hi Dave,

I’ve just came across this post. This looks awesome and thanks for making it GPL. In the ideal case this could evolve to a standard common denominator that both host and plugin developers can agree on. So when you’re plugin validates here, you can safely forward all tech support messages to the host that’s causing the problem :slight_smile:

Without digging too much in the internals I have a feature suggestion that would make it a killer tool for synth development. Basically you would load a plugin state, render a MIDI file to an audio file and compare this to earlier results. If the audio buffers don’t match (within reasonable tolerance), you know there’s something wrong. You can also check how it copes with different sample rates, buffer sizes etc.

In order to make this work, you would need objects with this structure:

struct PluginTestResult
{
    ValueTree testConditions; // contains information like samplerate, buffer size, etc.
    MemoryBlock pluginState; // contains the plugin's state as saved by the host
    MidiMessageSequence midiSequence; // contains arbitrary MIDI data that create sound
    AudioSampleBuffer renderedBuffer; // contains the rendered audio data
};

You would create a few of these objects for the 1.0.0 version (or whatever), save them to an external file, and then render and compare every of these cases whenever you want to run the test suite.

For audio effects you can omit the MidiMessageSequence, and use a input AudioSampleBuffer containing noise or other signals.

Is this something within the scope of this project, and if yes, where would you start implementing this? I’ve been toying with the idea of making a CI test app for synths for quite some time and would be happy to contribute.


#26

Hi Christoph, sorry for the delayed reply, I’ve been thinking about this for the past day. This is indeed a workflow I would like to encourage so would be massively grateful if you would like to contribute.

I think the best approach of this would be to create some kind of XML format that a test can understand and then link it to a test by an ID. (Adding IDs to tests is something I’ve been meaning to add). You could then pass a path to the test runner which could then examine the file for any information it needs to run that test.

In your example, say you had a:

struct PluginRenderTest : public PluginTest
{
    PluginRenderTest()
    int64 getID() override { return 0x42; }
    void setTestData (const ValueTree& dataToUse) override;
    void runTest (UnitTest& ut, AudioPluginInstance& instance) override
    {
        // Run test with dataToUse
    }

Where data is parsed from the file and extracted from the matching ID. The format could be similar to:

<TESTSETUPS>
    <TESTSETUP id="42" sampleRate="44100" bufferSize="512" base64:pluginState="..." base64: midiSequence ="..." base64:renderedBuffer=".." />
</TESTSETUPS>

Basically, all the properties apart from the id are simply passed to the test and interpreted accordingly. If the test needs this to run, (i.e. should be skipped if there is no entry) then it can simply pass or return some kind of “skipped” code.

This should also make it possible to run the same test with multiple sets of data, just have several TESTSETUPS.

The idea in pluginval is that you can write tests to do whatever you need, they just have to be general enough to be useful to everyone.

Does this sound like a sensible and flexible enough approach?


#27

Yes that might be enough. I could write a compliment app that creates these XML states (let’s call them plugin snapshots) by loading a plugin, load a state, record some MIDI input and render the audio.

Ideally, the XML file also contains the plugin ID, so you can run all tests in a directory with something like this:

pluginval --test_plugin_snapshots directory_with_xml_files

#28

Yes that sounds sensible and the kind of thing I had in mind.


#29

Thanks @dave96! I’ll defenitly get this to our testing grid! It’s really awesome.

One thing that I think would REALLY simplify things for development would be to use JS Scripting utilitizing JUCE’s already existing engine. (instead of re-compiling each time you’d like a new test).

In this thread you’ve chimed with great suggestion of mimicking some hosts workflow.

With simple scripting any methodology could be easily added without re-compiling. So you’ll end up with some scripts for common workflows.

Imagine how easy it would be to change callback order.

     var p = PluginInstance();
     for (i = 0; i < 16; i++)
     {
         p.prepareToPlay(44100.0, 64*i);
         for (y = 0; y < 200; y++)
         {
            p.processBlock();
         }
     }

With simple API that corresponds to common callbacks (state load/save, prepareToPlay, processBlock, processBlockBypassed, releaseResources, reset).

While scripting won’t be useful for most. they would benefit as it would be very simple to create simple scenarios very fast.
such as:
SpoofCubaseVST3, SpoofLiveVST, SpoofLiveAU etc…

Another thing that would make such utility great is diffing.
Since we’re in the audio-biz. I’d really like to be able and load 2 different VSTs and compare output for same/similar settings. so I could output audio and diff it.

Hope this feedback is worthy.
From here I guess pull-requests are the way to go :slight_smile:


#30

Hi Dave,

Does pluginval test load a folder of plug-ins in different configurations and in different load order (Plug-in A before B and then B before A, etc…)?

Cheers,

Rail


#31

Yes, this is exactly the kind of thing that would be extremely useful. I was thinking of ways to get users to easily write tests and was considering some kind of simple dll type plugin format but the overhead of that would probably be too much for anyone to bother.

Scripting bindings would mean people can simply write pure textual tests, dump them in a folder and run them, no need to even recompile.

After having just launched our new plugin a few days ago it and suffered some inevitable host compatibility this use case is rapidly climbing the ladder.

The only annoying thing with scripting is creating the bindings for everything. Oh how I wish we had reflection in C++.
I guess the AudioProcessor class won’t be too tricky to bind as there aren’t that many classes involved. It would be problematic if we go in to workflows similar to what @chrisboy2000 was mentioning earlier as we’d ned ValueTree bindings etc. too.

I’ve also been meaning to try out ChaiScript as it’s got a more complete Javascript syntax and creating bindings might be a bit simpler. This could be a good opportunity to do this…


#32

Hi Rail, no, at the moment, each plugin is tested in its won process sandbox to avoid previous plugins corrupting the heap for subsequent plugins.

The current architecture doesn’t really lend itself to this workflow but I’m sure something could be worked around. Maybe with the file based ValueTree setup we were discussing above with @chrisboy2000?


#33

Any scripting and you’ve got a happy camper.
The reason I’ve suggested JUCE since it’s already included. I guess no one expect scripting to provide EVERY possible method reflection. just enough for trying different flows.


#34

Yeah, I agree JUCE would also be a good candidate. The main reason I mentioned ChaiScript is that I’ve always liked the look of the project but never had a real chance to test it out and this seemed like a good fit.

You’re right, a subset of workable tests would be good. There’s also probably a bunch of helper methods that might also be useful so bind such as those I’ve already used for generating and testing AudioBuffers.


#35

The plug-in is Superchord.
I’ve built the app on macOS and it validates fine event at strictness = 10 (AU, VST, VST3).

I suspect the problem is something related to the platform specific dll loading code,
as many plug-ins I tried on Windows produce any kind of console output at level 5 (AAS, Waves, Steinberg, SSL, mine, etc.). The light stays green for a few seconds and process explorer shows very little cpu usage for the launched subprocess.

NI, u-he, Plugin Alliance plug-in’s do work though.

FYI I’m running a Windows 10 x64 April 2018 rig, built pluginval using latest Juce 5.3.2 pulled from the submodule.


#36

Thanks for the heads up. I think I’ve also hit this issue now in but only if the pluginval is run not under the debugger.

It seems under some circumstances the child process is failing to ping back to the master, it’s timing out and being killed. Taking a look now.


#37

Update: the VST3 versions of my plug-ins validate fine on Windows here now, it’s only the VST version that don’t. Steinberg Halion Sonic SE VST3 doesn’t produce any log still.


#38

I’ve been looking at this for ages now and narrowed down the cause to the place dispatching the incoming requests. The child process is definitely set up and receiving the request message, it’s just not passing it on to the validation stage for some reason.

The problem is, now that I’ve added a load of logging to a file, it runs perfectly every time!
Can you grab the latest tip and build it from source and if it fails, send me the log file? You can find the log file by pressing the new “Options” button in the UI and “Show settings folder”.

Additionally, I’ve added an option to validate in the main process which could be useful for debugging. Next step is to add this to the command line tools (this doesn’t currently work as it blocks the message thread).


#39

@dave96 I’ve grabbed the latest sources, unfortunately I get build errors (MSVC 15.7):

pluginval\source\maincomponent.cpp(121): error C2440: '<function-style-cast>': cannot convert from 'const MainComponent::{ctor}::<lambda_0438b006e63335d69881f297f7600b26> *'
	to 'juce::Component::SafePointer<MainComponent>' [pluginval\Builds\VisualStudio2017\pluginval_App.vcxproj]
	 pluginval\source\maincomponent.cpp(121): note: No constructor could take the source type, or constructor overload resolution was ambiguous
2>pluginval\source\maincomponent.cpp(121): error C2119: 'sp': the type for 'auto' cannot be deduced from an empty initializer [pluginval\Builds\VisualStudio2
017\pluginval_App.vcxproj]
2>pluginval\source\maincomponent.cpp(126): error C2227: left of '->validator' must point to class/struct/union/generic type [pluginval\Builds\VisualStudio201
7\pluginval_App.vcxproj]
 pluginval\source\maincomponent.cpp(126): note: type is 'int'
2>pluginval\source\maincomponent.cpp(126): error C2228: left of '.setValidateInProcess' must have class/struct/union [pluginval\Builds\VisualStudio2017\plugi
nval_App.vcxproj]

#40

Oh I see what’s happening. The onClick lambda isn’t marked as mutable so the this pointer is stored as const, this means it can’t be converted to a non-const SafePointer.

Should be fixed now.