Anouncing "pluginval" an open-source cross-platform plugin validation tool


#41

@dave96 Great, looks like you nailed it :slight_smile:
All my plug-in’s validate at level 10, and Halion Sonic at level 5.
This helped me notice I had a bug in my tail size code too :stuck_out_tongue_winking_eye:.

Thanks again,
Lorcan


#42

I am happy to see others are seeing output from pluginval. When I run it on my stable of 20 plugins (10 VST, 10 VST3), I get no output at all at level 10. I was worried that maybe I was doing something wrong. And, that was making me feel nervous because I was not sure if it was me, or if the test was really not finding any errors.

Perhaps we need some output indication when each plugin has passed?


#43

Cool! Glad it’s helping to find bug already!

Does it never fail to validate the plugin when you press “Test Selected” now then?
If not, that’s a pain as I’m sure there’s still something a little off with the message delivery. I can’t see why it would ever fail now though. If adding all the logging “fixed” the bug it must be some kind of race condition I’m not seeing…
I think the logging is just changing the timing of some calls.


#44

Can you try the latest tip (building from source) and if you see no output try enabling the “Validate in process” option.

The problem is that there’s a problem where the validation request doesn’t actually start. I’ve added an option to log the slave process output to a file to try and see what’s going on. If you get no output, can you send me that file (again you can find it from the “Options” button) and I might be able to see at what stage its failing.

Thanks everyone!


#45

@dave96 Yes, validation never fails now (tested maybe 20 times).
However I think I stumbled upon a Juce bug in the VST3 host code.
My plug-in returns Steinberg::Vst::kInfiniteTail (aka UINT32_MAX=0xFFFFFFFF),
but the latency is reported as 0 in pluginval.
I’ve narrowed it down to this piece of code in juce_audio_processors\format_types\juce_VST3PluginFormat.cpp line 2299:

return jlimit (0, 0x7fffffff, (int) processor->getTailSamples()) / sampleRate;

processor->getTailSamples() returns a unsigned 32bit int which is cast to an int, so kInfiniteTail=0xFFFFFFFF becomes -1, which is limited to 0 by jlimit().
And boom, just lost a whole tail …


#46

Ok, as soon as I turn off the slave logging I see the problem with no output. I’ll leave it enabled for now but keep looking in to the possible cause…


#47

After downloading and rebuilding, I get the same two errors on each plugin;

Finished validating: VST3-DASR-10c2ce47-db372f2f
*** FAILED: 4 TESTS
Random seed: 0x6e60cc4
Validation started: 24 May 2018 12:54:31pm

Testing plugin: VST3-DASR-10c2ce47-db372f2f

Starting test: PluginValidator / Open plugin (cold)…
!!! Test 1 failed: Expected value: , Actual value: No compatible plug-in format exists for this plug-in
!!! Test 2 failed: Unable to create AudioPluginInstance

Time taken to open plugin (cold): 0

FAILED!! 2 tests failed, out of a total of 2


Starting test: PluginValidator / Open plugin (warm)…
!!! Test 1 failed: Expected value: , Actual value: No compatible plug-in format exists for this plug-in
!!! Test 2 failed: Unable to create AudioPluginInstance

FAILED!! 2 tests failed, out of a total of 2


#48

Can you send me a link to the plugins so I can test them out please?

Did you try the new in-process option?


#49

Yes, I used the Validate option.

The easiest way to get the plugins is to create a login in account on my website, https://www.directap.com, and download what ever format you need, VST, VST3, AU, or AAX, for Mac or PC.

The plugins will be in trial mode for up to 30 uses. However, if you let me know when you have created the account, I’ll put licenses for each plugin in your account, so you can activate them and have unlimited access.


#50

By the way, I just noticed that Pluginval outputs the same two errors for the JuceDemoPlugin.


#51

Hi Dave,
If I get *** FAILED: VALIDATION CRASHED at level 2, do you have any suggestion where to start investigating?
Running pluginval on a Win10/32 laptop.

This also happens with commercial plugins, so it may have to do with my pluginval build and not the plugins…

Thanks for developing this great tool!
Peter


#52

Hi Peter, are you on the develop branch? I’ve been making some changes and additions to this over the past few days.

The main thing you may want to test is with the “–validate-in-process” flag on the command line (also available from the GUI “Options” button). This makes all the calls from the same process so excludes the pipe messaging system from the equation.

At the moment, I think there is a problem with the pipe losing connection. I’m in the process of debugging it but its a bit tricky to actually connect to the child process before the crash occurs and it’s not every time…

Will report more findings when I have them!


#53

Thanks! I actually just pulled the develop branch and am building it, lol :slight_smile:

And YES - this immediately shows me one of my own asserts! Thanks again!


#54

I’ve added some code on develop which should enable these kind of tests now.
I’ve tried to make it as generic as possible by simply allowing the user to supply a file path which gets passes to the tests. You can then grab file file, parse it and test with it as you will. E.g.

//==============================================================================
struct FileTest : public PluginTest
{
    FileTest()
        : PluginTest ("File", 1)
    {
    }

    void runTest (PluginTests& ut, AudioPluginInstance&) override
    {
        const auto f = ut.getOptions().dataFile;
        ut.logMessage ("*** FILE: " + f.getFullPathName());
        ut.logMessage ("*** DATA:\n" + f.loadFileAsString());
        // Parse file data in to a ValueTree or whatever here and then run tests if you understand the format that its in
    }
};

static FileTest fileTest;

This way you don’t actually need to worry abut specifying an ID etc. and the rest of the framework doesn’t need to know what formats each test understands.

The one problem we may run in two is if multiple tests require different files. One way currently around this is to simply re-run the validation specifying a different file each time. This will also rerun all the other tests though so this may need to get a bit smarter by either specifying several paths or limiting the tests to validate.

Lots of options but this should be enough to get you started.


#55

Hi Dave,

this looks like a very useful project.
I just ran my VST effect plugin through it. Passed all tests at level 10.
I was wondering: What do I have to do, in order to make my plugin fail? Just out of curiousity :slight_smile: And for making sure pluginval correctly reports, if a plugin behaves badly.


#56

Thanks @dave96 for this great tool, I have fixed several issues and my plugin now passes all tests
(any more tests upcoming? :wink: )


#57

If you passed then that’s great! In terms of getting it to fail, a crash is a guarantee fail…
On higher levels things like nans, infs and denormals are also fails.

The main thing pluginval tests for at the moment is coverage. It’s a way to make sure that all of your plugin methods are called (including things like labels, string values etc.) and that you can support changing block sizes etc.

For our CI, I download the latest binaries from github and run them on the command line on each of the newly built plugins. If any of the test fail, this fails the build and the reason gets logged. The dev who broke the build can then run the test locally and see why it’s failing.

I hope that helps?


#58

@dave96 don’t know if you have interest in adding AAX support but some code appeared recently on the Avid developer portal for building your own AAX testing tools


#59

There will be more tests coming, one thing I really need to do with a high priority is some multi-bus tests. Changing channels and processing etc. with different configurations is something that’s a real pain to test manually.

I’d also like some harder tests such as processing blocks bigger than the size specified in prepareToPlay as I’ve heard nasty rumours some hosts do that…

The main driver of this is if I (or QA) find a bug or an odd host behaviour, I’ll add a test for it so that’s then automated in future and gets applied to all our plugin builds.

The tests are relatively straightforward to write though, feel free to submit some or send me PRs. The more tests the better!


The next big thing I’d like to do is some AudioProcessor Javascript bindings so you can just deposit some Javascript files as text files and then have it run these. This isn’t hard, it’s just likely to be time-consuming to do.


#60

I certainly would…
AAX is a bit of pain though and there’s lots of secrecy involved with signing their licence agreements etc. Then there’s PACE…

I’ll take a look at the tools though, we might be able to work them in.