Anouncing "pluginval" an open-source cross-platform plugin validation tool


#81

Cool, let me know the results. It might be something that’s worth adding a test for?


#82

I’ve just added a bunch of documents which should help people get up and running with pluginval: https://github.com/Tracktion/pluginval/tree/develop/docs

They are:


#83

Quick question for @dave96 ! Could you add a test in PluginVal which calls the pluginprocessor function “reset” before “prepareToPlay” ? That’s something stupid I guess that AUVal does, and it might help people to prevent some issues when resetting stuff with dependancy to the current sample rate :wink:


#84

Do you mean before prepareToPlay is called at all? Or just in-between calls to releaseResources and prepareToPlay
You can always open a pull request :wink:


#85

It’s before it is call even once. Just found out that today :wink:


#86

I don’t think that’s actually possible with JUCE as it’s guarded against (at least in the VST case):

    void reset() override
    {
        if (isPowerOn)
        {
            setPower (false);
            setPower (true);
        }
    }

#87

I think there is a sort of race condition in the “ParameterThreadSafetyTest”. If the call to “instance.prepareToPlay” is long enough, the thread started by MessageManager::callAsync finishes before prepareToPlay() returns. Then waiter.signal() has been called twice in the async call, but has not yet been waited. The first waiter.wait(); passes, but the test then hangs on the second call waiter.wait(); and then timeouts.


#88

Does moving the MessageManager::callAsync call to just above the first waiter.wait() call solve this for you?

I think that’s still ok and in the spirit of the test.

Or maybe better still would be to use two WaitableEvents and wait/signal them separately. Does that work?


#89

Yes both approaches work. I also think the second (the two WaitableEvents) is the best one.


#90

edit: completely missed the other replies. I’ll try the two suggestions and report back later.

Just found an issue with the “Parameter thread safety test” with one of my plugins. Pluginval quits most of the time without actually crashing and doesn’t stop when running in debug mode (with Validate in process selected, but I get the *** FAILED: Timeout after 30 secs). Sometimes the test is successful though.
It looks like if I remove a line from prepareToPlay that’s taking around 50-60ms to process, it works as it should.
Should I be worried?

Also, I have two requests if possible:

  1. it would be nice if Pluginval window could remember the last plugin tested, or maybe implementing a search/filter to easily find a specific plugin.
  2. most of the time only one test fails, so instead of repeating all the tests while debugging, an option to repeat only one test would be better.

#91

30 seconds is quite a long time to hang for. Are you saying your prepareToPlay method is taking that long?

Yes, a search is on the list to do. It’s just fairly low priority atm.

Again this is on the list to do it’s just I haven’t decided how to stipulate a specific test yet. Some of the names are a bit transient and I’d want them to be short.

How long is validation taking your plugins? In most cases it should be much less than a minute. Is there a specific test you’re failing and trying to figure out?


#92

No no, on this specific plugin I have to initialize some impulses in prepareToPlay and that’s taking around 50-60ms. If I remove that, I don’t get any issue in the Parameter thread safety test. The problem though is that I’m not getting any error anyway, pluginval just quits without crashing. If I run it in debug I can see the timeout after 30 secs error, but nothing more. No indication of any specific error.

Yes, usually less than a minute, but I often run combined tests for more than 30mins, alternating pluginval with auvaltool and some other customs tests.
Right now I have issues with “Parameter thread safety test” and “Allocations during process”. The problem is that they don’t fail every time. It may happen once every 20 tests, so I’d really like to repeat just one test over and over while debugging, instead of waiting for all the tests to complete. I understand if that’s not a priority anyway :slight_smile:


#93

I think maybe you’re misunderstanding how the timeout works. It runs a thread in the background and a steady clock. The clock resets every time a log message is sent. So it’s really just checking that a plugin hasn’t hung for more than 30s.

I’m struggling to see how that 50-60ms of prepareToPlay time affects this. I would have thought there would be more jitter simply in CPU utilisation for this to have an effect of tipping the timeout over 30s.

The reason it just disappears is because it calls Process::terminate to kill the process without calling any signal handlers. That means the debugger doesn’t catch it either and hence why there’s an assertion in there.

If you want to know what it was doing, when the assertion hits, take a look at the stack traces of the other threads as this will be called from the timeout background thread.

Also, you can simply increase the timeout to 60s or so if it’s easier. But I would be concerned if one test is running for more than 30s without logging any output.

Maybe you should try setting the random-seed and repeat options to try and make the test setups more replicable? If you’re running in the debugger you can also configure the juce::UnitTestRunner to break on a test failure (or just put a break point in there) to see what exactly was being called at the time. I added some docs regarding this last week: https://github.com/Tracktion/pluginval/blob/develop/docs/Debugging%20a%20failed%20validation.md

Hope that helps


#94

I see, so the plugin just get stuck and pluginval quits. Anyway, I’ve tested your suggestion:

And the issue is gone. So, as @jpo said, if prepareToPlay is taking long enough, the thread started by MessageManager::callAsync finishes before prepareToPlay() returns, hence the additional 50-60ms that I have may cause problems.

That’s great thanks!


#95

Ah great. So it was that wait condition then. I’ll get that sorted. Thanks for the debugging!


#96

#97

Hello Dave, thx for pluginval. I have a feature request, inspired by a situation I had where a plugin passed pluginval without issues, but still had a major flaw in it. Reason why it passed pluginval was that this plugin really needs a preset loaded in it - when started “empty” it simply doesn’t do much. Is there a way to have pluginval have the plugin come up first, load a preset in it, and then perform the testing? PM me if you need details on this particular case.


#98

Possibly, but it’s a bit difficult to determine what would be the correct behaviour for each plugin.

Wouldn’t a simpler approach be to load a default preset internally on load?


#99

I’ve got another issue. I am testing the VST3 version of a few plugins and everything was fine with the latest pluginval release (0.2.0) from here.
However, if I build the latest from the develop branch, I get a few issues while testing the VST3 version (AU is working fine):

Open editor whilst processing EXC_BAD_ACCESS around ~VST3PluginWindow()

Editor Automation hits JUCE_ASSERT_MESSAGE_MANAGER_IS_LOCKED on any component methods called by my editor.

I’ve tested with a freshly built AudioPluginDemo with the latest JUCE commit in the develop branch and I get the same issue.

I’ve also tested the VST3 with a few hosts (including AudioPluginHost) and everything seems to be working fine anyway.


#100

Have you got a stack trace of where the assertion is being hit?
I can’t see anything obviously going wrong with it…