Anouncing "pluginval" an open-source cross-platform plugin validation tool

Correct behavior should be “not crashing” :slight_smile: - but I like your suggestion and will take that route - I’ll load a default preset and have the plugin automatically revert to "running’. Thx for the hint!

This is for the AudioPluginDemo while doing the Editor Automation:

0x000000010f4b250e in juce::Component::internalRepaintUnchecked(juce::Rectangle<int>, bool) at /JUCE/modules/juce_gui_basics/components/juce_Component.cpp:1789
0x000000010f4aaac7 in juce::Component::repaint() at /JUCE/modules/juce_gui_basics/components/juce_Component.cpp:1758
0x000000010f4ac0bd in juce::Component::setOpaque(bool) at /JUCE/modules/juce_gui_basics/components/juce_Component.cpp:723
0x000000010f247317 in juce::MidiKeyboardComponent::colourChanged() at /JUCE/modules/juce_audio_utils/gui/juce_MidiKeyboardComponent.cpp:178
0x000000010f2462d1 in juce::MidiKeyboardComponent::MidiKeyboardComponent(juce::MidiKeyboardState&, juce::MidiKeyboardComponent::Orientation) at /JUCE/modules/juce_audio_utils/gui/juce_MidiKeyboardComponent.cpp:86
0x000000010f246713 in juce::MidiKeyboardComponent::MidiKeyboardComponent(juce::MidiKeyboardState&, juce::MidiKeyboardComponent::Orientation) at /JUCE/modules/juce_audio_utils/gui/juce_MidiKeyboardComponent.cpp:70
0x000000010f0915d5 in JuceDemoPluginAudioProcessor::JuceDemoPluginAudioProcessorEditor::JuceDemoPluginAudioProcessorEditor(JuceDemoPluginAudioProcessor&) at /TESTS/AudioPluginDemo/Source/AudioPluginDemo.h:342
0x000000010f0914ed in JuceDemoPluginAudioProcessor::JuceDemoPluginAudioProcessorEditor::JuceDemoPluginAudioProcessorEditor(JuceDemoPluginAudioProcessor&) at /TESTS/AudioPluginDemo/Source/AudioPluginDemo.h:345
0x000000010f08b0bd in JuceDemoPluginAudioProcessor::createEditor() at /TESTS/AudioPluginDemo/Source/AudioPluginDemo.h:268
0x000000010f1af71d in juce::AudioProcessor::createEditorIfNeeded() at /JUCE/modules/juce_audio_processors/processors/juce_AudioProcessor.cpp:1060
0x000000010f05e165 in juce::JuceVST3EditController::JuceVST3Editor::ContentWrapperComponent::ContentWrapperComponent(juce::JuceVST3EditController::JuceVST3Editor&, juce::AudioProcessor&) at /JUCE/modules/juce_audio_plugin_client/VST3/juce_VST3_Wrapper.cpp:1044
0x000000010f05ca05 in juce::JuceVST3EditController::JuceVST3Editor::ContentWrapperComponent::ContentWrapperComponent(juce::JuceVST3EditController::JuceVST3Editor&, juce::AudioProcessor&) at /JUCE/modules/juce_audio_plugin_client/VST3/juce_VST3_Wrapper.cpp:1046
0x000000010f05c74a in juce::JuceVST3EditController::JuceVST3Editor::JuceVST3Editor(juce::JuceVST3EditController&, juce::AudioProcessor&) at /JUCE/modules/juce_audio_plugin_client/VST3/juce_VST3_Wrapper.cpp:813
0x000000010f05c4e5 in juce::JuceVST3EditController::JuceVST3Editor::JuceVST3Editor(juce::JuceVST3EditController&, juce::AudioProcessor&) at /JUCE/modules/juce_audio_plugin_client/VST3/juce_VST3_Wrapper.cpp:810
0x000000010f05af8a in juce::JuceVST3EditController::createView(char const*) at /JUCE/modules/juce_audio_plugin_client/VST3/juce_VST3_Wrapper.cpp:637
0x0000000100214390 in juce::VST3PluginInstance::tryCreatingView() const at /pluginval-develop/Builds/MacOSX/../../../JUCE/modules/juce_audio_processors/format_types/juce_VST3PluginFormat.cpp:2721
0x000000010020488b in juce::VST3PluginInstance::hasEditor() const at /pluginval-develop/Builds/MacOSX/../../../JUCE/modules/juce_audio_processors/format_types/juce_VST3PluginFormat.cpp:2219
0x00000001000355d7 in createAndShowEditorOnMessageThread(juce::AudioPluginInstance&) at /pluginval-develop/Source/tests/../TestUtilities.h:140
0x000000010003557d in ScopedEditorShower::ScopedEditorShower(juce::AudioPluginInstance&) at /pluginval-develop/Source/tests/../TestUtilities.h:203
0x000000010003552d in ScopedEditorShower::ScopedEditorShower(juce::AudioPluginInstance&) at /pluginval-develop/Source/tests/../TestUtilities.h:204
0x00000001000352b4 in EditorAutomationTest::runTest(PluginTests&, juce::AudioPluginInstance&) at /pluginval-develop/Source/tests/BasicTests.cpp:381
0x000000010002b8bb in PluginTests::testType(juce::PluginDescription const&) at /pluginval-develop/Source/PluginTests.cpp:181
0x000000010002aa5a in PluginTests::runTest() at /pluginval-develop/Source/PluginTests.cpp:102
0x000000010030bc0d in juce::UnitTest::performTest(juce::UnitTestRunner*) at /pluginval-develop/Builds/MacOSX/../../../JUCE/modules/juce_core/unit_tests/juce_UnitTest.cpp:77
0x000000010030c922 in juce::UnitTestRunner::runTests(juce::Array<juce::UnitTest*, juce::DummyCriticalSection, 0> const&, long long) at /pluginval-develop/Builds/MacOSX/../../../JUCE/modules/juce_core/unit_tests/juce_UnitTest.cpp:165
0x0000000100053d2f in runTests(PluginTests&, std::__1::function<void (juce::String const&)>) at /pluginval-develop/Source/Validator.cpp:194
0x00000001000538cf in validate(juce::PluginDescription const&, PluginTests::Options, std::__1::function<void (juce::String const&)>) at /pluginval-develop/Source/Validator.cpp:207
0x000000010005292b in ValidatorSlaveProcess::processRequest(juce::MemoryBlock) at /pluginval-develop/Source/Validator.cpp:498
0x0000000100051810 in ValidatorSlaveProcess::processRequests() at /pluginval-develop/Source/Validator.cpp:445
0x000000010004fb6c in ValidatorSlaveProcess::run() at /pluginval-develop/Source/Validator.cpp:412
0x000000010030305b in juce::Thread::threadEntryPoint() at /pluginval-develop/Builds/MacOSX/../../../JUCE/modules/juce_core/threads/juce_Thread.cpp:96
0x0000000100303525 in juce::juce_threadEntryPoint(void*) at /pluginval-develop/Builds/MacOSX/../../../JUCE/modules/juce_core/threads/juce_Thread.cpp:118
0x0000000100328b3e in juce::threadEntryProc(void*) at /pluginval-develop/Builds/MacOSX/../../../JUCE/modules/juce_core/native/juce_posix_SharedCode.h:914
0x0000000101b0fe6d in _pthread_body ()
0x0000000101b12eff in _pthread_start ()
0x0000000101b0ee61 in thread_start ()

Ah ok, looks like hasEditor() can’t be called from a background thread.
Will get that tidied up.

1 Like

Perfect, now the Editor Automation test doesn’t trigger the assert but I found another issue with it. When the strictness level is > 5, the test performs 1000 blocks instead of 100. The test times out before completing the 1000 blocks. Both pluginval and my plugin were running in debug mode. I’ve changed it to 500 and it seems to be ok, but it might be an issue with plugins with lots of parameters.

Open editor whilst processing is still causing issues but only with VST3 (AU and VST2 are fine). This is with AudioPluginDemo:

0x00007fff59f71b86 in __pthread_kill ()
0x0000000101b71884 in pthread_kill ()
0x00007fff59edb1c9 in abort ()
0x00007fff59fe96e2 in malloc_vreport ()
0x00007fff59fe94a3 in malloc_report ()
0x000000011065fb98 in juce::HeapBlock<juce::ComponentListener*, false>::~HeapBlock() at /JUCE/modules/juce_core/memory/juce_HeapBlock.h:138
0x000000011065fb45 in juce::HeapBlock<juce::ComponentListener*, false>::~HeapBlock() at /JUCE/modules/juce_core/memory/juce_HeapBlock.h:137
0x000000011065fa97 in juce::ArrayBase<juce::ComponentListener*, juce::DummyCriticalSection>::~ArrayBase() at /JUCE/modules/juce_core/containers/juce_ArrayBase.h:56
0x000000011065fa65 in juce::ArrayBase<juce::ComponentListener*, juce::DummyCriticalSection>::~ArrayBase() at /JUCE/modules/juce_core/containers/juce_ArrayBase.h:54
0x000000011065fa45 in juce::Array<juce::ComponentListener*, juce::DummyCriticalSection, 0>::~Array() at /JUCE/modules/juce_core/containers/juce_Array.h:132
0x000000011065fa25 in juce::Array<juce::ComponentListener*, juce::DummyCriticalSection, 0>::~Array() at /JUCE/modules/juce_core/containers/juce_Array.h:132
0x000000011065fa05 in juce::ListenerList<juce::ComponentListener, juce::Array<juce::ComponentListener*, juce::DummyCriticalSection, 0> >::~ListenerList() at /JUCE/modules/juce_core/containers/juce_ListenerList.h:76
0x00000001104aa405 in juce::ListenerList<juce::ComponentListener, juce::Array<juce::ComponentListener*, juce::DummyCriticalSection, 0> >::~ListenerList() at /JUCE/modules/juce_core/containers/juce_ListenerList.h:76
0x00000001104a9798 in juce::Component::~Component() at /JUCE/modules/juce_gui_basics/components/juce_Component.cpp:452
0x00000001104aa425 in juce::Component::~Component() at /JUCE/modules/juce_gui_basics/components/juce_Component.cpp:432
0x00000001104aa449 in juce::Component::~Component() at /JUCE/modules/juce_gui_basics/components/juce_Component.cpp:432
0x0000000110060afb in std::__1::default_delete<juce::AudioProcessorEditor>::operator()(juce::AudioProcessorEditor*) const [inlined] at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/memory:2239
0x0000000110060ad3 in std::__1::unique_ptr<juce::AudioProcessorEditor, std::__1::default_delete<juce::AudioProcessorEditor> >::reset(juce::AudioProcessorEditor*) [inlined] at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/memory:2552
0x0000000110060a56 in std::__1::unique_ptr<juce::AudioProcessorEditor, std::__1::default_delete<juce::AudioProcessorEditor> >::~unique_ptr() [inlined] at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/memory:2506
0x0000000110060a56 in std::__1::unique_ptr<juce::AudioProcessorEditor, std::__1::default_delete<juce::AudioProcessorEditor> >::~unique_ptr() [inlined] at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/memory:2506
0x0000000110060a56 in juce::JuceVST3EditController::JuceVST3Editor::ContentWrapperComponent::~ContentWrapperComponent() at /JUCE/modules/juce_audio_plugin_client/VST3/juce_VST3_Wrapper.cpp:1076
0x000000011005ea55 in juce::JuceVST3EditController::JuceVST3Editor::ContentWrapperComponent::~ContentWrapperComponent() at /JUCE/modules/juce_audio_plugin_client/VST3/juce_VST3_Wrapper.cpp:1070
0x000000011005ea79 in juce::JuceVST3EditController::JuceVST3Editor::ContentWrapperComponent::~ContentWrapperComponent() at /JUCE/modules/juce_audio_plugin_client/VST3/juce_VST3_Wrapper.cpp:1070
0x000000011005d0bd in std::__1::default_delete<juce::JuceVST3EditController::JuceVST3Editor::ContentWrapperComponent>::operator()(juce::JuceVST3EditController::JuceVST3Editor::ContentWrapperComponent*) const [inlined] at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/memory:2239
0x000000011005d098 in std::__1::unique_ptr<juce::JuceVST3EditController::JuceVST3Editor::ContentWrapperComponent, std::__1::default_delete<juce::JuceVST3EditController::JuceVST3Editor::ContentWrapperComponent> >::reset(juce::JuceVST3EditController::JuceVST3Editor::ContentWrapperComponent*) [inlined] at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/memory:2552
0x000000011005d045 in std::__1::unique_ptr<juce::JuceVST3EditController::JuceVST3Editor::ContentWrapperComponent, std::__1::default_delete<juce::JuceVST3EditController::JuceVST3Editor::ContentWrapperComponent> >::operator=(std::nullptr_t) [inlined] at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/memory:2510
0x000000011005d045 in juce::JuceVST3EditController::JuceVST3Editor::removed() at /JUCE/modules/juce_audio_plugin_client/VST3/juce_VST3_Wrapper.cpp:886
0x000000010023c0c0 in juce::VST3PluginWindow::~VST3PluginWindow() at /pluginval/Builds/MacOSX/../../modules/juce/modules/juce_audio_processors/format_types/juce_VST3PluginFormat.cpp:1136
0x000000010023b1c5 in juce::VST3PluginWindow::~VST3PluginWindow() at /pluginval/Builds/MacOSX/../../modules/juce/modules/juce_audio_processors/format_types/juce_VST3PluginFormat.cpp:1135
0x000000010023b1e9 in juce::VST3PluginWindow::~VST3PluginWindow() at /pluginval/Builds/MacOSX/../../modules/juce/modules/juce_audio_processors/format_types/juce_VST3PluginFormat.cpp:1135
0x0000000100036a0e in std::__1::default_delete<juce::AudioProcessorEditor>::operator()(juce::AudioProcessorEditor*) const [inlined] at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/memory:2239
0x00000001000369e6 in std::__1::unique_ptr<juce::AudioProcessorEditor, std::__1::default_delete<juce::AudioProcessorEditor> >::reset(juce::AudioProcessorEditor*) [inlined] at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/memory:2552
0x0000000100036969 in std::__1::unique_ptr<juce::AudioProcessorEditor, std::__1::default_delete<juce::AudioProcessorEditor> >::~unique_ptr() [inlined] at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/memory:2506
0x0000000100036969 in std::__1::unique_ptr<juce::AudioProcessorEditor, std::__1::default_delete<juce::AudioProcessorEditor> >::~unique_ptr() [inlined] at /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/memory:2506
0x0000000100036969 in ScopedEditorShower::~ScopedEditorShower() at /pluginval/Source/tests/../TestUtilities.h:212
0x00000001000328b5 in ScopedEditorShower::~ScopedEditorShower() at /pluginval/Source/tests/../TestUtilities.h:211
0x00000001000322bb in EditorWhilstProcessingTest::runTest(PluginTests&, juce::AudioPluginInstance&) at /pluginval/Source/tests/BasicTests.cpp:138
0x00000001000303d9 in PluginTests::testType(juce::PluginDescription const&)::$_1::operator()() at /pluginval/Source/PluginTests.cpp:174
0x000000010002fea9 in juce::MessageManager::AsyncCallInvoker<PluginTests::testType(juce::PluginDescription const&)::$_1>::messageCallback() at /pluginval/Builds/MacOSX/../../modules/juce/modules/juce_events/messages/juce_MessageManager.h:347
0x000000010040c32e in juce::MessageQueue::deliverNextMessage() at /pluginval/Builds/MacOSX/../../modules/juce/modules/juce_events/native/juce_osx_MessageQueue.h:82
0x000000010040c276 in juce::MessageQueue::runLoopCallback() at /pluginval/Builds/MacOSX/../../modules/juce/modules/juce_events/native/juce_osx_MessageQueue.h:93
0x000000010040c145 in juce::MessageQueue::runLoopSourceCallback(void*) at /pluginval/Builds/MacOSX/../../modules/juce/modules/juce_events/native/juce_osx_MessageQueue.h:101
0x00007fff2cdbb405 in __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ ()
0x00007fff2cdbb3ab in __CFRunLoopDoSource0 ()
0x00007fff2cd9eea9 in __CFRunLoopDoSources0 ()
0x00007fff2cd9e3fa in __CFRunLoopRun ()
0x00007fff2cd9dce4 in CFRunLoopRunSpecific ()
0x00007fff2c037895 in RunCurrentEventLoopInMode ()
0x00007fff2c0375cb in ReceiveNextEventCommon ()
0x00007fff2c037348 in _BlockUntilNextEventMatchingListInModeWithFilter ()
0x00007fff2a2f495b in _DPSNextEvent ()
0x00007fff2a2f36fa in -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] ()
0x00007fff2a2ed75d in -[NSApplication run] ()
0x00000001003f2531 in juce::MessageManager::runDispatchLoop() at /pluginval/Builds/MacOSX/../../modules/juce/modules/juce_events/native/juce_mac_MessageManager.mm:341
0x00000001003f2324 in juce::JUCEApplicationBase::main() at /pluginval/Builds/MacOSX/../../modules/juce/modules/juce_events/messages/juce_ApplicationBase.cpp:262
0x00000001003f1eec in juce::JUCEApplicationBase::main(int, char const**) at /pluginval/Builds/MacOSX/../../modules/juce/modules/juce_events/messages/juce_ApplicationBase.cpp:240
0x000000010000e213 in main at /pluginval/Source/Main.cpp:183
0x00007fff59e33085 in start ()

Ok, I think I’ve just figured out the VST3 editor thing: https://github.com/Tracktion/pluginval/commit/4f1dc7939a16b4625567e04a611dad0571d91fc6

I think the issue was that hasEditor was getting called which on VST3 has to fully create an editor to see if it has one rather than just returning a flag. I think I’ve worked around this now.

Timeout should be fixed now too:

1 Like

Thank you for the quick fix!

@dave96 I spent some time today to look into this issue myself before I saw the discussion at the end of this thread. May you merge your changes concerning the VST3 editor destruction from the develop branch to the master branch? Just to make it easier for other developers.

Ok, done. I’m in the process of making a new release too.

For everyone else reading, I manage the repo in a very similar way to JUCE, putting things on the develop branch and then merging to master periodically.
In general, I would recommend using the develop branch, especially if you’re seeing issues.

Any experimental or new features typically get created on feature branches then squash-merged back to develop so develop should be relatively stable.

2 Likes

Dave, I was just playing around with adding pluginval to CI, and I was following your notes for this. It was working fine this morning but as of your most recent commit the pre-built binary available at https://github.com/Tracktion/pluginval/releases/download/latest_release/pluginval_macOS.zip doesn’t want to be unzipped. I get the following error:

  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.

I can use the v0.2.0 pre-built release works fine, but anyone who is following your CI setup notes might hit this same problem!

Can you try it again now?
I’ve literally just updated that release so you might have tried it in the minute or two downtime…

Ah yeah works now, thanks!

Hey Dave. awesome work on this. I noticed an oddity when it came time for the Parameter Automation test. I ran the test on level 10. After my plugin passed, when I went into Logic ProX and instantiated my plugin, it opened with the same settings that PluginVal had last passed to it during the Parameter Automation test. I still haven’t figured out where my plugin was pulling those settings from, or where PluginVal was writing them to after the test was finished, but it was strange nonetheless. Has anyone else experienced this? It’s almost like the plugin was reading from the settings file that the Stand-Alone version generates, and PluginVal was forcing the plugin to write its settings to that location after the test finished. :thinking::thinking:

I’m not sure how this is actually possible?
Are you sure you don’t have some code in your plugin that’s saving settings to a file somewhere?

nevermind, I found the problem.

on a separate note, is anyone getting this assertion after cloning, building, running in xcode:

Starting test: CommandLineTests / Command line parser...
!!! Test 1 failed: An unhandled exception was thrown!
JUCE Assertion failure in juce_UnitTest.cpp:280

Sorry, I think was due to a change in juce::ConsoleApplication when I updated and rushed through. Will get a fix out today.

Should be fixed on the tip now. Just creating some new releases.

Can anyone verify if having DBG messages in DSP code might cause allocations on the Audio Thread? I keep getting allocation test failures on level 10 and 9 when testing my plugin on debug mode. When I compile and test in release mode, those tests start passing. The only thing I can find is that DBG messages are stripped out.

Separately:
I had 2 suggestions for PluginVal:

  1. it would be awesome to have a slider that lets us control the time when parameters are sent, so we can get a glimpse of their changes.
        // Set random parameter values
        while (--numBlocks >= 0)
        {
            for (auto parameter : parameters)
                parameter->setValue (r.nextFloat());

            ut.resetTimeout();
//            Thread::sleep (10);
            Thread::sleep(250); //can we get a slider for this time so we can see our GUI change?
        }
  1. the editor’s top is hidden by the menu bar on OS X.
        if (editor)
        {
            editor->addToDesktop (0);
            editor->setVisible (true);
//make room on OS X so we see the full editor window
            auto userArea = Desktop::getInstance().getDisplays().getMainDisplay().userArea;
            editor->setTopLeftPosition(userArea.getX() + 10,
                                       userArea.getY() + 10);

            // Pump the message loop for a couple of seconds for the window to initialise itself
            MessageManager::getInstance()->runDispatchLoopUntil (200);
        }

DBG ("something") will allocate. Just step through the process in the debugger and you’ll see what’s going on.

The binary build from github for Windows doesn’t appear to be working anymore. I get no output out of VST3 plugins on the GUI console. I first thought that maybe it was because none of the plugins I tested did anything out of the ordinary. I then tested a plugin of mine where I put an access violation on purpose in prepareToPlay. pluginval still reported nothing unusual happening.