Anouncing "pluginval" an open-source cross-platform plugin validation tool

Is that the whole stack trace?

Did you see the guide here about adding pluginval to CI? pluginval/Adding pluginval to CI.md at develop · Tracktion/pluginval · GitHub

1 Like

It is the whole thing, yes. Nope, I somehow missed that guide - having a look now. Thank you very much!

Thank you again, @dave96.

I updated my script to comply with the guidance you offered (thank you again), so the script generated from my python now looks like what is below (I formatted it here to be a little easier to read, but the call to pluginval is all one line).

pluginval fails about once out of every five times I run it on Windows. It runs very consistently on OSX and Linux. (EDIT - this is wrong…see below)

C:\Users\me\Documents\Validators\pluginval.exe 
	--output-dir C:\Users\me\Documents\SourceControl\gTest\_builds\plug-Example\plug\plug-Example_artefacts\Debug\VST3  
	--strictness-level 5 
	--validate C:\Users\me\Documents\SourceControl\gTest\_builds\plug-Example\plug\plug-Example_artefacts\Debug\VST3\plug-Example.vst3
if %ERRORLEVEL% neq 0 exit /b 1

Below is the output - it shows the last few lines from a successful run, then a failure about two minutes later. The line “Started - Validation running for Debug build of plug-Example on Windows 10” is generated by my python script based upon the OS and what cmake/ninja built as well as what folders exist post-build.

...Time taken to run all tests: 2 secs
All tests completed successfully

Finished validating: C:\Users\me\Documents\SourceControl\gTest\_builds\plug-Example\plug\plug-Example_artefacts\Debug\VST3\plug-Example.vst3
ALL TESTS PASSED
PS C:\Users\me\Documents\SourceControl\gTest> python .\scripts\testplug.py plug-Example

Started - Validation running for Debug build of plug-Example on Windows 10
pluginval v0.2.9 - JUCE v6.0.1

*** FAILED: VALIDATION CRASHED
11: BaseThreadInitThunk + 0x14
12: RtlUserThreadStart + 0x21

PS C:\Users\me\Documents\SourceControl\gTest> 

The thing really confusing me is that it is not consistent. The OS seems to be stable and has enough resources for everything I have tried to do with it. Currently, I am running the Python scripts manually but would like to put it into a CI.

Are there any other options to run pluginval that could help me determine what is happening, or has anyone any other ideas about what may cause this?


...I spoke too soon. Just had a failure on OSX (10.15.7). Stack trace below.

Note that I waited a minute or two (long enough to copy/paste what is below into a document) then ran the tests again and they worked just fine.

pluginval v0.2.9 - JUCE v6.0.1

*** FAILED: VALIDATION CRASHED
0   pluginval                           0x000000010883d1c0 _ZN4juce11SystemStats17getStackBacktraceEv + 64
1   pluginval                           0x000000010868e5f6 _ZN12_GLOBAL__N_119getCrashLogContentsEv + 38
2   pluginval                           0x000000010868e58b _Z11getCrashLogv + 107
3   pluginval                           0x0000000108689680 _ZN20CommandLineValidator14connectionLostEv + 112
4   pluginval                           0x00000001086a3278 _ZNSt3__110__function6__funcIZN9Validator16ensureConnectionEvE3$_1NS_9allocatorIS3_EEFvvEEclEv + 56
5   pluginval                           0x00000001086a3014 _ZN22ValidatorParentProcess20handleConnectionLostEv + 68
6   pluginval                           0x00000001088d400b _ZN4juce22InterprocessConnection15readNextMessageEv + 395
7   pluginval                           0x00000001088d418b _ZN4juce22InterprocessConnection9runThreadEv + 139
8   pluginval                           0x0000000108851f9a _ZN4juce6Thread16threadEntryPointEv + 282
9   pluginval                           0x000000010888423a _ZN4juceL15threadEntryProcEPv + 26
10  libsystem_pthread.dylib             0x00007fff6a861109 _pthread_start + 148
11  libsystem_pthread.dylib             0x00007fff6a85cb8b thread_start + 15

Binary Images:
0x108682000 pluginval
0x7fff6a85b000 libsystem_pthread.dylib


*** FAILED: VALIDATION CRASHED
0   pluginval                           0x000000010883d1c0 _ZN4juce11SystemStats17getStackBacktraceEv + 64
1   pluginval                           0x000000010868e5f6 _ZN12_GLOBAL__N_119getCrashLogContentsEv + 38
2   pluginval                           0x000000010868e58b _Z11getCrashLogv + 107
3   pluginval                           0x0000000108689680 _ZN20CommandLineValidator14connectionLostEv + 112
4   pluginval                           0x00000001086a3278 _ZNSt3__110__function6__funcIZN9Validator16ensureConnectionEvE3$_1NS_9allocatorIS3_EEFvvEEclEv + 56
5   pluginval                           0x00000001086a3014 _ZN22ValidatorParentProcess20handleConnectionLostEv + 68
6   pluginval                           0x00000001088db029 _ZN4juce12MessageQueue21runLoopSourceCallbackEPv + 57
7   CoreFoundation                      0x00007fff305717e2 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
8   CoreFoundation                      0x00007fff30571781 __CFRunLoopDoSource0 + 103
9   CoreFoundation                      0x00007fff3057159b __CFRunLoopDoSources0 + 209
10  CoreFoundation                      0x00007fff305702ca __CFRunLoopRun + 927
11  CoreFoundation                      0x00007fff3056f8ce CFRunLoopRunSpecific + 462
12  HIToolbox                           0x00007fff2f19babd RunCurrentEventLoopInMode + 292
13  HIToolbox                           0x00007fff2f19b7d5 ReceiveNextEventCommon + 584
14  HIToolbox                           0x00007fff2f19b579 _BlockUntilNextEventMatchingListInModeWithFilter + 64
15  AppKit                              0x00007fff2d7e2039 _DPSNextEvent + 883
16  AppKit                              0x00007fff2d7e0880 -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1352
17  AppKit                              0x00007fff2d7d258e -[NSApplication run] + 658
18  pluginval                           0x00000001088cfa20 _ZN4juce19JUCEApplicationBase4mainEv + 144
19  pluginval                           0x00000001088cf943 _ZN4juce19JUCEApplicationBase4mainEiPPKc + 83
20  libdyld.dylib                       0x00007fff6a65ccc9 start + 1

Binary Images:
0x108682000 pluginval
0x7fff304ed000 CoreFoundation
0x7fff2f16c000 HIToolbox
0x7fff2d7a1000 AppKit
0x7fff6a642000 libdyld.dylib

2021-07-05 16:24:51.764 pluginval[70910:1616406] *** Assertion failure in +[NSEvent startPeriodicEventsAfterDelay:withPeriod:], /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1894.60.101/AppKit.subproj/NSEvent.m:4453
2021-07-05 16:24:51.766 pluginval[70910:1616406] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Periodic events are already being generated'
*** First throw call stack:
(
        0   CoreFoundation                      0x00007fff305ed627 __exceptionPreprocess + 250
        1   libobjc.A.dylib                     0x00007fff694b45bf objc_exception_throw + 48
        2   CoreFoundation                      0x00007fff306167c8 +[NSException raise:format:arguments:] + 88
        3   Foundation                          0x00007fff32d0890d -[NSAssertionHandler handleFailureInMethod:object:file:lineNumber:description:] + 191
        4   AppKit                              0x00007fff2ddcbb6e +[NSEvent startPeriodicEventsAfterDelay:withPeriod:] + 380
        5   pluginval                           0x00000001088db071 _ZN4juce12MessageQueue21runLoopSourceCallbackEPv + 129
        6   CoreFoundation                      0x00007fff305717e2 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
        7   CoreFoundation                      0x00007fff30571781 __CFRunLoopDoSource0 + 103
        8   CoreFoundation                      0x00007fff3057159b __CFRunLoopDoSources0 + 209
        9   CoreFoundation                      0x00007fff305702ca __CFRunLoopRun + 927
        10  CoreFoundation                      0x00007fff3056f8ce CFRunLoopRunSpecific + 462
        11  HIToolbox                           0x00007fff2f19babd RunCurrentEventLoopInMode + 292
        12  HIToolbox                           0x00007fff2f19b7d5 ReceiveNextEventCommon + 584
        13  HIToolbox                           0x00007fff2f19b579 _BlockUntilNextEventMatchingListInModeWithFilter + 64
        14  AppKit                              0x00007fff2d7e2039 _DPSNextEvent + 883
        15  AppKit                              0x00007fff2d7e0880 -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1352
        16  AppKit                              0x00007fff2d7d258e -[NSApplication run] + 658
        17  pluginval                           0x00000001088cfa20 _ZN4juce19JUCEApplicationBase4mainEv + 144
        18  pluginval                           0x00000001088cf943 _ZN4juce19JUCEApplicationBase4mainEiPPKc + 83
        19  libdyld.dylib                       0x00007fff6a65ccc9 start + 1
)
libc++abi.dylib: terminating with uncaught exception of type NSException
sh: line 1: 70910 Abort trap: 6           /Applications/pluginval.app/Contents/MacOS/pluginval --output-dir /Users/me/Library/Audio/Plug-Ins/VST3 --strictness-level 5 --validate /Users/me/Library/Audio/Plug-Ins/VST3/plug-Example.vst3

If you’re running from the CLI, I’d add the --validate-in-process option to the command. That runs the validation in the same process so avoids any overhead or complexity from the child process handling. (I’m going to make this the default in future as the child process validation is only really useful when you have a GUI).

If you use that, do you still get those crashes?

Thank you @dave96, it appears to work consistently with that flag set. :+1:

I was getting crashes on mac, linux, and windows without it. I doubt this matters, but crashes on windows happened about twice as often.

@dave96 - thanks for this project! I’m just getting started with pluginval and it’s proving very useful!

I was hoping to use Validate In Process to debug issues with our plugin in Xcode. But with Validate In Process turned on, it hits JUCE_ASSERT_MESSAGE_THREAD. I know this is the recommended approach for debugging, so I’m wondering if I’m missing something obvious? Or did something change in the new JUCE releases?

With Validate In Process turned off, I can run validation passes, but it’s not as easy to debug the issues it’s finding!

I’m also seeing assertions in pluginval’s own unit tests when starting, and some crashes when Verbose Logging is on. I entered these as issues 54,55,56 at the pluginval repo. e.g. "Validate in process" hits JUCE_ASSERT_MESSAGE_THREAD · Issue #56 · Tracktion/pluginval · GitHub

Please let me know any advice!

Thanks,
John

But with Validate In Process turned on, it hits JUCE_ASSERT_MESSAGE_THREAD.

I’ve had this same issue as well. You can bypass it if you just hit continue enough, at least in Visual Studio. But it would be nice to know why this is happening

Thanks - I was hitting way too many asserts for that to be practical.

pluginval’s current juce link is to JUCE 6.1.0 (4f01392) and that exhibits the problem.

But, I just found by experimentation that if I build pluginval with an older JUCE 6.0.1, it hits a jassert at startup in SystemStats::getOperatingSystemType, which I can continue through. And then when validating my plugin, it only hits a couple jassertfalse (for not being in the juce message thread in VTS3_Wrapper setComponentState), but I can continue through those and get on with some debugging. Though this is obviously a hack, and has its own issues (it seems to get stuck doing the Editor Automation test forever)