Is that the whole stack trace?
Did you see the guide here about adding pluginval to CI? pluginval/Adding pluginval to CI.md at develop · Tracktion/pluginval · GitHub
1 Like
It is the whole thing, yes. Nope, I somehow missed that guide - having a look now. Thank you very much!
Thank you again, @dave96.
I updated my script to comply with the guidance you offered (thank you again), so the script generated from my python now looks like what is below (I formatted it here to be a little easier to read, but the call to pluginval is all one line).
pluginval fails about once out of every five times I run it on Windows. It runs very consistently on OSX and Linux. (EDIT - this is wrong…see below)
C:\Users\me\Documents\Validators\pluginval.exe
--output-dir C:\Users\me\Documents\SourceControl\gTest\_builds\plug-Example\plug\plug-Example_artefacts\Debug\VST3
--strictness-level 5
--validate C:\Users\me\Documents\SourceControl\gTest\_builds\plug-Example\plug\plug-Example_artefacts\Debug\VST3\plug-Example.vst3
if %ERRORLEVEL% neq 0 exit /b 1
Below is the output - it shows the last few lines from a successful run, then a failure about two minutes later. The line “Started - Validation running for Debug build of plug-Example on Windows 10” is generated by my python script based upon the OS and what cmake/ninja built as well as what folders exist post-build.
...Time taken to run all tests: 2 secs
All tests completed successfully
Finished validating: C:\Users\me\Documents\SourceControl\gTest\_builds\plug-Example\plug\plug-Example_artefacts\Debug\VST3\plug-Example.vst3
ALL TESTS PASSED
PS C:\Users\me\Documents\SourceControl\gTest> python .\scripts\testplug.py plug-Example
Started - Validation running for Debug build of plug-Example on Windows 10
pluginval v0.2.9 - JUCE v6.0.1
*** FAILED: VALIDATION CRASHED
11: BaseThreadInitThunk + 0x14
12: RtlUserThreadStart + 0x21
PS C:\Users\me\Documents\SourceControl\gTest>
The thing really confusing me is that it is not consistent. The OS seems to be stable and has enough resources for everything I have tried to do with it. Currently, I am running the Python scripts manually but would like to put it into a CI.
Are there any other options to run pluginval that could help me determine what is happening, or has anyone any other ideas about what may cause this?
...I spoke too soon. Just had a failure on OSX (10.15.7). Stack trace below.
Note that I waited a minute or two (long enough to copy/paste what is below into a document) then ran the tests again and they worked just fine.
pluginval v0.2.9 - JUCE v6.0.1
*** FAILED: VALIDATION CRASHED
0 pluginval 0x000000010883d1c0 _ZN4juce11SystemStats17getStackBacktraceEv + 64
1 pluginval 0x000000010868e5f6 _ZN12_GLOBAL__N_119getCrashLogContentsEv + 38
2 pluginval 0x000000010868e58b _Z11getCrashLogv + 107
3 pluginval 0x0000000108689680 _ZN20CommandLineValidator14connectionLostEv + 112
4 pluginval 0x00000001086a3278 _ZNSt3__110__function6__funcIZN9Validator16ensureConnectionEvE3$_1NS_9allocatorIS3_EEFvvEEclEv + 56
5 pluginval 0x00000001086a3014 _ZN22ValidatorParentProcess20handleConnectionLostEv + 68
6 pluginval 0x00000001088d400b _ZN4juce22InterprocessConnection15readNextMessageEv + 395
7 pluginval 0x00000001088d418b _ZN4juce22InterprocessConnection9runThreadEv + 139
8 pluginval 0x0000000108851f9a _ZN4juce6Thread16threadEntryPointEv + 282
9 pluginval 0x000000010888423a _ZN4juceL15threadEntryProcEPv + 26
10 libsystem_pthread.dylib 0x00007fff6a861109 _pthread_start + 148
11 libsystem_pthread.dylib 0x00007fff6a85cb8b thread_start + 15
Binary Images:
0x108682000 pluginval
0x7fff6a85b000 libsystem_pthread.dylib
*** FAILED: VALIDATION CRASHED
0 pluginval 0x000000010883d1c0 _ZN4juce11SystemStats17getStackBacktraceEv + 64
1 pluginval 0x000000010868e5f6 _ZN12_GLOBAL__N_119getCrashLogContentsEv + 38
2 pluginval 0x000000010868e58b _Z11getCrashLogv + 107
3 pluginval 0x0000000108689680 _ZN20CommandLineValidator14connectionLostEv + 112
4 pluginval 0x00000001086a3278 _ZNSt3__110__function6__funcIZN9Validator16ensureConnectionEvE3$_1NS_9allocatorIS3_EEFvvEEclEv + 56
5 pluginval 0x00000001086a3014 _ZN22ValidatorParentProcess20handleConnectionLostEv + 68
6 pluginval 0x00000001088db029 _ZN4juce12MessageQueue21runLoopSourceCallbackEPv + 57
7 CoreFoundation 0x00007fff305717e2 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
8 CoreFoundation 0x00007fff30571781 __CFRunLoopDoSource0 + 103
9 CoreFoundation 0x00007fff3057159b __CFRunLoopDoSources0 + 209
10 CoreFoundation 0x00007fff305702ca __CFRunLoopRun + 927
11 CoreFoundation 0x00007fff3056f8ce CFRunLoopRunSpecific + 462
12 HIToolbox 0x00007fff2f19babd RunCurrentEventLoopInMode + 292
13 HIToolbox 0x00007fff2f19b7d5 ReceiveNextEventCommon + 584
14 HIToolbox 0x00007fff2f19b579 _BlockUntilNextEventMatchingListInModeWithFilter + 64
15 AppKit 0x00007fff2d7e2039 _DPSNextEvent + 883
16 AppKit 0x00007fff2d7e0880 -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1352
17 AppKit 0x00007fff2d7d258e -[NSApplication run] + 658
18 pluginval 0x00000001088cfa20 _ZN4juce19JUCEApplicationBase4mainEv + 144
19 pluginval 0x00000001088cf943 _ZN4juce19JUCEApplicationBase4mainEiPPKc + 83
20 libdyld.dylib 0x00007fff6a65ccc9 start + 1
Binary Images:
0x108682000 pluginval
0x7fff304ed000 CoreFoundation
0x7fff2f16c000 HIToolbox
0x7fff2d7a1000 AppKit
0x7fff6a642000 libdyld.dylib
2021-07-05 16:24:51.764 pluginval[70910:1616406] *** Assertion failure in +[NSEvent startPeriodicEventsAfterDelay:withPeriod:], /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1894.60.101/AppKit.subproj/NSEvent.m:4453
2021-07-05 16:24:51.766 pluginval[70910:1616406] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Periodic events are already being generated'
*** First throw call stack:
(
0 CoreFoundation 0x00007fff305ed627 __exceptionPreprocess + 250
1 libobjc.A.dylib 0x00007fff694b45bf objc_exception_throw + 48
2 CoreFoundation 0x00007fff306167c8 +[NSException raise:format:arguments:] + 88
3 Foundation 0x00007fff32d0890d -[NSAssertionHandler handleFailureInMethod:object:file:lineNumber:description:] + 191
4 AppKit 0x00007fff2ddcbb6e +[NSEvent startPeriodicEventsAfterDelay:withPeriod:] + 380
5 pluginval 0x00000001088db071 _ZN4juce12MessageQueue21runLoopSourceCallbackEPv + 129
6 CoreFoundation 0x00007fff305717e2 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
7 CoreFoundation 0x00007fff30571781 __CFRunLoopDoSource0 + 103
8 CoreFoundation 0x00007fff3057159b __CFRunLoopDoSources0 + 209
9 CoreFoundation 0x00007fff305702ca __CFRunLoopRun + 927
10 CoreFoundation 0x00007fff3056f8ce CFRunLoopRunSpecific + 462
11 HIToolbox 0x00007fff2f19babd RunCurrentEventLoopInMode + 292
12 HIToolbox 0x00007fff2f19b7d5 ReceiveNextEventCommon + 584
13 HIToolbox 0x00007fff2f19b579 _BlockUntilNextEventMatchingListInModeWithFilter + 64
14 AppKit 0x00007fff2d7e2039 _DPSNextEvent + 883
15 AppKit 0x00007fff2d7e0880 -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1352
16 AppKit 0x00007fff2d7d258e -[NSApplication run] + 658
17 pluginval 0x00000001088cfa20 _ZN4juce19JUCEApplicationBase4mainEv + 144
18 pluginval 0x00000001088cf943 _ZN4juce19JUCEApplicationBase4mainEiPPKc + 83
19 libdyld.dylib 0x00007fff6a65ccc9 start + 1
)
libc++abi.dylib: terminating with uncaught exception of type NSException
sh: line 1: 70910 Abort trap: 6 /Applications/pluginval.app/Contents/MacOS/pluginval --output-dir /Users/me/Library/Audio/Plug-Ins/VST3 --strictness-level 5 --validate /Users/me/Library/Audio/Plug-Ins/VST3/plug-Example.vst3
If you’re running from the CLI, I’d add the --validate-in-process option to the command. That runs the validation in the same process so avoids any overhead or complexity from the child process handling. (I’m going to make this the default in future as the child process validation is only really useful when you have a GUI).
If you use that, do you still get those crashes?
Thank you @dave96, it appears to work consistently with that flag set. 
I was getting crashes on mac, linux, and windows without it. I doubt this matters, but crashes on windows happened about twice as often.
@dave96 - thanks for this project! I’m just getting started with pluginval and it’s proving very useful!
I was hoping to use Validate In Process to debug issues with our plugin in Xcode. But with Validate In Process turned on, it hits JUCE_ASSERT_MESSAGE_THREAD. I know this is the recommended approach for debugging, so I’m wondering if I’m missing something obvious? Or did something change in the new JUCE releases?
With Validate In Process turned off, I can run validation passes, but it’s not as easy to debug the issues it’s finding!
I’m also seeing assertions in pluginval’s own unit tests when starting, and some crashes when Verbose Logging is on. I entered these as issues 54,55,56 at the pluginval repo. e.g. "Validate in process" hits JUCE_ASSERT_MESSAGE_THREAD · Issue #56 · Tracktion/pluginval · GitHub
Please let me know any advice!
Thanks,
John
But with Validate In Process turned on, it hits JUCE_ASSERT_MESSAGE_THREAD.
I’ve had this same issue as well. You can bypass it if you just hit continue enough, at least in Visual Studio. But it would be nice to know why this is happening
Thanks - I was hitting way too many asserts for that to be practical.
pluginval’s current juce link is to JUCE 6.1.0 (4f01392) and that exhibits the problem.
But, I just found by experimentation that if I build pluginval with an older JUCE 6.0.1, it hits a jassert at startup in SystemStats::getOperatingSystemType, which I can continue through. And then when validating my plugin, it only hits a couple jassertfalse (for not being in the juce message thread in VTS3_Wrapper setComponentState), but I can continue through those and get on with some debugging. Though this is obviously a hack, and has its own issues (it seems to get stuck doing the Editor Automation test forever)
I’ve left some comments on the GitHub issue.
The reason it’s only hitting the assertion with validate-in-process turned on is that with it off, the debugger isn’t attached to the validation child-process.
Thanks Dave- makes sense. I followed up in the GitHub issue.
Another question-- validating a test plugin on strictness level 10, with Num Repeats set to like 30 or 40, i intermittently (rarely) see this failure in VST3Parameter::getText when obtaining the MessageManagerLock:
Starting test: pluginval / Fuzz parameters...
*** FAILED: VALIDATION CRASHED
0 pluginval 0x000000010442c4ec _ZN4juce11SystemStats17getStackBacktraceEv + 88
1 pluginval 0x000000010410bd6c _ZN12_GLOBAL__N_119getCrashLogContentsEv + 32
2 pluginval 0x000000010410bc00 _ZN12_GLOBAL__N_111handleCrashEPv + 24
3 pluginval 0x000000010442c718 _ZN4juceL11handleCrashEi + 32
4 libsystem_platform.dylib 0x00000001996904e4 _sigtramp + 56
5 ??? 0xffff8001995c9ca4 0x0 + 18446603343089147044
6 libc++.1.dylib 0x00000001995c9ca4 _ZNSt3__15mutex4lockEv + 16
7 pluginval 0x00000001044ac2ec _ZNSt3__110lock_guardINS_5mutexEEC2ERS1_ + 44
8 pluginval 0x000000010446551c _ZNSt3__110lock_guardINS_5mutexEEC1ERS1_ + 36
9 pluginval 0x000000010443a868 _ZNK4juce13WaitableEvent6signalEv + 40
10 pluginval 0x00000001044f4f4c _ZNK4juce14MessageManager4Lock4exitEv + 180
11 pluginval 0x00000001044f5834 _ZN4juce18MessageManagerLockD2Ev + 48
12 pluginval 0x00000001044f5874 _ZN4juce18MessageManagerLockD1Ev + 28
13 pluginval 0x000000010431dc44 _ZNK4juce18VST3PluginInstance13VST3Parameter7getTextEfi + 296
14 pluginval 0x000000010415efa4 _ZN18FuzzParametersTest17fuzzTestParameterER11PluginTestsRN4juce23AudioProcessorParameterE + 340
15 pluginval 0x000000010415eca0 _ZN18FuzzParametersTest7runTestER11PluginTestsRN4juce19AudioPluginInstanceE + 136
16 pluginval 0x000000010413289c _ZN11PluginTests8testTypeERKN4juce17PluginDescriptionE + 1976
17 pluginval 0x0000000104131f9c _ZN11PluginTests7runTestEv + 872
18 pluginval 0x000000010444288c _ZN4juce8UnitTest11performTestEPNS_14UnitTestRunnerE + 108
19 pluginval 0x0000000104443418 _ZN4juce14UnitTestRunner8runTestsERKNS_5ArrayIPNS_8UnitTestENS_20DummyCriticalSectionELi0EEEx + 380
20 pluginval 0x0000000104174fe8 _Z8runTestsR11PluginTestsNSt3__18functionIFvRKN4juce6StringEEEE + 240
21 pluginval 0x000000010417386c _Z8validateRKN4juce17PluginDescriptionEN11PluginTests7OptionsENSt3__18functionIFvRKNS_6StringEEEE + 148
22 pluginval 0x00000001041729c8 _ZN21ValidatorChildProcess14processRequestEN4juce11MemoryBlockE + 2412
23 pluginval 0x0000000104171e70 _ZN21ValidatorChildProcess15processRequestsEv + 164
24 pluginval 0x0000000104170404 _ZN21ValidatorChildProcess3runEv + 48
25 pluginval 0x000000010443b220 _ZN4juce6Thread16threadEntryPointEv + 260
26 pluginval 0x000000010443b61c _ZN4juce21juce_threadEntryPointEPv + 24
27 pluginval 0x0000000104461710 _ZN4juceL15threadEntryProcEPv + 40
I’ve seen it on both apple silicon and on intel, in both debug and release builds.
The trace maybe implies memory corruption, but in the debug build, I have Xcode debug settings enabled for address sanitizer, check stack use after return, malloc scribble, and zombie objects.
I’m not sure how much to worry about this. Any ideas?
Thanks again!
-John
Have you tried testing with thread sanitizer enabled?
Do you get the same problem if you use the --validate-in-process flag?
I’m thinking about restructuring the internals to avoid having to do all the complicated IPC which should reduce the potential area for bugs in that regard. It would be interesting to know if this is one of those.
@dave96 -I haven’t really been able to test with validate-in-process due to all the assertions I mentioned previously in issue 56.
@reuk - with thread sanitizer on, it always hits a failure immediately in pluginval when I press the “Test Selected” button (race between IPC thread via sendPingMessage and main thread via launchSlaveProcess, both in writePipe). I reported that as issue 57
Thanks! I think I may have just encountered a very similar issue in the AudioPluginHost. I’ll fix that shortly, and hopefully Pluginval can pull in the fix after that.
2 Likes
Thanks @johnplanetz and @reuk.
There seems to be a few things at play here so I’ll need to clear a bit of time to look in to them.
1 Like
Sounds good! Thanks again for this pluginval project- it is immensely useful!
-John
1 Like
Hey guys, has the JUCE_ASSERT_MESSAGE_THREAD issue been fixed? I’m currently on Juce 6.0.8 and latest pluginval version from the repo and I’m hitting that assert as soon as I start testing with validate in process enabled.
Thanks!
I am assuming this is connected, using JUCE @ develop and pluginval v0.3.0 on macOS 10.15.7, I am hitting asserts in juce_VST3_Wrapper.cpp, @ lines 942, 2669, 2935, and 3022.
I am also (still) running into JUCE_ASSERT_MESSAGE_THREAD whilst trying to integrate Pluginval into my builds.
0.3.0, in debug mode within Visual Studio 22 latest on Windows 10. Juce latest dev 7.0 (but i tried this several times in the last half year, at least).
I thought I was not diving deeply enough into it and kept postponing the integration 
I am curious what the devs who are using it in their pipleline have done, or not done, in order to use Pluginval. Maybe use a much older version of Juce?