Hey, i’m failing a level 7 test in PluginVal, but the stack trace doesn’t show any of my code being called:
*** FAILED: VALIDATION CRASHED
0 pluginval 0x000000010056fa8d _ZN4juce11SystemStats17getStackBacktraceEv + 109
1 pluginval 0x0000000100014fd8 _ZN12_GLOBAL__N_119getCrashLogContentsEv + 40
2 pluginval 0x0000000100014d9c _ZN12_GLOBAL__N_111handleCrashEPv + 44
3 pluginval 0x000000010056fde6 _ZN4juceL11handleCrashEi + 54
4 libclang_rt.tsan_osx_dynamic.dylib 0x0000000101fe7868 _ZN6__tsanL21CallUserSignalHandlerEPNS_11ThreadStateEbbbiPN11__sanitizer19__sanitizer_siginfoEPv + 648
5 libclang_rt.tsan_osx_dynamic.dylib 0x0000000101ffadf6 _ZL14rtl_sighandleri + 390
6 libsystem_platform.dylib 0x00007fff666cbf5a _sigtramp + 26
7 pluginval 0x0000000100415bc6 _ZNK4juce19VST3ComponentHolder23fillInPluginDescriptionERNS_17PluginDescriptionE + 4182
8 pluginval 0x000000010007323e _ZNK4juce10OwnedArrayINS_14AudioProcessor3BusENS_20DummyCriticalSectionEEixEi + 78
9 pluginval 0x0000000100072d29 _ZN4juce14AudioProcessor6getBusEbi + 121
10 pluginval 0x000000010007478b _ZNK4juce14AudioProcessor6getBusEbi + 75
11 pluginval 0x00000001000746fb _ZNK4juce14AudioProcessor20getChannelCountOfBusEbi + 75
12 pluginval 0x00000001000729f0 _ZNK4juce14AudioProcessor26getMainBusNumInputChannelsEv + 48
13 pluginval 0x0000000100302ec3 _ZN4juce14AudioProcessor15processBypassedIfEEvRNS_11AudioBufferIT_EERNS_10MidiBufferE + 147
14 PFMCPP_Project10 0x0000000108dfafa9 _ZN4juce14AudioProcessor20processBlockBypassedERNS_11AudioBufferIfEERNS_10MidiBufferE + 73
15 PFMCPP_Project10 0x0000000108b93990 _ZN4juce17JuceVST3Component12processAudioIfEEvRN9Steinberg3Vst11ProcessDataERNS_5ArrayIPT_NS_20DummyCriticalSectionELi0EEE + 3440
16 PFMCPP_Project10 0x0000000108b717c2 _ZN4juce17JuceVST3Component7processERN9Steinberg3Vst11ProcessDataE + 1602
17 PFMCPP_Project10 0x0000000108b71c7d _ZThn8_N4juce17JuceVST3Component7processERN9Steinberg3Vst11ProcessDataE + 61
18 pluginval 0x00000001003fbc0c _ZN4juce18VST3PluginInstance12processAudioIfEEvRNS_11AudioBufferIT_EERNS_10MidiBufferEN9Steinberg3Vst19SymbolicSampleSizesEb + 556
19 pluginval 0x00000001003e88a0 _ZN4juce18VST3PluginInstance12processBlockERNS_11AudioBufferIfEERNS_10MidiBufferE + 240
20 pluginval 0x000000010006c5a6 _ZN25ParameterThreadSafetyTest7runTestER11PluginTestsRN4juce19AudioPluginInstanceE + 1382
21 pluginval 0x000000010004ac56 _ZN11PluginTests8testTypeERKN4juce17PluginDescriptionE + 5446
22 pluginval 0x00000001000494da _ZN11PluginTests7runTestEv + 1402
23 pluginval 0x000000010059617e _ZN4juce8UnitTest11performTestEPNS_14UnitTestRunnerE + 206
24 pluginval 0x000000010059766a _ZN4juce14UnitTestRunner8runTestsERKNS_5ArrayIPNS_8UnitTestENS_20DummyCriticalSectionELi0EEEx + 618
25 pluginval 0x00000001000a1bf3 _Z8runTestsR11PluginTestsNSt3__18functionIFvRKN4juce6StringEEEE + 723
26 pluginval 0x00000001000a14a2 _Z8validateRKN4juce17PluginDescriptionEN11PluginTests7OptionsENSt3__18functionIFvRKNS_6StringEEEE + 242
27 pluginval 0x000000010009fe19 _ZN21ValidatorSlaveProcess14processRequestEN4juce11MemoryBlockE + 4521
28 pluginval 0x000000010009e72b _ZN21ValidatorSlaveProcess15processRequestsEv + 1115
29 pluginval 0x000000010009b73b _ZN21ValidatorSlaveProcess3runEv + 91
30 pluginval 0x000000010009b8dd _ZThn24_N21ValidatorSlaveProcess3runEv + 45
31 pluginval 0x0000000100588e6d _ZN4juce6Thread16threadEntryPointEv + 445
32 pluginval 0x0000000100589599 _ZN4juce21juce_threadEntryPointEPv + 41
33 pluginval 0x00000001005dd7ca _ZN4juceL15threadEntryProcEPv + 58
34 libclang_rt.tsan_osx_dynamic.dylib 0x0000000101fe430d __tsan_thread_start_func + 141
35 libsystem_pthread.dylib 0x00000001032ad665 _pthread_body + 340
36 libsystem_pthread.dylib 0x00000001032ad511 _pthread_body + 0
37 libsystem_pthread.dylib 0x00000001032acbfd thread_start + 13
These 4 calls here:
13 pluginval 0x0000000100302ec3 _ZN4juce14AudioProcessor15processBypassedIfEEvRNS_11AudioBufferIT_EERNS_10MidiBufferE + 147
14 PFMCPP_Project10 0x0000000108dfafa9 _ZN4juce14AudioProcessor20processBlockBypassedERNS_11AudioBufferIfEERNS_10MidiBufferE + 73
15 PFMCPP_Project10 0x0000000108b93990 _ZN4juce17JuceVST3Component12processAudioIfEEvRN9Steinberg3Vst11ProcessDataERNS_5ArrayIPT_NS_20DummyCriticalSectionELi0EEE + 3440
16 PFMCPP_Project10 0x0000000108b717c2 _ZN4juce17JuceVST3Component7processERN9Steinberg3Vst11ProcessDataE + 1602
17 PFMCPP_Project10 0x0000000108b71c7d _ZThn8_N4juce17JuceVST3Component7processERN9Steinberg3Vst11ProcessDataE + 61
It seems to be calling:
juce::AudioProcessor::processBlockBypassed(AudioBuffer& MidiBuffer&)
I haven’t defined this function in my project, so it’s using the default implementation.
Any ideas how to figure this out? I turned on ASAN and TSAN, and the crash kept happening with no change to the crash report.
1 Like
dave96
January 21, 2020, 8:09am
#2
Have you tried building pluginval from source and running the tests in debug?
yes, built it from source. But I have been using PluginVal as the executable to run when running the plugin itself. Are you saying to run PluginVal via the debugger?
dave96
January 21, 2020, 8:14am
#4
Yeah, clone the repo and build then run the pluginval binary. Then you’ll catch the crash in the debugger and get a full stack trace.
This is detailed here: https://github.com/Tracktion/pluginval/blob/master/docs/Debugging%20a%20failed%20validation.md
Same crash with ASAN. Can you explain what this test does?
Starting test: pluginval / Background thread state...
*** FAILED: VALIDATION CRASHED
0 pluginval 0x000000010130dcf8 _ZN4juce11SystemStats17getStackBacktraceEv + 424
1 pluginval 0x0000000100039b22 _ZN12_GLOBAL__N_119getCrashLogContentsEv + 290
2 pluginval 0x00000001000394af _ZN12_GLOBAL__N_111handleCrashEPv + 271
3 pluginval 0x000000010130e458 _ZN4juceL11handleCrashEi + 24
4 libsystem_platform.dylib 0x00007fff666cbf5a _sigtramp + 26
5 pluginval 0x0000000100221093 _ZN21ValidatorSlaveProcess14processRequestEN4juce11MemoryBlockE + 12307
6 pluginval 0x000000010018d984 _ZNK4juce10OwnedArrayINS_14AudioProcessor3BusENS_20DummyCriticalSectionEEixEi + 420
7 pluginval 0x000000010018cab2 _ZN4juce14AudioProcessor6getBusEbi + 242
8 pluginval 0x0000000100190e21 _ZNK4juce14AudioProcessor6getBusEbi + 145
9 pluginval 0x0000000100190ce1 _ZNK4juce14AudioProcessor20getChannelCountOfBusEbi + 145
10 pluginval 0x000000010018c446 _ZNK4juce14AudioProcessor26getMainBusNumInputChannelsEv + 86
11 pluginval 0x0000000100a5e526 _ZN4juce14AudioProcessor15processBypassedIfEEvRNS_11AudioBufferIT_EERNS_10MidiBufferE + 166
12 PFMCPP_Project10 0x000000010e95bd85 _ZN4juce14AudioProcessor20processBlockBypassedERNS_11AudioBufferIfEERNS_10MidiBufferE + 37
13 PFMCPP_Project10 0x000000010e58fb66 _ZN4juce17JuceVST3Component12processAudioIfEEvRN9Steinberg3Vst11ProcessDataERNS_5ArrayIPT_NS_20DummyCriticalSectionELi0EEE + 6870
14 PFMCPP_Project10 0x000000010e55ba63 _ZN4juce17JuceVST3Component7processERN9Steinberg3Vst11ProcessDataE + 3123
15 pluginval 0x0000000100db71a8 _ZN4juce18VST3PluginInstance12processAudioIfEEvRNS_11AudioBufferIT_EERNS_10MidiBufferEN9Steinberg3Vst19SymbolicSampleSizesEb + 3160
16 pluginval 0x0000000100d81f85 _ZN4juce18VST3PluginInstance12processBlockERNS_11AudioBufferIfEERNS_10MidiBufferE + 581
17 pluginval 0x000000010017ab9c _ZN25ParameterThreadSafetyTest7runTestER11PluginTestsRN4juce19AudioPluginInstanceE + 4572
18 pluginval 0x00000001001019c3 _ZN11PluginTests8testTypeERKN4juce17PluginDescriptionE + 14675
19 pluginval 0x00000001000fd98e _ZN11PluginTests7runTestEv + 3518
20 pluginval 0x000000010138c8eb _ZN4juce8UnitTest11performTestEPNS_14UnitTestRunnerE + 443
21 pluginval 0x000000010139055f _ZN4juce14UnitTestRunner8runTestsERKNS_5ArrayIPNS_8UnitTestENS_20DummyCriticalSectionELi0EEEx + 1823
22 pluginval 0x0000000100226386 _Z8runTestsR11PluginTestsNSt3__18functionIFvRKN4juce6StringEEEE + 3702
23 pluginval 0x0000000100224901 _Z8validateRKN4juce17PluginDescriptionEN11PluginTests7OptionsENSt3__18functionIFvRKNS_6StringEEEE + 817
24 pluginval 0x0000000100221093 _ZN21ValidatorSlaveProcess14processRequestEN4juce11MemoryBlockE + 12307
25 pluginval 0x000000010021c2ce _ZN21ValidatorSlaveProcess15processRequestsEv + 5582
26 pluginval 0x000000010020ff03 _ZN21ValidatorSlaveProcess3runEv + 371
27 pluginval 0x0000000101366517 _ZN4juce6Thread16threadEntryPointEv + 1527
28 pluginval 0x00000001013674ff _ZN4juce21juce_threadEntryPointEPv + 79
29 pluginval 0x00000001014803a6 _ZN4juceL15threadEntryProcEPv + 38
30 libsystem_pthread.dylib 0x00000001065cb665 _pthread_body + 340
31 libsystem_pthread.dylib 0x00000001065cb511 _pthread_body + 0
32 libsystem_pthread.dylib 0x00000001065cabfd thread_start + 13
Looks like it’s this test:
/** Sets plugin state from a background thread whilst the plugin window is
created on the main thread. This simulates behaviour seen in certain hosts.
*/
struct BackgroundThreadStateTest : public PluginTest
{
BackgroundThreadStateTest()
: PluginTest ("Background thread state", 7,
{ Requirements::Thread::backgroundThread, Requirements::GUI::requiresGUI })
{
}
void runTest (PluginTests& ut, AudioPluginInstance& instance) override
{
auto r = ut.getRandom();
ScopedEditorShower editor (instance);
auto parameters = getNonBypassAutomatableParameters (instance);
MemoryBlock originalState;
// Read state
instance.getStateInformation (originalState);
// Set random parameter values
for (auto parameter : parameters)
parameter->setValue (r.nextFloat());
// Restore original state
instance.setStateInformation (originalState.getData(), (int) originalState.getSize());
// Allow for async reaction to state changes
Thread::sleep (2000);
}
};
My plugin has no parameters, and doesn’t save/restore state currently.
dave96
January 21, 2020, 8:24am
#7
I can’t remember off the top of my head but it will be in the source if you grep for “Background thread state”.
My guess would be that it is processing the plugin on one thread whilst setting parameters from another thread.
The crash could then be from a data race on some object you have?
Also, you might want to test with the “–validate-in-process” option to avoid any of the IPC stuff affecting it. You’ll also get a cleaner stack trace.
dave96:
–validate-in-process
Alright, thanks. that turned out to be really helpful:
Somewhere further into that stack trace, the allocatedBytes for the buffer is 0.
And it looks to be in this function:
template <typename FloatType>
void processAudio (Vst::ProcessData& data, Array<FloatType*>& channelList)
Any ideas on this oddity?
Again, this isn’t my code. this is JUCE code causing the crash.
@dave96 do you recommend using the latest commit of PluginVal or the Master branch release from March 2019?
dave96
January 21, 2020, 9:51am
#10
The latest tip of develop is usually the most up to date so I’d probably go with that.
The reason that hasn’t been merged to develop and a new release been has been because of notarisation and the ever changing requirements of that.
Now we’re through the final (hopefully) iteration of that I can get the builds notarised and tested to bring everything in line.
dave96
January 21, 2020, 9:54am
#11
Btw, allocatedBytes in the snippet you showed is correct as it uses the setDataToReferTo method.
Do you have any ideas why, when I step into in the debugger after adding a breakpoint on that instance.processBlock(ab, mb); line in the first screenshot, if I continue stepping in, eventually the program kills the debugger, but continues running. I’m left with some rogue copies of PluginVal running that don’t show up in Activity Monitor, and can only be killed via the Force Quit popup.
dave96
January 21, 2020, 5:47pm
#13
That sounds like a broken pipe and then zombie process, possibly resulting from the cause of the crash.
Does this happen running with the —validate-in-process option I mentioned previously?
That’s actually the only place it happened. When I disabled that feature, it didn’t crash. Also, updating PluginVal’s JUCE Submodule to the latest tip of JUCE seems to have resolved the crash. I can’t get it to happen anymore. The project passed Level10 testing.
I was also doing std::vector<Atomic<float>> levels in my project, and levels.resize(getTotalNumInputChannels()) in prepareToPlay() but changed that to a fixed number of atomics, so maybe that was part of it. Who knows… I can’t get it to crash or kernel panic anymore as a result, which is good.
dave96
January 21, 2020, 7:33pm
#15
It was causing kernel panics?
It could be a data race in the juce VST3 wrapper that has been fixed recently. I’ll update in pluginval soon.
Yeah. Happened 2 times. I think I was using an audio buffer inside a fifo from the GUI thread before it had been initialized in preparedToPlay. The kernel panic error log showed something along the lines of audio kit drivers exploding.
I’m still trying to figure out what actually caused it…
Looks like the Kernel Panic is still there. the log is not helpful:
*** Panic Report ***
panic(cpu 6 caller 0xffffff8011d887af): Kernel trap at 0xffffff8011b9305b, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0xffffff9144345000, CR3: 0x0000000015979000, CR4: 0x00000000001627e0
RAX: 0xffffff80e3a22fb8, RBX: 0xffffff9144325000, RCX: 0x00000000000053f7, RDX: 0x0000000000044000
RSP: 0xffffff9122333ce8, RBP: 0xffffff9122333d70, RSI: 0xffffff9144345000, RDI: 0xffffff80e3a3d000
R8: 0x0000000000008800, R9: 0xffffff8029f8e300, R10: 0xffffff7f92759090, R11: 0x0000000000000000
R12: 0xffffff8029f8e250, R13: 0x00000000000093f7, R14: 0xffffff8029f8e250, R15: 0x0000000000000bf7
RFL: 0x0000000000010206, RIP: 0xffffff8011b9305b, CS: 0x0000000000000008, SS: 0x0000000000000010
Fault CR2: 0xffffff9144345000, Error code: 0x0000000000000000, Fault CPU: 0x6, PL: 0, VF: 1
Backtrace (CPU 6), Frame : Return Address
0xffffff91223337b0 : 0xffffff8011c6ce46
0xffffff9122333800 : 0xffffff8011d963b4
0xffffff9122333840 : 0xffffff8011d88584
0xffffff91223338b0 : 0xffffff8011c1ee60
0xffffff91223338d0 : 0xffffff8011c6c8bc
0xffffff9122333a00 : 0xffffff8011c6c67c
0xffffff9122333a60 : 0xffffff8011d887af
0xffffff9122333bd0 : 0xffffff8011c1ee60
0xffffff9122333bf0 : 0xffffff8011b9305b
0xffffff9122333d70 : 0xffffff7f92519d8f
0xffffff9122333dc0 : 0xffffff7f92519b5c
0xffffff9122333de0 : 0xffffff7f92519915
0xffffff9122333e30 : 0xffffff7f9251465e
0xffffff9122333ea0 : 0xffffff7f92510be3
0xffffff9122333ed0 : 0xffffff8011ca6a24
0xffffff9122333f40 : 0xffffff8011ca6585
0xffffff9122333fa0 : 0xffffff8011c1e557
Kernel Extensions in backtrace:
com.apple.iokit.IOAudioFamily(206.5)[2F090399-C028-3F14-AE4F-BF49A3D602E5]@0xffffff7f92506000->0xffffff7f92544fff
dependency: com.apple.vecLib.kext(1.2.0)[F41EEA34-47C4-3ADE-BA77-D7A3CCA437B9]@0xffffff7f92451000
If anyone has ever experienced a kernel panic due to the iokit.IOAudioFamily extension, I’d love to learn more about your experience!
