Weird PluginVal crash

Hey, i’m failing a level 7 test in PluginVal, but the stack trace doesn’t show any of my code being called:

*** FAILED: VALIDATION CRASHED

0   pluginval                           0x000000010056fa8d _ZN4juce11SystemStats17getStackBacktraceEv + 109
1   pluginval                           0x0000000100014fd8 _ZN12_GLOBAL__N_119getCrashLogContentsEv + 40
2   pluginval                           0x0000000100014d9c _ZN12_GLOBAL__N_111handleCrashEPv + 44
3   pluginval                           0x000000010056fde6 _ZN4juceL11handleCrashEi + 54
4   libclang_rt.tsan_osx_dynamic.dylib  0x0000000101fe7868 _ZN6__tsanL21CallUserSignalHandlerEPNS_11ThreadStateEbbbiPN11__sanitizer19__sanitizer_siginfoEPv + 648
5   libclang_rt.tsan_osx_dynamic.dylib  0x0000000101ffadf6 _ZL14rtl_sighandleri + 390
6   libsystem_platform.dylib            0x00007fff666cbf5a _sigtramp + 26
7   pluginval                           0x0000000100415bc6 _ZNK4juce19VST3ComponentHolder23fillInPluginDescriptionERNS_17PluginDescriptionE + 4182
8   pluginval                           0x000000010007323e _ZNK4juce10OwnedArrayINS_14AudioProcessor3BusENS_20DummyCriticalSectionEEixEi + 78
9   pluginval                           0x0000000100072d29 _ZN4juce14AudioProcessor6getBusEbi + 121
10  pluginval                           0x000000010007478b _ZNK4juce14AudioProcessor6getBusEbi + 75
11  pluginval                           0x00000001000746fb _ZNK4juce14AudioProcessor20getChannelCountOfBusEbi + 75
12  pluginval                           0x00000001000729f0 _ZNK4juce14AudioProcessor26getMainBusNumInputChannelsEv + 48
13  pluginval                           0x0000000100302ec3 _ZN4juce14AudioProcessor15processBypassedIfEEvRNS_11AudioBufferIT_EERNS_10MidiBufferE + 147
14  PFMCPP_Project10                    0x0000000108dfafa9 _ZN4juce14AudioProcessor20processBlockBypassedERNS_11AudioBufferIfEERNS_10MidiBufferE + 73
15  PFMCPP_Project10                    0x0000000108b93990 _ZN4juce17JuceVST3Component12processAudioIfEEvRN9Steinberg3Vst11ProcessDataERNS_5ArrayIPT_NS_20DummyCriticalSectionELi0EEE + 3440
16  PFMCPP_Project10                    0x0000000108b717c2 _ZN4juce17JuceVST3Component7processERN9Steinberg3Vst11ProcessDataE + 1602
17  PFMCPP_Project10                    0x0000000108b71c7d _ZThn8_N4juce17JuceVST3Component7processERN9Steinberg3Vst11ProcessDataE + 61
18  pluginval                           0x00000001003fbc0c _ZN4juce18VST3PluginInstance12processAudioIfEEvRNS_11AudioBufferIT_EERNS_10MidiBufferEN9Steinberg3Vst19SymbolicSampleSizesEb + 556
19  pluginval                           0x00000001003e88a0 _ZN4juce18VST3PluginInstance12processBlockERNS_11AudioBufferIfEERNS_10MidiBufferE + 240
20  pluginval                           0x000000010006c5a6 _ZN25ParameterThreadSafetyTest7runTestER11PluginTestsRN4juce19AudioPluginInstanceE + 1382
21  pluginval                           0x000000010004ac56 _ZN11PluginTests8testTypeERKN4juce17PluginDescriptionE + 5446
22  pluginval                           0x00000001000494da _ZN11PluginTests7runTestEv + 1402
23  pluginval                           0x000000010059617e _ZN4juce8UnitTest11performTestEPNS_14UnitTestRunnerE + 206
24  pluginval                           0x000000010059766a _ZN4juce14UnitTestRunner8runTestsERKNS_5ArrayIPNS_8UnitTestENS_20DummyCriticalSectionELi0EEEx + 618
25  pluginval                           0x00000001000a1bf3 _Z8runTestsR11PluginTestsNSt3__18functionIFvRKN4juce6StringEEEE + 723
26  pluginval                           0x00000001000a14a2 _Z8validateRKN4juce17PluginDescriptionEN11PluginTests7OptionsENSt3__18functionIFvRKNS_6StringEEEE + 242
27  pluginval                           0x000000010009fe19 _ZN21ValidatorSlaveProcess14processRequestEN4juce11MemoryBlockE + 4521
28  pluginval                           0x000000010009e72b _ZN21ValidatorSlaveProcess15processRequestsEv + 1115
29  pluginval                           0x000000010009b73b _ZN21ValidatorSlaveProcess3runEv + 91
30  pluginval                           0x000000010009b8dd _ZThn24_N21ValidatorSlaveProcess3runEv + 45
31  pluginval                           0x0000000100588e6d _ZN4juce6Thread16threadEntryPointEv + 445
32  pluginval                           0x0000000100589599 _ZN4juce21juce_threadEntryPointEPv + 41
33  pluginval                           0x00000001005dd7ca _ZN4juceL15threadEntryProcEPv + 58
34  libclang_rt.tsan_osx_dynamic.dylib  0x0000000101fe430d __tsan_thread_start_func + 141
35  libsystem_pthread.dylib             0x00000001032ad665 _pthread_body + 340
36  libsystem_pthread.dylib             0x00000001032ad511 _pthread_body + 0
37  libsystem_pthread.dylib             0x00000001032acbfd thread_start + 13

These 4 calls here:

13  pluginval                           0x0000000100302ec3 _ZN4juce14AudioProcessor15processBypassedIfEEvRNS_11AudioBufferIT_EERNS_10MidiBufferE + 147
14  PFMCPP_Project10                    0x0000000108dfafa9 _ZN4juce14AudioProcessor20processBlockBypassedERNS_11AudioBufferIfEERNS_10MidiBufferE + 73
15  PFMCPP_Project10                    0x0000000108b93990 _ZN4juce17JuceVST3Component12processAudioIfEEvRN9Steinberg3Vst11ProcessDataERNS_5ArrayIPT_NS_20DummyCriticalSectionELi0EEE + 3440
16  PFMCPP_Project10                    0x0000000108b717c2 _ZN4juce17JuceVST3Component7processERN9Steinberg3Vst11ProcessDataE + 1602
17  PFMCPP_Project10                    0x0000000108b71c7d _ZThn8_N4juce17JuceVST3Component7processERN9Steinberg3Vst11ProcessDataE + 61

It seems to be calling:

juce::AudioProcessor::processBlockBypassed(AudioBuffer& MidiBuffer&)

I haven’t defined this function in my project, so it’s using the default implementation.

Any ideas how to figure this out? I turned on ASAN and TSAN, and the crash kept happening with no change to the crash report.

1 Like

Have you tried building pluginval from source and running the tests in debug?

yes, built it from source. But I have been using PluginVal as the executable to run when running the plugin itself. Are you saying to run PluginVal via the debugger?

Yeah, clone the repo and build then run the pluginval binary. Then you’ll catch the crash in the debugger and get a full stack trace.

This is detailed here: https://github.com/Tracktion/pluginval/blob/master/docs/Debugging%20a%20failed%20validation.md

Same crash with ASAN. Can you explain what this test does?

Starting test: pluginval / Background thread state...

*** FAILED: VALIDATION CRASHED

0   pluginval                           0x000000010130dcf8 _ZN4juce11SystemStats17getStackBacktraceEv + 424
1   pluginval                           0x0000000100039b22 _ZN12_GLOBAL__N_119getCrashLogContentsEv + 290
2   pluginval                           0x00000001000394af _ZN12_GLOBAL__N_111handleCrashEPv + 271
3   pluginval                           0x000000010130e458 _ZN4juceL11handleCrashEi + 24
4   libsystem_platform.dylib            0x00007fff666cbf5a _sigtramp + 26
5   pluginval                           0x0000000100221093 _ZN21ValidatorSlaveProcess14processRequestEN4juce11MemoryBlockE + 12307
6   pluginval                           0x000000010018d984 _ZNK4juce10OwnedArrayINS_14AudioProcessor3BusENS_20DummyCriticalSectionEEixEi + 420
7   pluginval                           0x000000010018cab2 _ZN4juce14AudioProcessor6getBusEbi + 242
8   pluginval                           0x0000000100190e21 _ZNK4juce14AudioProcessor6getBusEbi + 145
9   pluginval                           0x0000000100190ce1 _ZNK4juce14AudioProcessor20getChannelCountOfBusEbi + 145
10  pluginval                           0x000000010018c446 _ZNK4juce14AudioProcessor26getMainBusNumInputChannelsEv + 86
11  pluginval                           0x0000000100a5e526 _ZN4juce14AudioProcessor15processBypassedIfEEvRNS_11AudioBufferIT_EERNS_10MidiBufferE + 166
12  PFMCPP_Project10                    0x000000010e95bd85 _ZN4juce14AudioProcessor20processBlockBypassedERNS_11AudioBufferIfEERNS_10MidiBufferE + 37
13  PFMCPP_Project10                    0x000000010e58fb66 _ZN4juce17JuceVST3Component12processAudioIfEEvRN9Steinberg3Vst11ProcessDataERNS_5ArrayIPT_NS_20DummyCriticalSectionELi0EEE + 6870
14  PFMCPP_Project10                    0x000000010e55ba63 _ZN4juce17JuceVST3Component7processERN9Steinberg3Vst11ProcessDataE + 3123
15  pluginval                           0x0000000100db71a8 _ZN4juce18VST3PluginInstance12processAudioIfEEvRNS_11AudioBufferIT_EERNS_10MidiBufferEN9Steinberg3Vst19SymbolicSampleSizesEb + 3160
16  pluginval                           0x0000000100d81f85 _ZN4juce18VST3PluginInstance12processBlockERNS_11AudioBufferIfEERNS_10MidiBufferE + 581
17  pluginval                           0x000000010017ab9c _ZN25ParameterThreadSafetyTest7runTestER11PluginTestsRN4juce19AudioPluginInstanceE + 4572
18  pluginval                           0x00000001001019c3 _ZN11PluginTests8testTypeERKN4juce17PluginDescriptionE + 14675
19  pluginval                           0x00000001000fd98e _ZN11PluginTests7runTestEv + 3518
20  pluginval                           0x000000010138c8eb _ZN4juce8UnitTest11performTestEPNS_14UnitTestRunnerE + 443
21  pluginval                           0x000000010139055f _ZN4juce14UnitTestRunner8runTestsERKNS_5ArrayIPNS_8UnitTestENS_20DummyCriticalSectionELi0EEEx + 1823
22  pluginval                           0x0000000100226386 _Z8runTestsR11PluginTestsNSt3__18functionIFvRKN4juce6StringEEEE + 3702
23  pluginval                           0x0000000100224901 _Z8validateRKN4juce17PluginDescriptionEN11PluginTests7OptionsENSt3__18functionIFvRKNS_6StringEEEE + 817
24  pluginval                           0x0000000100221093 _ZN21ValidatorSlaveProcess14processRequestEN4juce11MemoryBlockE + 12307
25  pluginval                           0x000000010021c2ce _ZN21ValidatorSlaveProcess15processRequestsEv + 5582
26  pluginval                           0x000000010020ff03 _ZN21ValidatorSlaveProcess3runEv + 371
27  pluginval                           0x0000000101366517 _ZN4juce6Thread16threadEntryPointEv + 1527
28  pluginval                           0x00000001013674ff _ZN4juce21juce_threadEntryPointEPv + 79
29  pluginval                           0x00000001014803a6 _ZN4juceL15threadEntryProcEPv + 38
30  libsystem_pthread.dylib             0x00000001065cb665 _pthread_body + 340
31  libsystem_pthread.dylib             0x00000001065cb511 _pthread_body + 0
32  libsystem_pthread.dylib             0x00000001065cabfd thread_start + 13

Looks like it’s this test:

/** Sets plugin state from a background thread whilst the plugin window is
    created on the main thread. This simulates behaviour seen in certain hosts.
 */
struct BackgroundThreadStateTest    : public PluginTest
{
    BackgroundThreadStateTest()
        : PluginTest ("Background thread state", 7,
                      { Requirements::Thread::backgroundThread, Requirements::GUI::requiresGUI })
    {
    }

    void runTest (PluginTests& ut, AudioPluginInstance& instance) override
    {
        auto r = ut.getRandom();
        ScopedEditorShower editor (instance);

        auto parameters = getNonBypassAutomatableParameters (instance);
        MemoryBlock originalState;

        // Read state
        instance.getStateInformation (originalState);

        // Set random parameter values
        for (auto parameter : parameters)
            parameter->setValue (r.nextFloat());

        // Restore original state
        instance.setStateInformation (originalState.getData(), (int) originalState.getSize());

        // Allow for async reaction to state changes
        Thread::sleep (2000);
    }
};

My plugin has no parameters, and doesn’t save/restore state currently.

I can’t remember off the top of my head but it will be in the source if you grep for “Background thread state”.
Most of the tests are pretty minimal and easy to see what’s going on.

My guess would be that it is processing the plugin on one thread whilst setting parameters from another thread.

The crash could then be from a data race on some object you have?

Also, you might want to test with the “–validate-in-process” option to avoid any of the IPC stuff affecting it. You’ll also get a cleaner stack trace.

Alright, thanks. that turned out to be really helpful:

Somewhere further into that stack trace, the allocatedBytes for the buffer is 0.

And it looks to be in this function:

    template <typename FloatType>
    void processAudio (Vst::ProcessData& data, Array<FloatType*>& channelList)


Any ideas on this oddity?
Again, this isn’t my code. this is JUCE code causing the crash.

@dave96 do you recommend using the latest commit of PluginVal or the Master branch release from March 2019?

The latest tip of develop is usually the most up to date so I’d probably go with that.

The reason that hasn’t been merged to develop and a new release been has been because of notarisation and the ever changing requirements of that.

Now we’re through the final (hopefully) iteration of that I can get the builds notarised and tested to bring everything in line.

Btw, allocatedBytes in the snippet you showed is correct as it uses the setDataToReferTo method.

Do you have any ideas why, when I step into in the debugger after adding a breakpoint on that instance.processBlock(ab, mb); line in the first screenshot, if I continue stepping in, eventually the program kills the debugger, but continues running. I’m left with some rogue copies of PluginVal running that don’t show up in Activity Monitor, and can only be killed via the Force Quit popup.

That sounds like a broken pipe and then zombie process, possibly resulting from the cause of the crash.

Does this happen running with the —validate-in-process option I mentioned previously?
With that, you shouldn’t get multiple processes so it should be impossible to get these zombies if the app is still running and being debugged.

That’s actually the only place it happened. When I disabled that feature, it didn’t crash. Also, updating PluginVal’s JUCE Submodule to the latest tip of JUCE seems to have resolved the crash. I can’t get it to happen anymore. The project passed Level10 testing.

I was also doing std::vector<Atomic<float>> levels in my project, and levels.resize(getTotalNumInputChannels()) in prepareToPlay() but changed that to a fixed number of atomics, so maybe that was part of it. Who knows… I can’t get it to crash or kernel panic anymore as a result, which is good.

It was causing kernel panics?
That’s pretty extreme and usually only possible with driver level stuff (and very uncommon on macOS).

It could be a data race in the juce VST3 wrapper that has been fixed recently. I’ll update in pluginval soon.

Yeah. Happened 2 times. I think I was using an audio buffer inside a fifo from the GUI thread before it had been initialized in preparedToPlay. The kernel panic error log showed something along the lines of audio kit drivers exploding.

I’m still trying to figure out what actually caused it…

Looks like the Kernel Panic is still there. the log is not helpful:

*** Panic Report ***
panic(cpu 6 caller 0xffffff8011d887af): Kernel trap at 0xffffff8011b9305b, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0xffffff9144345000, CR3: 0x0000000015979000, CR4: 0x00000000001627e0
RAX: 0xffffff80e3a22fb8, RBX: 0xffffff9144325000, RCX: 0x00000000000053f7, RDX: 0x0000000000044000
RSP: 0xffffff9122333ce8, RBP: 0xffffff9122333d70, RSI: 0xffffff9144345000, RDI: 0xffffff80e3a3d000
R8:  0x0000000000008800, R9:  0xffffff8029f8e300, R10: 0xffffff7f92759090, R11: 0x0000000000000000
R12: 0xffffff8029f8e250, R13: 0x00000000000093f7, R14: 0xffffff8029f8e250, R15: 0x0000000000000bf7
RFL: 0x0000000000010206, RIP: 0xffffff8011b9305b, CS:  0x0000000000000008, SS:  0x0000000000000010
Fault CR2: 0xffffff9144345000, Error code: 0x0000000000000000, Fault CPU: 0x6, PL: 0, VF: 1

Backtrace (CPU 6), Frame : Return Address
0xffffff91223337b0 : 0xffffff8011c6ce46 
0xffffff9122333800 : 0xffffff8011d963b4 
0xffffff9122333840 : 0xffffff8011d88584 
0xffffff91223338b0 : 0xffffff8011c1ee60 
0xffffff91223338d0 : 0xffffff8011c6c8bc 
0xffffff9122333a00 : 0xffffff8011c6c67c 
0xffffff9122333a60 : 0xffffff8011d887af 
0xffffff9122333bd0 : 0xffffff8011c1ee60 
0xffffff9122333bf0 : 0xffffff8011b9305b 
0xffffff9122333d70 : 0xffffff7f92519d8f 
0xffffff9122333dc0 : 0xffffff7f92519b5c 
0xffffff9122333de0 : 0xffffff7f92519915 
0xffffff9122333e30 : 0xffffff7f9251465e 
0xffffff9122333ea0 : 0xffffff7f92510be3 
0xffffff9122333ed0 : 0xffffff8011ca6a24 
0xffffff9122333f40 : 0xffffff8011ca6585 
0xffffff9122333fa0 : 0xffffff8011c1e557 
      Kernel Extensions in backtrace:
         com.apple.iokit.IOAudioFamily(206.5)[2F090399-C028-3F14-AE4F-BF49A3D602E5]@0xffffff7f92506000->0xffffff7f92544fff
            dependency: com.apple.vecLib.kext(1.2.0)[F41EEA34-47C4-3ADE-BA77-D7A3CCA437B9]@0xffffff7f92451000

If anyone has ever experienced a kernel panic due to the iokit.IOAudioFamily extension, I’d love to learn more about your experience!