Analyzing Windows Crash Dumps

Audio Plugins
1

Hi, currently I’m facing the issue that my VST3 plugin keeps crashing on a remote Windows 10 machine in Cubase 13, while I can not reproduce the crashes on my machine.

I am provided with crash dumps (user-mode minidump) which I started to analyze using WinDbg.
The !analyze -v command shows an access violation in an unknown module with the following stack trace:

...: 0x98b0af90
...: ntdll!RtlpTpTimerCallback+0x83
...: ntdll!TppTimerpExecuteCallback+0xb3
...: ntdll!TppWorkerThread+0x8f6
...: kernel32!BaseThreadInitThunk+0x1d
...: ntdll!RtlUserThreadStart+0x28

The command !findstack myPlugin shows that all plugin threads were idle at the time of the crash. It seems that the access violation happened on a different thread, however, I assume that the crashes are related to myPlugin since they have occurred since the plugin was installed.

I am wondering why the crashes occur on a different thread and how to continue debugging from this point.

Thanks in advance for any help!

2

Well if it’s definitely caused by your plugin, you’ve got two other options I can think of:

  • You’ve scribbled on someone elses memory
  • You’ve triggered a latent bug in the DAW.

Or … it’s not caused by your plugin but is triggered when your plugin is loaded. This much harder to figure out, but I once had a bug where someone elses antipiracy plugin was overwriting some of my plugin’s memory. I wasn’t happy :slight_smile:

Have you got all the symbols saved with symserve or something so you can get the best possible stack trace info out of the debugger. That 0x98b0af90 address isnt’ inside your plugins memory space is it? Can you figure that out from WinDbg?

I normally look at mini-dumps in Visual Studio (you can just double click on them) as I’m rubbish with WinDbg :slight_smile:

1 Like
3

I kept the .pdb files and loaded them into WinDbg. Symbolication seems to work since I can see the plugin’s symbols in other threads:

0:291> !findstack myPlugin 2
Thread 352, 6 frame(s) match
          00 000000016bf9fc60 00007ff9e9c796bb ntdll!NtWaitForAlertByThreadId+0x14
          01 000000016bf9fce0 00007ff9e72cf259 ntdll!RtlSleepConditionVariableSRW+0x13b
          02 000000016bf9fd20 00007ff94b18cd61 KERNELBASE!SleepConditionVariableSRW+0x29
        * 03 000000016bf9fd90 00007ff94abdf93e myPlugin!_Cnd_timedwait+0x7d
        * 04 000000016bf9fe20 00007ff94ac08d5c myPlugin!juce::WaitableEvent::wait+0x22e
        * 05 000000016bf9fe80 00007ff94abda64a myPlugin!juce::Timer::TimerThread::run+0xec
        * 06 000000016bf9fec0 00007ff94abda7fc myPlugin!juce::Thread::threadEntryPoint+0x6a
        * 07 000000016bf9fef0 00007ff94b2269aa myPlugin!juce::threadEntryProc+0x4c
        * 08 000000016bf9ff20 00007ff9e8e8257d myPlugin!thread_start<unsigned int (__cdecl*)(void *),1>+0x5a
          09 000000016bf9ff50 00007ff9e9c6af28 kernel32!BaseThreadInitThunk+0x1d
          10 000000016bf9ffd0 0000000000000000 ntdll!RtlUserThreadStart+0x28

Thread 353, 4 frame(s) match
          00 000000017dd5fd50 00007ff9e9c65683 ntdll!NtDelayExecution+0x14
          01 000000017dd5fd80 00007ff9e72aa03d ntdll!RtlDelayExecution+0x43
          02 000000017dd5fe00 00007ff94b14be6b KERNELBASE!SleepEx+0x7d
        * 03 000000017dd5fe80 00007ff94abda64a myPlugin!juce::dsp::BackgroundMessageQueue::run+0x6b
        * 04 000000017dd5fec0 00007ff94abda7fc myPlugin!juce::Thread::threadEntryPoint+0x6a
        * 05 000000017dd5fef0 00007ff94b2269aa myPlugin!juce::threadEntryProc+0x4c
        * 06 000000017dd5ff20 00007ff9e8e8257d myPlugin!thread_start<unsigned int (__cdecl*)(void *),1>+0x5a
          07 000000017dd5ff50 00007ff9e9c6af28 kernel32!BaseThreadInitThunk+0x1d
          08 000000017dd5ffd0 0000000000000000 ntdll!RtlUserThreadStart+0x28

Thread 354, 6 frame(s) match
          00 000000017fe7fca0 00007ff9e9c796bb ntdll!NtWaitForAlertByThreadId+0x14
          01 000000017fe7fd20 00007ff9e72cf259 ntdll!RtlSleepConditionVariableSRW+0x13b
          02 000000017fe7fd60 00007ff94b18ce0d KERNELBASE!SleepConditionVariableSRW+0x29
        * 03 000000017fe7fd90 00007ff94abdf7ad myPlugin!_Cnd_wait+0x25
        * 04 000000017fe7fe20 00007ff94b024dd7 myPlugin!juce::WaitableEvent::wait+0x9d
        * 05 000000017fe7fe80 00007ff94abda64a myPlugin!ConvolutionProcessor::ParameterUpdateThread::run+0x177
        * 06 000000017fe7fec0 00007ff94abda7fc myPlugin!juce::Thread::threadEntryPoint+0x6a
        * 07 000000017fe7fef0 00007ff94b2269aa myPlugin!juce::threadEntryProc+0x4c
        * 08 000000017fe7ff20 00007ff9e8e8257d myPlugin!thread_start<unsigned int (__cdecl*)(void *),1>+0x5a
          09 000000017fe7ff50 00007ff9e9c6af28 kernel32!BaseThreadInitThunk+0x1d
          10 000000017fe7ffd0 0000000000000000 ntdll!RtlUserThreadStart+0x28

Thread 355, 4 frame(s) match
          00 00000001887dfd50 00007ff9e9c65683 ntdll!NtDelayExecution+0x14
          01 00000001887dfd80 00007ff9e72aa03d ntdll!RtlDelayExecution+0x43
          02 00000001887dfe00 00007ff94b14be6b KERNELBASE!SleepEx+0x7d
        * 03 00000001887dfe80 00007ff94abda64a myPlugin!juce::dsp::BackgroundMessageQueue::run+0x6b
        * 04 00000001887dfec0 00007ff94abda7fc myPlugin!juce::Thread::threadEntryPoint+0x6a
        * 05 00000001887dfef0 00007ff94b2269aa myPlugin!juce::threadEntryProc+0x4c
        * 06 00000001887dff20 00007ff9e8e8257d myPlugin!thread_start<unsigned int (__cdecl*)(void *),1>+0x5a
          07 00000001887dff50 00007ff9e9c6af28 kernel32!BaseThreadInitThunk+0x1d
          08 00000001887dffd0 0000000000000000 ntdll!RtlUserThreadStart+0x28

Thread 356, 6 frame(s) match
          00 000000018beafca0 00007ff9e9c796bb ntdll!NtWaitForAlertByThreadId+0x14
          01 000000018beafd20 00007ff9e72cf259 ntdll!RtlSleepConditionVariableSRW+0x13b
          02 000000018beafd60 00007ff94b18ce0d KERNELBASE!SleepConditionVariableSRW+0x29
        * 03 000000018beafd90 00007ff94abdf7ad myPlugin!_Cnd_wait+0x25
        * 04 000000018beafe20 00007ff94b024dd7 myPlugin!juce::WaitableEvent::wait+0x9d
        * 05 000000018beafe80 00007ff94abda64a myPlugin!ConvolutionProcessor::ParameterUpdateThread::run+0x177
        * 06 000000018beafec0 00007ff94abda7fc myPlugin!juce::Thread::threadEntryPoint+0x6a
        * 07 000000018beafef0 00007ff94b2269aa myPlugin!juce::threadEntryProc+0x4c
        * 08 000000018beaff20 00007ff9e8e8257d myPlugin!thread_start<unsigned int (__cdecl*)(void *),1>+0x5a
          09 000000018beaff50 00007ff9e9c6af28 kernel32!BaseThreadInitThunk+0x1d
          10 000000018beaffd0 0000000000000000 ntdll!RtlUserThreadStart+0x28

Thread 357, 4 frame(s) match
          00 000000018e2ffd50 00007ff9e9c65683 ntdll!NtDelayExecution+0x14
          01 000000018e2ffd80 00007ff9e72aa03d ntdll!RtlDelayExecution+0x43
          02 000000018e2ffe00 00007ff94b14be6b KERNELBASE!SleepEx+0x7d
        * 03 000000018e2ffe80 00007ff94abda64a myPlugin!juce::dsp::BackgroundMessageQueue::run+0x6b
        * 04 000000018e2ffec0 00007ff94abda7fc myPlugin!juce::Thread::threadEntryPoint+0x6a
        * 05 000000018e2ffef0 00007ff94b2269aa myPlugin!juce::threadEntryProc+0x4c
        * 06 000000018e2fff20 00007ff9e8e8257d myPlugin!thread_start<unsigned int (__cdecl*)(void *),1>+0x5a
          07 000000018e2fff50 00007ff9e9c6af28 kernel32!BaseThreadInitThunk+0x1d
          08 000000018e2fffd0 0000000000000000 ntdll!RtlUserThreadStart+0x28

Thread 358, 6 frame(s) match
          00 000000018f46fca0 00007ff9e9c796bb ntdll!NtWaitForAlertByThreadId+0x14
          01 000000018f46fd20 00007ff9e72cf259 ntdll!RtlSleepConditionVariableSRW+0x13b
          02 000000018f46fd60 00007ff94b18ce0d KERNELBASE!SleepConditionVariableSRW+0x29
        * 03 000000018f46fd90 00007ff94abdf7ad myPlugin!_Cnd_wait+0x25
        * 04 000000018f46fe20 00007ff94b024dd7 myPlugin!juce::WaitableEvent::wait+0x9d
        * 05 000000018f46fe80 00007ff94abda64a myPlugin!ConvolutionProcessor::ParameterUpdateThread::run+0x177
        * 06 000000018f46fec0 00007ff94abda7fc myPlugin!juce::Thread::threadEntryPoint+0x6a
        * 07 000000018f46fef0 00007ff94b2269aa myPlugin!juce::threadEntryProc+0x4c
        * 08 000000018f46ff20 00007ff9e8e8257d myPlugin!thread_start<unsigned int (__cdecl*)(void *),1>+0x5a
          09 000000018f46ff50 00007ff9e9c6af28 kernel32!BaseThreadInitThunk+0x1d
          10 000000018f46ffd0 0000000000000000 ntdll!RtlUserThreadStart+0x28

Thread 382, 4 frame(s) match
          00 00000002c4e7fd50 00007ff9e9c65683 ntdll!NtDelayExecution+0x14
          01 00000002c4e7fd80 00007ff9e72aa03d ntdll!RtlDelayExecution+0x43
          02 00000002c4e7fe00 00007ff94b14be6b KERNELBASE!SleepEx+0x7d
        * 03 00000002c4e7fe80 00007ff94abda64a myPlugin!juce::dsp::BackgroundMessageQueue::run+0x6b
        * 04 00000002c4e7fec0 00007ff94abda7fc myPlugin!juce::Thread::threadEntryPoint+0x6a
        * 05 00000002c4e7fef0 00007ff94b2269aa myPlugin!juce::threadEntryProc+0x4c
        * 06 00000002c4e7ff20 00007ff9e8e8257d myPlugin!thread_start<unsigned int (__cdecl*)(void *),1>+0x5a
          07 00000002c4e7ff50 00007ff9e9c6af28 kernel32!BaseThreadInitThunk+0x1d
          08 00000002c4e7ffd0 0000000000000000 ntdll!RtlUserThreadStart+0x28

Thread 383, 6 frame(s) match
          00 00000002c4f7fca0 00007ff9e9c796bb ntdll!NtWaitForAlertByThreadId+0x14
          01 00000002c4f7fd20 00007ff9e72cf259 ntdll!RtlSleepConditionVariableSRW+0x13b
          02 00000002c4f7fd60 00007ff94b18ce0d KERNELBASE!SleepConditionVariableSRW+0x29
        * 03 00000002c4f7fd90 00007ff94abdf7ad myPlugin!_Cnd_wait+0x25
        * 04 00000002c4f7fe20 00007ff94b024dd7 myPlugin!juce::WaitableEvent::wait+0x9d
        * 05 00000002c4f7fe80 00007ff94abda64a myPlugin!ConvolutionProcessor::ParameterUpdateThread::run+0x177
        * 06 00000002c4f7fec0 00007ff94abda7fc myPlugin!juce::Thread::threadEntryPoint+0x6a
        * 07 00000002c4f7fef0 00007ff94b2269aa myPlugin!juce::threadEntryProc+0x4c
        * 08 00000002c4f7ff20 00007ff9e8e8257d myPlugin!thread_start<unsigned int (__cdecl*)(void *),1>+0x5a
          09 00000002c4f7ff50 00007ff9e9c6af28 kernel32!BaseThreadInitThunk+0x1d
          10 00000002c4f7ffd0 0000000000000000 ntdll!RtlUserThreadStart+0x28

Thread 384, 4 frame(s) match
          00 00000002c507fd50 00007ff9e9c65683 ntdll!NtDelayExecution+0x14
          01 00000002c507fd80 00007ff9e72aa03d ntdll!RtlDelayExecution+0x43
          02 00000002c507fe00 00007ff94b14be6b KERNELBASE!SleepEx+0x7d
        * 03 00000002c507fe80 00007ff94abda64a myPlugin!juce::dsp::BackgroundMessageQueue::run+0x6b
        * 04 00000002c507fec0 00007ff94abda7fc myPlugin!juce::Thread::threadEntryPoint+0x6a
        * 05 00000002c507fef0 00007ff94b2269aa myPlugin!juce::threadEntryProc+0x4c
        * 06 00000002c507ff20 00007ff9e8e8257d myPlugin!thread_start<unsigned int (__cdecl*)(void *),1>+0x5a
          07 00000002c507ff50 00007ff9e9c6af28 kernel32!BaseThreadInitThunk+0x1d
          08 00000002c507ffd0 0000000000000000 ntdll!RtlUserThreadStart+0x28

Thread 385, 6 frame(s) match
          00 00000002f824fca0 00007ff9e9c796bb ntdll!NtWaitForAlertByThreadId+0x14
          01 00000002f824fd20 00007ff9e72cf259 ntdll!RtlSleepConditionVariableSRW+0x13b
          02 00000002f824fd60 00007ff94b18ce0d KERNELBASE!SleepConditionVariableSRW+0x29
        * 03 00000002f824fd90 00007ff94abdf7ad myPlugin!_Cnd_wait+0x25
        * 04 00000002f824fe20 00007ff94b024dd7 myPlugin!juce::WaitableEvent::wait+0x9d
        * 05 00000002f824fe80 00007ff94abda64a myPlugin!ConvolutionProcessor::ParameterUpdateThread::run+0x177
        * 06 00000002f824fec0 00007ff94abda7fc myPlugin!juce::Thread::threadEntryPoint+0x6a
        * 07 00000002f824fef0 00007ff94b2269aa myPlugin!juce::threadEntryProc+0x4c
        * 08 00000002f824ff20 00007ff9e8e8257d myPlugin!thread_start<unsigned int (__cdecl*)(void *),1>+0x5a
          09 00000002f824ff50 00007ff9e9c6af28 kernel32!BaseThreadInitThunk+0x1d
          10 00000002f824ffd0 0000000000000000 ntdll!RtlUserThreadStart+0x28

Thread 386, 4 frame(s) match
          00 000000030575fd50 00007ff9e9c65683 ntdll!NtDelayExecution+0x14
          01 000000030575fd80 00007ff9e72aa03d ntdll!RtlDelayExecution+0x43
          02 000000030575fe00 00007ff94b14be6b KERNELBASE!SleepEx+0x7d
        * 03 000000030575fe80 00007ff94abda64a myPlugin!juce::dsp::BackgroundMessageQueue::run+0x6b
        * 04 000000030575fec0 00007ff94abda7fc myPlugin!juce::Thread::threadEntryPoint+0x6a
        * 05 000000030575fef0 00007ff94b2269aa myPlugin!juce::threadEntryProc+0x4c
        * 06 000000030575ff20 00007ff9e8e8257d myPlugin!thread_start<unsigned int (__cdecl*)(void *),1>+0x5a
          07 000000030575ff50 00007ff9e9c6af28 kernel32!BaseThreadInitThunk+0x1d
          08 000000030575ffd0 0000000000000000 ntdll!RtlUserThreadStart+0x28

Thread 387, 6 frame(s) match
          00 000000034a8ffca0 00007ff9e9c796bb ntdll!NtWaitForAlertByThreadId+0x14
          01 000000034a8ffd20 00007ff9e72cf259 ntdll!RtlSleepConditionVariableSRW+0x13b
          02 000000034a8ffd60 00007ff94b18ce0d KERNELBASE!SleepConditionVariableSRW+0x29
        * 03 000000034a8ffd90 00007ff94abdf7ad myPlugin!_Cnd_wait+0x25
        * 04 000000034a8ffe20 00007ff94b024dd7 myPlugin!juce::WaitableEvent::wait+0x9d
        * 05 000000034a8ffe80 00007ff94abda64a myPlugin!ConvolutionProcessor::ParameterUpdateThread::run+0x177
        * 06 000000034a8ffec0 00007ff94abda7fc myPlugin!juce::Thread::threadEntryPoint+0x6a
        * 07 000000034a8ffef0 00007ff94b2269aa myPlugin!juce::threadEntryProc+0x4c
        * 08 000000034a8fff20 00007ff9e8e8257d myPlugin!thread_start<unsigned int (__cdecl*)(void *),1>+0x5a
          09 000000034a8fff50 00007ff9e9c6af28 kernel32!BaseThreadInitThunk+0x1d
          10 000000034a8fffd0 0000000000000000 ntdll!RtlUserThreadStart+0x28

That seems like a reasonable explaination. However there is no way to confirm that this is the cause of the crash, right?

I’m doing manual memory management at some places, I will try to get rid of that and check if the crashes still occur.

Thank you!

1 Like
4

The Microsoft Address Sanitiser could be useful here:

2 Likes
5

Yeah, we run all our debug builds with this on all the time pretty much (on Mac anyway…). … it’s very good.

2 Likes