[BUG] ASIO driver crash when enumerating devices with insertDefaultDeviceNames

[andrewj]

JUCE version: 8.0.12
Platform: Windows (x64)
Affected file: juce_ASIO_windows.cpp

Summary

When insertDefaultDeviceNames is called with the ASIO device type active, the application crashes with a 0xC0000005 access violation inside the ASIO driver DLL. In my case the crashing driver is asiobtin.dll (Steinberg built-in ASIO Driver), and I only notice it in debug builds. However, this could potentially affect any ASIO driver that spawns internal threads during start().

Steps to Reproduce

I believe the bug is reproducible whenever insertDefaultDeviceNames iterates past the Steinberg built-in ASIO Driver to probe the next device.

• Have multiple ASIO drivers installed (I have four), with one being the Steinberg built-in ASIO Driver
• Build DemoRunner in debug and open Settings (or the Audio Settings demo)
• Switch from Windows Audio to ASIO then choose an ASIO driver which isn’t the Steinberg built-in driver
• Switch back to Windows Audio, then close the app
• Reopen the app and then switch back to ASIO
• initialise >> insertDefaultDeviceNames is called to find a valid device pair

Root Cause

insertDefaultDeviceNames calls type->createDevice(...) to construct a temporary ASIOAudioIODevice for each installed ASIO driver. During construction, openDevice() calls start() and stop() on the device - but it doesn’t call disposeBuffers(). Note that disposeBuffers() is guarded by the buffersCreated flag, which is only set to true in open(), not in createDummyBuffers().

See at juce_ASIO_Windows.cpp line 1271:

createDummyBuffers (preferredBufferSize);
// ...
err = asioObject->start();
Thread::sleep (80);
asioObject->stop();

When the temporary device is destroyed, the destructor calls removeCurrentDriver() >> asioObject->Release() without ever having called disposeBuffers() (as required by the ASIO SDK). Because disposeBuffers() is skipped, the driver’s internal audio thread (spawned during start()) is still running when Release() frees the COM object. That thread then dereferences internal state that has just been freed, leading to a null pointer crash inside the driver DLL.

Proposed Fix

In openDevice(), call asioObject->disposeBuffers() directly after asioObject->stop():

// start and stop because cubase does it
err = asioObject->start();
// ignore an error here, as it might start later after setting other stuff up
JUCE_ASIO_LOG_ERROR ("start", err);

Thread::sleep (80);
asioObject->stop();
Thread::sleep (10);                  // add: allow stop to settle (10ms is just a guess, but it works for me)
asioObject->disposeBuffers();        // add: complete the required SDK teardown sequence

[andrewj]

Just as a follow-up, I believe I have seen instances of this in Release builds too.

[reuk]

Thanks for the report. So far I haven’t been able to reproduce this crash. My system has the following ASIO drivers installed:

• Ableton Move
• Ableton Push
• ASIO4ALL v2
• FL Studio ASIO
• MT ASIO Bridge
• ReaRoute ASIO (x64)
• Steinberg built-in ASIO driver
• WASM

My Steinberg ASIO driver version is 1.0.9, which appears to be the latest available.

@andrewj

Because disposeBuffers() is skipped, the driver’s internal audio thread (spawned during start()) is still running when Release() frees the COM object.

I’m not convinced by this. It looks like we’re still calling stop() before Release(), and I would expect stop() to synchronously join any internal thread so that a call to disposeBuffers() can be made safely. Therefore, I believe that the problem is likely to be elsewhere, but I’m not sure where yet.

The docs say this about ASIOStop():

On return from ASIOStop(), the driver must not call the hosts bufferSwitch() routine. On a preemptive multitasking OS, you must make sure that no pending events will call bufferSwitch() after the driver returned from this function.

I interpret this to mean that, if a driver’s audio thread is continuing after stop() returns, this indicates a bug in the driver.

It would be helpful if you could provide the version of the Steinberg ASIO driver you have installed. If it’s older than 1.0.9, please try updating and check whether the problem persists.

It would also be helpful to see a stack trace at the point of the crash, along with the name of the crashed thread.

Finally, please could you try enabling the preprocessor flag JUCE_ASIO_DEBUGGING=1, and provide us with the console output that’s produced when you execute the repro steps that you provided.

[andrewj]

Hi @reuk, thanks for looking at this.

I’m using Steinberg built-in ASIO Driver v1.0.9.26 x64. The other drivers I have installed are ASIO Fireface, Maschine MK3, Yamaha Steinberg USB ASIO.

Something else that might be pertinent for the repro is that before I switch from ASIO to Windows Audio, I select the Yamaha ASIO device that happens to be the default device for Windows Audio. Another thing I noticed is that often I can repro three times in a row, then it’s ok on the fourth.

I see a few variations of the stack trace:

Exception thrown at 0x00007FF83F4AC801 (ntdll.dll) in DemoRunner.exe: 0xC0000005: Access violation reading location 0x0000000000000010.
>	ntdll.dll!RtlpEnterCriticalSectionContended()	Unknown
 	ntdll.dll!RtlEnterCriticalSection()	Unknown
 	asiobtin.dll!00000001800081aa()	Unknown
 	kernel32.dll!00007ff83e0be8d7()	Unknown
 	ntdll.dll!RtlUserThreadStart()	Unknown

Exception thrown at 0x0000000180008196 (asiobtin.dll) in DemoRunner.exe: 0xC0000005: Access violation reading location 0x0000000000000068.
>	asiobtin.dll!0000000180008196()	Unknown
 	kernel32.dll!00007ff83e0be8d7()	Unknown
 	ntdll.dll!RtlUserThreadStart()	Unknown

Exception thrown at 0x00007FF83F4AC801 (ntdll.dll) in DemoRunner.exe: 0xC0000005: Access violation reading location 0x0000000000000010.
>	ntdll.dll!RtlpWaitOnCriticalSection()	Unknown
 	ntdll.dll!RtlpEnterCriticalSectionContended()	Unknown
 	ntdll.dll!RtlEnterCriticalSection()	Unknown
 	asiobtin.dll!00000001800081aa()	Unknown
 	kernel32.dll!00007ff83e0be8d7()	Unknown
 	ntdll.dll!RtlUserThreadStart()	Unknown

When I set JUCE_ASIO_DEBUGGING=1 then the crash happens far less frequently. However, I eventually managed to capture one:

<snip>
ASIO: found ASIO Fireface
ASIO: found Maschine MK3
ASIO: found Steinberg built-in ASIO Driver
ASIO: found Yamaha Steinberg USB ASIO
ASIO: found ASIO Fireface
ASIO: found Maschine MK3
ASIO: found Steinberg built-in ASIO Driver
ASIO: found Yamaha Steinberg USB ASIO
<snip>
The thread 'JUCE v8.0.12: WASAPI' (16576) has exited with code 0 (0x0).
ASIO: opening device: ASIO Fireface
'DemoRunner.exe' (Win32): Loaded 'C:\Windows\System32\fireface_asio_64.dll'. Symbol loading disabled by Include/Exclude setting.
'DemoRunner.exe' (Win32): Loaded 'C:\Windows\System32\vcruntime140.dll'. Symbol loading disabled by Include/Exclude setting.
ASIO: closed
ASIO: opening device: ASIO Fireface
ASIO: closed
ASIO: opening device: Maschine MK3
'DemoRunner.exe' (Win32): Loaded 'C:\Program Files\Native Instruments\Maschine MK3 Driver\asio\nimc3asio64.dll'. Symbol loading disabled by Include/Exclude setting.
'DemoRunner.exe' (Win32): Loaded 'C:\Windows\System32\msvcp140.dll'. Symbol loading disabled by Include/Exclude setting.
'DemoRunner.exe' (Win32): Loaded 'C:\Windows\System32\hid.dll'. Symbol loading disabled by Include/Exclude setting.
'DemoRunner.exe' (Win32): Loaded 'C:\Windows\System32\vcruntime140_1.dll'. Symbol loading disabled by Include/Exclude setting.
'DemoRunner.exe' (Win32): Loaded 'C:\Windows\System32\dbgcore.dll'. Symbol loading disabled by Include/Exclude setting.
ASIO: closed
ASIO: opening device: Steinberg built-in ASIO Driver
'DemoRunner.exe' (Win32): Loaded 'C:\Program Files\Steinberg\Asio\asiobtin.dll'. Symbol loading disabled by Include/Exclude setting.
ASIO: 1 in, 2 out
ASIO: 480->480, 480, 480
ASIO: outputReady true
ASIO: Rates: 8000 12000 16000 24000 32000 44100 48000 64000 88200 96000 128000 176400 192000 256000 352800 384000 512000 705600 768000
ASIO: Latencies: in = 480, out = 960
ASIO: creating buffers (dummy): 3, 480
'DemoRunner.exe' (Win32): Loaded 'C:\Program Files\Steinberg\Asio\soxr.dll'. Symbol loading disabled by Include/Exclude setting.
ASIO: Latencies: in = 480, out = 960
ASIO: device open
ASIO: closed
ASIO: opening device: Yamaha Steinberg USB ASIO
<snip>
'DemoRunner.exe' (Win32): Loaded 'C:\Program Files (x86)\Yamaha\Yamaha Steinberg USB Driver\ysusb_asio64.dll'. Symbol loading disabled by Include/Exclude setting.
'DemoRunner.exe' (Win32): Unloaded 'C:\Program Files (x86)\Yamaha\Yamaha Steinberg USB Driver\ysusb_asio64.dll'
'DemoRunner.exe' (Win32): Loaded 'C:\Program Files (x86)\Yamaha\Yamaha Steinberg USB Driver\ysusb_asio64.dll'. Symbol loading disabled by Include/Exclude setting.
Exception thrown at 0x0000000180008196 (asiobtin.dll) in DemoRunner.exe: 0xC0000005: Access violation reading location 0x0000000000000068.

I understand what you’re saying about it being safe to call disposeBuffers() after stop(). However, the point is that when scanning the ASIO drivers there is no call to disposeBuffers(). If I add this after the stop() call, then the crash seems to be fixed. Fyi, I tried this without the additional sleep() and it was OK.

Here is the debug output with the patch:

ASIO: found ASIO Fireface
ASIO: found Maschine MK3
ASIO: found Steinberg built-in ASIO Driver
ASIO: found Yamaha Steinberg USB ASIO
ASIO: found ASIO Fireface
ASIO: found Maschine MK3
ASIO: found Steinberg built-in ASIO Driver
ASIO: found Yamaha Steinberg USB ASIO
<snip>
ASIO: opening device: ASIO Fireface
'DemoRunner.exe' (Win32): Loaded 'C:\Windows\System32\fireface_asio_64.dll'. Symbol loading disabled by Include/Exclude setting.
'DemoRunner.exe' (Win32): Loaded 'C:\Windows\System32\vcruntime140.dll'. Symbol loading disabled by Include/Exclude setting.
ASIO: closed
ASIO: opening device: ASIO Fireface
ASIO: closed
ASIO: opening device: Maschine MK3
'DemoRunner.exe' (Win32): Loaded 'C:\Program Files\Native Instruments\Maschine MK3 Driver\asio\nimc3asio64.dll'. Symbol loading disabled by Include/Exclude setting.
<snip>
ASIO: closed
ASIO: opening device: Steinberg built-in ASIO Driver
'DemoRunner.exe' (Win32): Loaded 'C:\Program Files\Steinberg\Asio\asiobtin.dll'. Symbol loading disabled by Include/Exclude setting.
ASIO: 1 in, 2 out
ASIO: 480->480, 480, 480
ASIO: outputReady true
ASIO: Rates: 8000 12000 16000 24000 32000 44100 48000 64000 88200 96000 128000 176400 192000 256000 352800 384000 512000 705600 768000
ASIO: Latencies: in = 480, out = 960
ASIO: creating buffers (dummy): 3, 480
'DemoRunner.exe' (Win32): Loaded 'C:\Program Files\Steinberg\Asio\soxr.dll'. Symbol loading disabled by Include/Exclude setting.
ASIO: Latencies: in = 480, out = 960
The thread 21180 has exited with code 1 (0x1).
'DemoRunner.exe' (Win32): Unloaded 'C:\Program Files\Steinberg\Asio\soxr.dll'
ASIO: device open
ASIO: closed
ASIO: opening device: Yamaha Steinberg USB ASIO
<snip>
'DemoRunner.exe' (Win32): Loaded 'C:\Program Files (x86)\Yamaha\Yamaha Steinberg USB Driver\ysusb_asio64.dll'. Symbol loading disabled by Include/Exclude setting.
'DemoRunner.exe' (Win32): Unloaded 'C:\Program Files (x86)\Yamaha\Yamaha Steinberg USB Driver\ysusb_asio64.dll'
'DemoRunner.exe' (Win32): Loaded 'C:\Program Files (x86)\Yamaha\Yamaha Steinberg USB Driver\ysusb_asio64.dll'. Symbol loading disabled by Include/Exclude setting.
ASIO: 2 in, 2 out
ASIO: 64->64, 64, 0
ASIO: outputReady true
ASIO: Rates: 44100 48000 88200 96000 176400 192000
ASIO: Latencies: in = 184, out = 232
ASIO: creating buffers (dummy): 4, 64
ASIO: Latencies: in = 184, out = 232
The thread 12972 has exited with code 0 (0x0).
ASIO: device open
ASIO: closed
The thread 34092 has exited with code 0 (0x0).
ASIO: opening device: Maschine MK3
ASIO: closed
ASIO: opening device: Steinberg built-in ASIO Driver
ASIO: 1 in, 2 out
ASIO: 480->480, 480, 480
ASIO: outputReady true
ASIO: Rates: 8000 12000 16000 24000 32000 44100 48000 64000 88200 96000 128000 176400 192000 256000 352800 384000 512000 705600 768000
ASIO: Latencies: in = 480, out = 960
ASIO: creating buffers (dummy): 3, 480
'DemoRunner.exe' (Win32): Loaded 'C:\Program Files\Steinberg\Asio\soxr.dll'. Symbol loading disabled by Include/Exclude setting.
ASIO: Latencies: in = 480, out = 960
The thread 8376 has exited with code 1 (0x1).
'DemoRunner.exe' (Win32): Unloaded 'C:\Program Files\Steinberg\Asio\soxr.dll'
ASIO: device open
ASIO: closed
ASIO: opening device: Steinberg built-in ASIO Driver
ASIO: 1 in, 2 out
ASIO: 480->480, 480, 480
ASIO: outputReady true
ASIO: Rates: 8000 12000 16000 24000 32000 44100 48000 64000 88200 96000 128000 176400 192000 256000 352800 384000 512000 705600 768000
ASIO: Latencies: in = 480, out = 960
ASIO: creating buffers (dummy): 3, 480
'DemoRunner.exe' (Win32): Loaded 'C:\Program Files\Steinberg\Asio\soxr.dll'. Symbol loading disabled by Include/Exclude setting.
ASIO: Latencies: in = 480, out = 960
The thread 7984 has exited with code 1 (0x1).
'DemoRunner.exe' (Win32): Unloaded 'C:\Program Files\Steinberg\Asio\soxr.dll'
ASIO: device open
ASIO: clock: Internal (cur)
ASIO: disposing buffers
ASIO: creating buffers: 3, 480
'DemoRunner.exe' (Win32): Loaded 'C:\Program Files\Steinberg\Asio\soxr.dll'. Symbol loading disabled by Include/Exclude setting.
ASIO: channel format: 19
ASIO: Latencies: in = 480, out = 960
ASIO: 480->480, 480, 480
ASIO: starting
ASIO: found ASIO Fireface
ASIO: found Maschine MK3
ASIO: found Steinberg built-in ASIO Driver
ASIO: found Yamaha Steinberg USB ASIO

[reuk]

Thanks for the stack traces. As you said, it definitely looks like the crash is happening in the Steinberg driver, and the crashes on RtlpEnterCriticalSectionContended might indicate a thread trying to lock a mutex that’s already been destroyed.

I suppose we could add in a call to disposeBuffers(), but it would be nice to know whether this is really a robust fix, or whether it just happens to avoid the problem due to adding more work, and therefore a brief delay, on the main thread. In the latter case, maybe the issue will still be present under other circumstances (e.g. different system load).

@andrewj

I select the Yamaha ASIO device that happens to be the default device for Windows Audio

I tried installing this ASIO driver, but it won’t start for me since I don’t have a Yamaha USB device. What version of the Yamaha Steinberg USB Driver do you have installed? Do you see the same behaviour on 2.1.9? I also wonder whether the crash goes away if you disconnect the Yamaha USB device from the computer and/or switch the device used by Windows Audio.

It might be worth reporting the issue to Steinberg. Presumably they’d be able to debug the driver itself, and might be able to tell us whether the problem is due to a bug in the driver, or misuse of the ASIO API by JUCE.

[andrewj]

Turns out I was using the 2.1.7 Yamaha driver, however upgrading to 2.1.9 hasn’t resolved the issue. It seemed better at first, but after half a dozen attempts it started crashing again. I was also able to see the crash with the USB interface removed.

Again, adding the disposeBuffers() call rectified the issue. Noting your speculation about timing, I tried adding a 1000msec sleep instead of disposeBuffers(), but that didn’t fix the issue.

Exception thrown at 0x00007FF83F4AAA73 (ntdll.dll) in DemoRunner.exe: 0xC0000005: Access violation writing location 0x0000000000000024.
>	ntdll.dll!RtlpWaitOnCriticalSection()	Unknown
 	ntdll.dll!RtlpEnterCriticalSectionContended()	Unknown
 	ntdll.dll!RtlEnterCriticalSection()	Unknown
 	asiobtin.dll!00000001800081aa()	Unknown
 	kernel32.dll!00007ff83e0be8d7()	Unknown
 	ntdll.dll!RtlUserThreadStart()	Unknown

It’s weird but at times it can take over a dozen attempts to get the crash to occur. Once it does, it tends to keep happening for a while.

I also tried switching Windows Audio to other devices and was still able to reproduce the crash.

Incidentally, with the USB interface removed I was seeing a handled exception on asioObject->Release() for the Yamaha driver. After continuing I was able to see the original crash. For posterity, here’s the Release() exception:

Exception thrown at 0x00007FF83F605BFA (ntdll.dll) in DemoRunner.exe: 0xC0000008: An invalid handle was specified.
 	ntdll.dll!KiRaiseUserExceptionDispatcher()	Unknown
 	KernelBase.dll!00007ff83c5f5a09()	Unknown
 	ysusb_asio64.dll!0000020bcac71eef()	Unknown
 	ysusb_asio64.dll!0000020bcac71c84()	Unknown
 	ysusb_asio64.dll!0000020bcac74b16()	Unknown
>	DemoRunner.exe!juce::ASIOAudioIODevice::removeCurrentDriver() Line 1119	C++
 	DemoRunner.exe!juce::ASIOAudioIODevice::openDevice() Line 1303	C++
 	DemoRunner.exe!juce::ASIOAudioIODevice::ASIOAudioIODevice(juce::ASIOAudioIODeviceType * ownerType, const juce::String & devName, _GUID clsID, int slotNumber) Line 346	C++
 	DemoRunner.exe!juce::ASIOAudioIODeviceType::createDevice(const juce::String & outputDeviceName, const juce::String & inputDeviceName) Line 1561	C++
 	DemoRunner.exe!juce::AudioDeviceManager::insertDefaultDeviceNames::__l5::juce::Array<double,juce::DummyCriticalSection,0> <lambda>(juce::AudioDeviceManager::insertDefaultDeviceNames::__l2::Direction, const juce::String &)::__l2::<lambda>() Line 632	C++
 	DemoRunner.exe!juce::AudioDeviceManager::insertDefaultDeviceNames::__l5::<lambda>(juce::AudioDeviceManager::insertDefaultDeviceNames::__l2::Direction dir, const juce::String & deviceName) Line 624	C++
 	DemoRunner.exe!juce::AudioDeviceManager::insertDefaultDeviceNames::__l5::<lambda>(const juce::String & outputDeviceName, const juce::String & inputDeviceName) Line 648	C++
 	DemoRunner.exe!juce::AudioDeviceManager::insertDefaultDeviceNames(juce::AudioDeviceManager::AudioDeviceSetup & setup) Line 676	C++
 	DemoRunner.exe!juce::AudioDeviceManager::setCurrentAudioDeviceType(const juce::String & type, bool treatAsChosenDevice) Line 767	C++
 	DemoRunner.exe!juce::AudioDeviceSelectorComponent::updateDeviceType() Line 1206	C++
 	DemoRunner.exe!juce::AudioDeviceSelectorComponent::{ctor}::__l23::<lambda>() Line 1094	C++
 	[External Code]	
 	DemoRunner.exe!juce::NullCheckedInvocation::invoke<std::function<void __cdecl(void)> &>(std::function<void __cdecl(void)> & fn) Line 73	C++
 	DemoRunner.exe!juce::ComboBox::handleAsyncUpdate() Line 643	C++
 	DemoRunner.exe!juce::AsyncUpdater::AsyncUpdaterMessage::messageCallback() Line 46	C++
 	DemoRunner.exe!juce::InternalMessageQueue::dispatchMessage(juce::MessageManager::MessageBase * message) Line 203	C++
 	DemoRunner.exe!juce::InternalMessageQueue::dispatchMessages() Line 245	C++
 	DemoRunner.exe!juce::InternalMessageQueue::dispatchNextMessage(bool returnIfNoPendingMessages) Line 130	C++
 	DemoRunner.exe!juce::detail::dispatchNextMessageOnSystemQueue(bool returnIfNoPendingMessages) Line 272	C++
 	DemoRunner.exe!juce::MessageManager::runDispatchLoop() Line 124	C++
 	DemoRunner.exe!juce::JUCEApplicationBase::main() Line 277	C++
 	DemoRunner.exe!WinMain(HINSTANCE__ * __formal, HINSTANCE__ * __formal, char * __formal, int __formal) Line 191	C++
 	[External Code]	

[andrewj]

Hi @reuk, has this one stalled out?

[reuk]

I’m not sure there’s much we can do on our side. I’m reluctant to make changes that we can’t verify, and so far we haven’t been able to reproduce the crash here, or find documentation indicating that JUCE’s implementation is incorrect. Your proposal might be a fix for the problem, but it also might just be obscuring the issue, or moving it around. Given that the crash is inside the ASIO driver itself, and assuming that JUCE is obeying the ASIO API contracts, that would indicate a bug in the driver, so the issue should be reported to the driver manufacturer.

[andrewj]

Hi @reuk, I understand your reticence if you can’t reproduce it. However, I dispute the assumption that JUCE is obeying the ASIO API contracts in this case.

My reading of the state machine in the ASIO spec is that after calling ASIOStop(), you need to call ASIODisposeBuffers() to revert to INITIALIZED state, then ASIOExit() to revert to LOADED. However, the description for ASIOExit() says that it implies ASIOStop() and ASIODisposeBuffers() - so it should be OK to call ASIOExit() without calling ASIODisposeBuffers().

In its destructor, juce::ASIOAudioDevice calls close() and removeCurrentDriver(). close() calls ASIOStop() and ASIODisposeBuffers(). removeCurrentDriver() calls Release() on the COM object. All good.

In the buffer size checking code in openDevices(), a temporary device is created but after ASIOStop() is called it goes straight to Release() on the COM object. There is no call to either ASIODisposeBuffers() or ASIOExit() - hence the ASIO API contracts are not being obeyed.

Note that line 1301 does call disposeBuffers(), but this is guarded by buffersCreated which hasn’t been set to true for the dummy buffers. This suggests an alternate fix: you could set buffersCreated = true in createDummyBuffers(). I haven’t looked at the code to see if that might have side effects.

Why doesn’t this cause more exceptions? I guess most ASIO drivers clean themselves up nicely when COM Release() is called. Perhaps this is an edge case that some drivers handle poorly (and intermittently!). I’ll see if I can get a response from Steinberg on their built-in driver.

EDIT: Possible bug in built-in ASIO driver - Developer / ASIO - Steinberg Forums