Coreaudio deadlock when stopping and restarting device

MacOSX and iOS
1

The deadlock happens on my mac mini M1, but not on my macbook 2016 running catalina. At some point the ‘callbackLock’ mutex of juce_mac_CoreAudio.cpp is locked by both the audioCallback() and the start() member functions, and none of them progresses anymore.

It seems that the audioCallback() is called while AudioDeviceStart() is being executed, and AudioDeviceStart() waits for audioCallback to complete, but audioCallback cannot complete because it can’t lock callbackLock…

Here is a trace of the last events before the deadlock:

locking audioCallback() ! thread=0x16bedf000 audioDeviceStopPending=0
lock acquired for audioCallback() !
audioCallback() finished!, unlocking

locking audioCallback! thread=0x16bedf000 audioDeviceStopPending=0
lock acquired for audioCallback()!
audioCallback ()finished!, unlocking

locking audioCallback()! thread=0x16bedf000 audioDeviceStopPending=0
lock acquired for audioCallback()!
audioCallback() finished!, unlocking

locking at stop(), leaveInterruptRunning=1
unlocking at stop()
locking at stop(), leaveInterruptRunning=0
 stop(): audioDeviceStopPending=1

locking audioCallback() ! thread=0x16bedf000 audioDeviceStopPending=1
lock acquired for audioCallback()!
returning after AudioDeviceStop() at audioCallback()
 stop(): audioDeviceStopPending=0
unlocking at stop()
locking at stop(), leaveInterruptRunning=0
unlocking at stop()
locking at updateDetailsFromDevice()
unlocking at updateDetailsFromDevice()
locking at stop(), leaveInterruptRunning=0
unlocking at stop()
locking at updateDetailsFromDevice()
unlocking at updateDetailsFromDevice()
locking at updateDetailsFromDevice()
unlocking at updateDetailsFromDevice()
locking at start() thread=0x1057d4580
 started=0

locking audioCallback() ! thread=0x16bedf000 audioDeviceStopPending=0

I have noted that the is always a call to audioCallback with “audioDeviceStopPending=1” just before the deadlock happens.

The relevant part of the stack traces is:

thread #1, name = 'SYNTH.APP', queue = 'com.apple.main-thread'
  * frame #0: 0x00000001a95445c0 libsystem_kernel.dylib`__psynch_mutexwait + 8
    frame #1: 0x00000001a957a364 libsystem_pthread.dylib`_pthread_mutex_firstfit_lock_wait + 84
    frame #2: 0x00000001a9577c98 libsystem_pthread.dylib`_pthread_mutex_firstfit_lock_slow + 240
    frame #3: 0x00000001a957d894 libsystem_pthread.dylib`_pthread_cond_wait + 1368
    frame #4: 0x00000001ab50e41c CoreAudio`HALB_Guard::WaitFor(unsigned long long) + 180
    frame #5: 0x00000001ab2dd308 CoreAudio`HALB_IOThread::_WaitForState(unsigned int) + 184
    frame #6: 0x00000001ab2dd68c CoreAudio`HALB_IOThread::StartAndWaitForState(unsigned int) + 292
    frame #7: 0x00000001ab115d24 CoreAudio`HALC_ProxyIOContext::_StartIO() + 288
    frame #8: 0x00000001ab0dfbcc CoreAudio`HAL_HardwarePlugIn_DeviceStart(AudioHardwarePlugInInterface**, unsigned int, int (*)(unsigned int, AudioTimeStamp const*, AudioBufferList const*, AudioTimeStamp const*, AudioBufferList*, AudioTimeStamp const*, void*)) + 808
    frame #9: 0x00000001ab5071f4 CoreAudio`HALDevice::StartIOProc(int (*)(unsigned int, AudioTimeStamp const*, AudioBufferList const*, AudioTimeStamp const*, AudioBufferList*, AudioTimeStamp const*, void*)) + 104
    frame #10: 0x00000001aaf536b4 CoreAudio`AudioDeviceStart + 524
    frame #11: 0x0000000100907480 juce::CoreAudioClasses::CoreAudioInternal::start(this=0x0000000107f0f550, callbackToNotify=<unavailable>) at juce_mac_CoreAudio.cpp:705:29 [opt]
    frame #12: 0x0000000100904ad0 juce::CoreAudioClasses::CoreAudioIODevice::start(this=0x0000600003e74400, callback=0x0000600000c499b0) at juce_mac_CoreAudio.cpp:1107:23 [opt]
    frame #13: 0x00000001008fedf0 juce::AudioDeviceManager::setAudioDeviceSetup(this=0x0000000102817c50, newSetup=0x000000016fdfe588, treatAsChosenDevice=true) at juce_AudioDeviceManager.cpp:722:29 [opt]
frame #14: 0x00000001008fdcc4 juce::AudioDeviceManager::initialiseFromXML(this=0x0000000102817c50, xml=0x0000600000eab180, selectDefaultDeviceOnFailure=false, preferredDefaultDeviceName=0x0000000102817d60, preferredSetupOptions=0x0000000000000000) at juce_AudioDeviceManager.cpp:428:13 [opt]


thread #14, name = 'com.apple.audio.IOThread.client'
    frame #0: 0x00000001a95445c0 libsystem_kernel.dylib`__psynch_mutexwait + 8
    frame #1: 0x00000001a957a364 libsystem_pthread.dylib`_pthread_mutex_firstfit_lock_wait + 84
    frame #2: 0x00000001a9577c98 libsystem_pthread.dylib`_pthread_mutex_firstfit_lock_slow + 240
    frame #3: 0x000000010082d1f0 juce::CriticalSection::enter(this=<unavailable>) const at juce_posix_SharedCode.h:39:55 [opt]
    frame #4: 0x0000000100907610 juce::CoreAudioClasses::CoreAudioInternal::audioCallback(this=0x0000000107f0f550, inInputData=0x0000600000071be0, outOutputData=0x00006000005c4a60) at juce_mac_CoreAudio.cpp:767:20 [opt]
    frame #5: 0x0000000100907558 juce::CoreAudioClasses::CoreAudioInternal::audioIOProc((null)=<unavailable>, (null)=<unavailable>, inInputData=<unavailable>, (null)=<unavailable>, outOutputData=<unavailable>, (null)=<unavailable>, device=<unavailable>) at juce_mac_CoreAudio.cpp:906:51 [opt]
    frame #6: 0x00000001ab113974 CoreAudio`HALC_ProxyIOContext::IOWorkLoop() + 6288
    frame #7: 0x00000001ab111ae0 CoreAudio`invocation function for block in HALC_ProxyIOContext::HALC_ProxyIOContext(unsigned int, unsigned int) + 100
    frame #8: 0x00000001ab2dd420 CoreAudio`HALB_IOThread::Entry(void*) + 88
    frame #9: 0x00000001a957d240 libsystem_pthread.dylib`_pthread_start + 148

Adding a ScopedUnlock unsl(callbackLock) in start(), just before the if (OK (AudioDeviceStart (deviceID, audioProcID))) call solves the issue, but I guess that is not a “good” fix…

2 Likes
2

Have you made any progress with this? I’m struggling with the same issue. One detail I noticed that it seems to only happen (or at least seems to be more likely) when I open and start AudioIODevice with a different sampling rate than it was before opening. I’m using JUCE 6.1.6, Intel Macbook Pro & Big Sur.

3

Nope no progress. Maybe the JUCE team will have something to say but I guess they are a bit overwhelmed right now. It would probably be useful if we were able to reproduce it on a simple example in the juce DemoRunner – I have not tried, so far.

1 Like
4

Welcome to JUCE :clap:

It looks like, for some reason, your running a lock on audioCallback() and despite your logs saying it unlocked, it didn’t.

I use the standard STL mutex_lock and mutex classes. Maybe they will help since they’re RAII? If you really need to lock audioCallback() twice in succession, STL has a recursive_lock that could be helpful.

5

I made a simple test project to demonstrate this issue. When sampling rate of internal speaker is first set to 44.1kHz in macOS Audio & MIDI setup, the app sometimes ends up in a deadlock as described in the original post when selecting internal speaker and AudioIODevice is started.

JUCE 7.0.1 & Big Sur

MainComponent.h:

#pragma once

#include <JuceHeader.h>

using namespace juce;
class MainComponent : public Component, public AudioIODeviceCallback
{
public:
    
    MainComponent();
    ~MainComponent() override;

    void paint (juce::Graphics& g) override;
    void resized() override;

    ComboBox audioSource;
    
    void audioDeviceIOCallback (const float** inputChannelData,
                                        int numInputChannels,
                                        float** outputChannelData,
                                        int numOutputChannels,
                                int numSamples) override;
    
    void audioDeviceAboutToStart (juce::AudioIODevice* device) override;
    void audioDeviceStopped() override;
    
private:
    
    OwnedArray<AudioIODeviceType> types;
    std::unique_ptr<AudioIODevice> device;
 
    JUCE_DECLARE_NON_COPYABLE_WITH_LEAK_DETECTOR (MainComponent)
};

MainComponent.cpp

#include "MainComponent.h"

using namespace juce;

MainComponent::MainComponent()
{

    setSize (800, 600);

    // Some platforms require permissions to open input channels so request that here
    if (juce::RuntimePermissions::isRequired (juce::RuntimePermissions::recordAudio)
        && ! juce::RuntimePermissions::isGranted (juce::RuntimePermissions::recordAudio))
    {
        juce::RuntimePermissions::request (juce::RuntimePermissions::recordAudio,
                                           [&] (bool granted) {  });
    }
    
    audioSource.setBounds(100, 100, 400, 50);
    addAndMakeVisible(audioSource);
    
    AudioDeviceManager deviceManager;
    deviceManager.createAudioDeviceTypes(types);
    types[0]->scanForDevices();
    
    StringArray deviceNames = types[0]->getDeviceNames(false);

    for (int j = 0; j < deviceNames.size(); ++j)
    {
        String devName = deviceNames [j];
        audioSource.addItem(devName, j+1);
    }
    
    audioSource.onChange = [this] {
    
        String name = audioSource.getItemText(audioSource.getSelectedItemIndex());
        device.reset(types[0]->createDevice(name, {}));
        
        juce::BigInteger bii;
        bii.setBit(0, true);
        bii.setBit(1, true);
        
        String error = device->open(0, bii, 48000, 64);
        DBG("Error = " << error);
        device->start(this);
        DBG("Started");
    };
    
}

MainComponent::~MainComponent() {}

void MainComponent::audioDeviceIOCallback (const float** inputChannelData,
                                    int numInputChannels,
                                    float** outputChannelData,
                                    int numOutputChannels,
                                           int numSamples) {}
void MainComponent::audioDeviceAboutToStart (juce::AudioIODevice* device) {}
void MainComponent::audioDeviceStopped() {}
void MainComponent::paint (juce::Graphics& g) {}
void MainComponent::resized() {}
2 Likes
6

We are also running into the exactly the same issue. Any updates on this?

2 Likes
7

After diving into this problem a bit more, I’d like to hear if others can confirm the following:

  • This problem only happens on Apple Silicon. Intel is not affected. (1)
  • It only happens when opening a device with a sample rate not equal to the current rate of the device.
  • In only happens with the internal audio device (Speakers and External headphones). Other devices (virtual and real) don’t seem to trigger this problem.

If anyone found anything in the above not true, please let me know.

I have been running a simple test application which re-creates and opens a device every second, while alternating between two sample rates (44.1K and 48K). With this I’m able to consistently reproduce well under 5 minutes.

void MainComponent::timerCallback() // Called every second
{
    auto deviceNames = mAudioIODeviceType->getDeviceNames();
    auto defaultIndex = mAudioIODeviceType->getDefaultDeviceIndex (false);

    if (mAudioIODevice)
        mAudioIODevice->stop();

    mAudioIODevice.reset (mAudioIODeviceType->createDevice (deviceNames[defaultIndex], {}));

    if (!mAudioIODevice)
        throw std::runtime_error ("Failed to create default device");

    mAudioIODevice->open (0, 3, mSampleRateToggle ? 44100.0 : 48000.0, 128);
    mAudioIODevice->start (this);

    mSampleRateToggle = !mSampleRateToggle; // When commenting this line out, the deadlock doesn't happen.
}

=========================================================================

  1. Since most Apple Silicon computers are newer than the Intel ones, and potentially have different hardware, the problem might nog be tied to Apple Silicon vs Intel but rather to newer/different hardware. Especially considering that this only seems to happen with the built-in audio device.
8

@jpo What exactly do you not find “good” about your ScopedUnlock proposal?

9

It’s just that I don’t know the consequences of removing that lock around AudioDeviceStart, maybe it just creates are race condition.

10

I have only seen it happening with internal devices and only in the case the current rate is not equal. But it does happen on my Intel Macbook.

11

That is interesting. What model (year) is your Intel MacBook?

12

Its MacBook Pro (13-inch, 2020, Four Thunderbolt 3 ports). macOS 11.6.1. For me with my test app, the deadlock occurs more or less every second time.

13

I asked about this problem on StackOverflow:

1 Like
14

I just opened a PR with a fix: Bugfix CoreAudio deadlock when opening a device by ruurdadema · Pull Request #1102 · juce-framework/JUCE · GitHub

1 Like
15

I’ve pushed a fix for this issue:

4 Likes
16

Thanks ! I can confirm that the issue is fixed for me.

18

Thanks, I can confirm the issue being fixed as well.

19

Is it possible to add this update to 6.1.6?

20

Bump. Is it possible to add this update to 6.1.x ?

21

Not under the terms of the v6 license. If you want to use this change, you’ll need to abide by the terms of the v7 license, and you’ll need to maintain your own patched version of v6.