JUCE plug-ins crashing under Logic Pro 10.8.1

Logic Pro 10.8.1 was released yesterday 30th Nov 2023 and sadly I’ve witnessed many crashes of JUCE-based AUs under this new version. I have reported this to Apple.

In most cases, the crashed thread is “JUCE IPC”. I’m seeing this mostly in Solid State Logic’s plug-ins.

I’m not a dev, so I’m sorry I don’t have more details, happy to share crash logs if that’s useful to folks here.

1 Like

On my studio mac now, so I can paste in some crash details…

Process:               AUHostingServiceXPC_arrow [5402]
Path:                  /System/Library/Frameworks/AudioToolbox.framework/XPCServices/AUHostingServiceXPC_arrow.xpc/Contents/MacOS/AUHostingServiceXPC_arrow
Identifier:            com.apple.audio.AUHostingService.arm64e
Version:               1.0 (1)
Build Info:            CoreAudioServices_AUHostingServiceXPC_arrow-1319099000000000~3
Code Type:             ARM-64 (Native)
Parent Process:        launchd [1]
Responsible:           Logic Pro X [5397]
User ID:               501

Date/Time:             2023-12-01 11:01:38.1953 -0800
OS Version:            macOS 13.6.1 (22G313)
Report Version:        12
Anonymous UUID:        868B4C85-10A5-EEB0-7077-69531270AC44


Time Awake Since Boot: 43000 seconds

System Integrity Protection: enabled

Crashed Thread:        5  JUCE Timer

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000000
Exception Codes:       0x0000000000000001, 0x0000000000000000

Termination Reason:    Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process:   exc handler [5402]

…

Thread 5 Crashed:: JUCE Timer
0   libsystem_pthread.dylib       	       0x1aa162558 pthread_mutex_lock + 12
1   SSL Native Channel Strip 2    	       0x111478860 JuceAUFactory + 8758352
2   SSL Native Channel Strip 2    	       0x11147b040 JuceAUFactory + 8768560
3   SSL Native Channel Strip 2    	       0x111442694 JuceAUFactory + 8536708
4   libsystem_pthread.dylib       	       0x1aa167fa8 _pthread_start + 148
5   libsystem_pthread.dylib       	       0x1aa162da0 thread_start + 8

Thread 5 crashed with ARM Thread State (64-bit):
    x0: 0x0000000000000000   x1: 0x000000013381b660   x2: 0x0000000000000101   x3: 0x0000000000119a01
    x4: 0x0011990000119a00   x5: 0x0000000000119a01   x6: 0x0000000000000000   x7: 0x0000000011e19ec5
    x8: 0x00000001124ab000   x9: 0x00000f0000000f02  x10: 0x00000f0000000f00  x11: 0x00000f0000000f02
   x12: 0x00000f0000000f02  x13: 0x00000f0000000f00  x14: 0x0000000000000100  x15: 0x0000000000000000
   x16: 0x00000001aa16254c  x17: 0x0000000209d18c88  x18: 0x0000000000000000  x19: 0x0000000000000000
   x20: 0x000000013381b660  x21: 0x000000013381ba28  x22: 0x000000013381b910  x23: 0x00000001124ab910
   x24: 0x00000001124ab000  x25: 0x0000000111f4d000  x26: 0x00000001124ab8a0  x27: 0x0000000002966f98
   x28: 0x0000000002966f98   fp: 0x000000016c07aef0   lr: 0xaf27000111478860
    sp: 0x000000016c07aef0   pc: 0x00000001aa162558 cpsr: 0x60001000
   far: 0x0000000000000000  esr: 0x92000006 (Data Abort) byte read Translation fault

It’s probably a good idea to report this to SSL (and any other JUCE plugin manufacturers that you experience crashes with). They’ll be able to run this up with a debugger attached or produce debug versions that will add more detail to those crash reports. Unfortunately I can’t quite tell what is going wrong from that crash report. We’ll probably take a look at our demo plugins in the latest Logic version early next week. Are you doing anything when, or shortly before, the crash happens? or do you just need to leave the plugin running?

Thanks for the quick reply! I have already reported this to both Eventide (emote) and SSL (Native Channel Strip 2) Support.

The crashes happen spontaneously, generally within 15 minutes if leaving a session open with e.g. 16 tracks each with a single plug-in (SSL Channel Strip 2 for example).

I have also reported this to Apple and have sent a few screen recordings to them as I had a case open with with their Pro Apps support folks regarding some MIDI issues with 10.8.

1 Like

I’m starting to get reports of random notes and crashes in Logic 10.8.1 from our MIDI effect plugin.

Previously it was only Apple Silicon machines affected but now it seems to be Intel machines too.

I’ve updated Logic this morning and tried running up some of our example plugins with no crashes yet unfortunately. I’ll keep an eye out. Also JUCE IPC was mentioned and we don’t have any interprocess connections in any of our examples so that could be a factor.

macOS Ventura 13.6.3 was just released today (11th Dec 2023) and since AUHostingServiceXPC_arrow is part of the AudioToolbox.framework, I was hopeful Apple would have found an issue and fixed it in an OS software release (as opposed to a Logic Pro update).

A quick test of my Logic Pro document which reliably crashed 100% of the time under Logic Pro 10.8.1 on macOS 13.6.1 shows promise — no crash after 30 minutes of letting my test project sit idle!

I thought it might be a good idea to grab some snapshots of the old AUHostingServiceXPC_arrow binary, and this confirmed that the binary has at least changed since I ran the OS update:

Before:

-rwxr-xr-x 1 root wheel 90640 Oct 12 04:10 /System/Library/Frameworks/AudioToolbox.framework/XPCServices/AUHostingServiceXPC_arrow.xpc/Contents/MacOS/AUHostingServiceXPC_arrow*

Sha1 Sum: 1cab85254098af44813ef4396006492dd1043d61 /System/Library/Frameworks/AudioToolbox.framework/XPCServices/AUHostingServiceXPC_arrow.xpc/Contents/MacOS/AUHostingServiceXPC_arrow

After:

-rwxr-xr-x 1 root wheel 90640 Dec 1 23:00 /System/Library/Frameworks/AudioToolbox.framework/XPCServices/AUHostingServiceXPC_arrow.xpc/Contents/MacOS/AUHostingServiceXPC_arrow*

Sha1 Sum: bbfb6f71ee1e1dc12de4150dbee8af6b2708fe54 /System/Library/Frameworks/AudioToolbox.framework/XPCServices/AUHostingServiceXPC_arrow.xpc/Contents/MacOS/AUHostingServiceXPC_arrow

Hopefully this OS update solves the crashing for everyone!

2 Likes

Well I guess I was overly hopeful… still seeing crashes in JUCE AudioUnits.

Thread 71 Crashed:: JUCE Timer
0   libsystem_pthread.dylib       	       0x1a5f06558 pthread_mutex_lock + 12
1   SSL 4K E                      	       0x170f62fa8 JuceAUFactory + 9028800
2   SSL 4K E                      	       0x170f65788 JuceAUFactory + 9039008
3   SSL 4K E                      	       0x170f24990 JuceAUFactory + 8773288
4   libsystem_pthread.dylib       	       0x1a5f0bfa8 _pthread_start + 148
5   libsystem_pthread.dylib       	       0x1a5f06da0 thread_start + 8

(other crash)

Thread 69 Crashed:: JUCE IPC
0   libsystem_pthread.dylib       	       0x1a5f06558 pthread_mutex_lock + 12
1   SSL 4K E                      	       0x171532fa8 JuceAUFactory + 9028800
2   SSL 4K E                      	       0x1715364bc JuceAUFactory + 9042388
3   SSL 4K E                      	       0x1714f4990 JuceAUFactory + 8773288
4   libsystem_pthread.dylib       	       0x1a5f0bfa8 _pthread_start + 148
5   libsystem_pthread.dylib       	       0x1a5f06da0 thread_start + 8

(other crash)

Thread 81 Crashed:
0   libsystem_pthread.dylib       	       0x1a5f06558 pthread_mutex_lock + 12
1   Emote                         	       0x12f7b9e4c juce::MessageQueue::post(juce::MessageManager::MessageBase*) + 28
2   Emote                         	       0x12f750d94 state_impl::client_controller_t::do_select(base::event_dispatcher_t*, unsigned int, bool) + 2052
3   Emote                         	       0x12f752fcc state_impl::controller_t::runnable_main() + 80
4   Emote                         	       0x12f753168 base::runnable_t::__run(base::runnable_t*) + 52
5   Emote                         	       0x12f7534d8 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (*)(base::runnable_t*), state_impl::controller_t*>>(void*) + 56
6   libsystem_pthread.dylib       	       0x1a5f0bfa8 _pthread_start + 148
7   libsystem_pthread.dylib       	       0x1a5f06da0 thread_start + 8

IME when I see a bad access on a pthread_mutex_lock (which I assume it what is happening here), it’s normally that the object being accessed, so the Timer, InterprocessConnection, and MessageQueue in this case, have been deleted (probably from another thread in this instance), then the lock either tries to lock/unlock and kaboom it’s a bad access.

I recently came across this in the Timer (although I couldn’t reproduce it) and we pushed some fixes to hopefully resolve that one.

Looking again at the InterprocessConnection I think I can see something there too so I’ll push something for that.

In regards to the third one, that thread isn’t started and managed by JUCE so I think the thread created in that plugin is probably outliving the MessageQueue, so some other object on the main thread probably needs to notify the thread to stop and then wait for it to stop to ensure that never happens.

I guess what’s interesting about these crashes is that I can only imagine them happening when the plugin is being destroyed. Maybe you could share the whole crash logs with me? we might well see that on another thread.

If you’re just running these plugins in the background I wonder if Logic is doing something that FinalCut has done for years. Spawning up instances to process small chunks of audio in parallel. In FinalCut they do this in order to generate the waveforms. Maybe they are doing that in Logic now? If I’m right it’s worth noting that even touching a parameter or changing some automation for a plugin on the same channel as the plugin in question could trigger the issue. So you may not always being doing something with the plugin in questions but if your doing anything in Logic it would be useful to know.

1 Like

I’ve looked at this a little more and the IPC thread does the same thing the Timer thread was doing, it gives it 4 seconds to stop the thread and if that fails it tries to forcibly kill the thread. The problem with trying to forcibly kill a thread is that it’s really just a request and I have no idea how much more code the thread might try to run after that point. I’m assuming in all these cases a lock just holds for much longer than expected. In the case of the IPC thread I’m also a little more concerned that it could actually get stuck so I’ll need to dive into that, but that’s not what I find interesting.

@reFX very kindly shared some crash reports they were seeing come up relatively regularly with their customers in the TimerThread that now I look back seem remarkably similar. What’s interesting is that almost all of the crashes shared with me were in Logic Pro X or MainStage 3 running on Apple Silicon (I’ve double checked and two were in FL Studio).

I can’t help but think these issues are general issues but for some reason they are more likely to occur in Logic for one reason or another.

I’m going to let my brain linger on the issue for a little longer.

One thing we could maybe try is creating a plugin we share with you, (@ddv) as you seem to be able to reproduce this issue relatively easily, that has all the symbols and some better logging so hopefully we can see what’s going on. I probably won’t be able to get to that today unfortunately though.

2 Likes

I just tried opening another case with Apple “pro apps” support, however this time they were far less cooperative. They were quick to blame “third-party” plug-ins and refused to accept any log/crash files.

Is it worth using my (#dayjob) Enterprise Support route to try to get this to Apple engineering, or is this, as the annoying tech support person said, “typical scenario where plug-in developers fail to update their apps in a timely manner when macOS updates are released.”? — I responded aghast with “there are beta programs which these developers use to prevent this kind of problem.”

What a joke Apple support was today. I’m still all worked up! If you think it’s worth escalating with Apple please let me know. I’m happy to use whatever tools I have to get this fixed.

To be honest I’m not surprised.

I think we need more data and a smoking gun before pointing all the blame at Apple.

Apple may well be doing something that makes this more likely to occur but arguably it does seem like this is likely a problem both in JUCE and our customers code that should be fixed.

1 Like

Thank you for your help, @anthony-nicholls and for setting my expectations :smile:

I appreciate any assistance, whenever time permits, and in the meantime I am working with SSL and Eventide to provide reproducible test-cases for the crashes I’m seeing. Interestingly (and not surprisingly), SSL has said they have yet to reproduce this under macOS 13.5.1 (M1) and Logic 10.8.1 (I’m using macOS 13.6.1, and as of Monday, 13.6.3)

I also have this problem and a lot of crash reports. I’ve seen it with a lot of different plugins and almost always when logic is sitting idle for a few minutes.

All Apple Silicon AUs are run within the AUHostingServiceXPC_arrow process (or AUHostingServiceXPC if they are Intel AUs). This sadly can mean: if a user has many AUs installed and any one of them smashes the memory within the process, any other AU can go down. The exact cause of that can not be found via a crashlog.

We’ve seen these “unexplainable” crashes a lot in the past within Logic Pro. Since we forced all AUs out-of-process when releasing the Apple Silicon version, all these unexplainable crashes are gone and we are left with our own crashes. I wrote scripts to parse thousands of crashlogs and find patterns, sometimes you are lucky and find a match: your code crashes, but in every single instance a specific AU was also used.

Said that: this obviously does not have the be the case in this situation. Another extremely common issue with crashes happen during the instantiation or the destruction of AUs. Typically race conditions of e.g. a timer firing “too early” or “on last time” after the destructor was already done. Xcode has debug options to find these race conditions – and it was horrible the first time I used them. Took a while to fix all bugs.

1 Like

Yes, this. I can reliably spin-lock LOGIC (not the AUhostingService) by opening a project with 16 SSL channel strip 2 AUs and deleting tracks from logic (not from mixer, from track list). I have sent this to Solid State Logic support and they can reproduce this, FWIW.

I guess I’ll go bug my vendors again as this sh*t is getting really old.