Logic Pro 10.8.1 was released yesterday 30th Nov 2023 and sadly I’ve witnessed many crashes of JUCE-based AUs under this new version. I have reported this to Apple.
In most cases, the crashed thread is “JUCE IPC”. I’m seeing this mostly in Solid State Logic’s plug-ins.
I’m not a dev, so I’m sorry I don’t have more details, happy to share crash logs if that’s useful to folks here.
On my studio mac now, so I can paste in some crash details…
Process: AUHostingServiceXPC_arrow [5402]
Path: /System/Library/Frameworks/AudioToolbox.framework/XPCServices/AUHostingServiceXPC_arrow.xpc/Contents/MacOS/AUHostingServiceXPC_arrow
Identifier: com.apple.audio.AUHostingService.arm64e
Version: 1.0 (1)
Build Info: CoreAudioServices_AUHostingServiceXPC_arrow-1319099000000000~3
Code Type: ARM-64 (Native)
Parent Process: launchd [1]
Responsible: Logic Pro X [5397]
User ID: 501
Date/Time: 2023-12-01 11:01:38.1953 -0800
OS Version: macOS 13.6.1 (22G313)
Report Version: 12
Anonymous UUID: 868B4C85-10A5-EEB0-7077-69531270AC44
Time Awake Since Boot: 43000 seconds
System Integrity Protection: enabled
Crashed Thread: 5 JUCE Timer
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000000
Exception Codes: 0x0000000000000001, 0x0000000000000000
Termination Reason: Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process: exc handler [5402]
…
Thread 5 Crashed:: JUCE Timer
0 libsystem_pthread.dylib 0x1aa162558 pthread_mutex_lock + 12
1 SSL Native Channel Strip 2 0x111478860 JuceAUFactory + 8758352
2 SSL Native Channel Strip 2 0x11147b040 JuceAUFactory + 8768560
3 SSL Native Channel Strip 2 0x111442694 JuceAUFactory + 8536708
4 libsystem_pthread.dylib 0x1aa167fa8 _pthread_start + 148
5 libsystem_pthread.dylib 0x1aa162da0 thread_start + 8
Thread 5 crashed with ARM Thread State (64-bit):
x0: 0x0000000000000000 x1: 0x000000013381b660 x2: 0x0000000000000101 x3: 0x0000000000119a01
x4: 0x0011990000119a00 x5: 0x0000000000119a01 x6: 0x0000000000000000 x7: 0x0000000011e19ec5
x8: 0x00000001124ab000 x9: 0x00000f0000000f02 x10: 0x00000f0000000f00 x11: 0x00000f0000000f02
x12: 0x00000f0000000f02 x13: 0x00000f0000000f00 x14: 0x0000000000000100 x15: 0x0000000000000000
x16: 0x00000001aa16254c x17: 0x0000000209d18c88 x18: 0x0000000000000000 x19: 0x0000000000000000
x20: 0x000000013381b660 x21: 0x000000013381ba28 x22: 0x000000013381b910 x23: 0x00000001124ab910
x24: 0x00000001124ab000 x25: 0x0000000111f4d000 x26: 0x00000001124ab8a0 x27: 0x0000000002966f98
x28: 0x0000000002966f98 fp: 0x000000016c07aef0 lr: 0xaf27000111478860
sp: 0x000000016c07aef0 pc: 0x00000001aa162558 cpsr: 0x60001000
far: 0x0000000000000000 esr: 0x92000006 (Data Abort) byte read Translation fault
It’s probably a good idea to report this to SSL (and any other JUCE plugin manufacturers that you experience crashes with). They’ll be able to run this up with a debugger attached or produce debug versions that will add more detail to those crash reports. Unfortunately I can’t quite tell what is going wrong from that crash report. We’ll probably take a look at our demo plugins in the latest Logic version early next week. Are you doing anything when, or shortly before, the crash happens? or do you just need to leave the plugin running?
Thanks for the quick reply! I have already reported this to both Eventide (emote) and SSL (Native Channel Strip 2) Support.
The crashes happen spontaneously, generally within 15 minutes if leaving a session open with e.g. 16 tracks each with a single plug-in (SSL Channel Strip 2 for example).
I have also reported this to Apple and have sent a few screen recordings to them as I had a case open with with their Pro Apps support folks regarding some MIDI issues with 10.8.
I’m starting to get reports of random notes and crashes in Logic 10.8.1 from our MIDI effect plugin.
Previously it was only Apple Silicon machines affected but now it seems to be Intel machines too.
I’ve updated Logic this morning and tried running up some of our example plugins with no crashes yet unfortunately. I’ll keep an eye out. Also JUCE IPC was mentioned and we don’t have any interprocess connections in any of our examples so that could be a factor.
macOS Ventura 13.6.3 was just released today (11th Dec 2023) and since AUHostingServiceXPC_arrow is part of the AudioToolbox.framework, I was hopeful Apple would have found an issue and fixed it in an OS software release (as opposed to a Logic Pro update).
A quick test of my Logic Pro document which reliably crashed 100% of the time under Logic Pro 10.8.1 on macOS 13.6.1 shows promise — no crash after 30 minutes of letting my test project sit idle!
I thought it might be a good idea to grab some snapshots of the old AUHostingServiceXPC_arrow binary, and this confirmed that the binary has at least changed since I ran the OS update:
Before:
-rwxr-xr-x 1 root wheel 90640 Oct 12 04:10 /System/Library/Frameworks/AudioToolbox.framework/XPCServices/AUHostingServiceXPC_arrow.xpc/Contents/MacOS/AUHostingServiceXPC_arrow*
Sha1 Sum: 1cab85254098af44813ef4396006492dd1043d61 /System/Library/Frameworks/AudioToolbox.framework/XPCServices/AUHostingServiceXPC_arrow.xpc/Contents/MacOS/AUHostingServiceXPC_arrow
After:
-rwxr-xr-x 1 root wheel 90640 Dec 1 23:00 /System/Library/Frameworks/AudioToolbox.framework/XPCServices/AUHostingServiceXPC_arrow.xpc/Contents/MacOS/AUHostingServiceXPC_arrow*
Sha1 Sum: bbfb6f71ee1e1dc12de4150dbee8af6b2708fe54 /System/Library/Frameworks/AudioToolbox.framework/XPCServices/AUHostingServiceXPC_arrow.xpc/Contents/MacOS/AUHostingServiceXPC_arrow
Hopefully this OS update solves the crashing for everyone!
Well I guess I was overly hopeful… still seeing crashes in JUCE AudioUnits.
Thread 71 Crashed:: JUCE Timer
0 libsystem_pthread.dylib 0x1a5f06558 pthread_mutex_lock + 12
1 SSL 4K E 0x170f62fa8 JuceAUFactory + 9028800
2 SSL 4K E 0x170f65788 JuceAUFactory + 9039008
3 SSL 4K E 0x170f24990 JuceAUFactory + 8773288
4 libsystem_pthread.dylib 0x1a5f0bfa8 _pthread_start + 148
5 libsystem_pthread.dylib 0x1a5f06da0 thread_start + 8
(other crash)
Thread 69 Crashed:: JUCE IPC
0 libsystem_pthread.dylib 0x1a5f06558 pthread_mutex_lock + 12
1 SSL 4K E 0x171532fa8 JuceAUFactory + 9028800
2 SSL 4K E 0x1715364bc JuceAUFactory + 9042388
3 SSL 4K E 0x1714f4990 JuceAUFactory + 8773288
4 libsystem_pthread.dylib 0x1a5f0bfa8 _pthread_start + 148
5 libsystem_pthread.dylib 0x1a5f06da0 thread_start + 8
(other crash)
Thread 81 Crashed:
0 libsystem_pthread.dylib 0x1a5f06558 pthread_mutex_lock + 12
1 Emote 0x12f7b9e4c juce::MessageQueue::post(juce::MessageManager::MessageBase*) + 28
2 Emote 0x12f750d94 state_impl::client_controller_t::do_select(base::event_dispatcher_t*, unsigned int, bool) + 2052
3 Emote 0x12f752fcc state_impl::controller_t::runnable_main() + 80
4 Emote 0x12f753168 base::runnable_t::__run(base::runnable_t*) + 52
5 Emote 0x12f7534d8 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (*)(base::runnable_t*), state_impl::controller_t*>>(void*) + 56
6 libsystem_pthread.dylib 0x1a5f0bfa8 _pthread_start + 148
7 libsystem_pthread.dylib 0x1a5f06da0 thread_start + 8
IME when I see a bad access on a pthread_mutex_lock (which I assume it what is happening here), it’s normally that the object being accessed, so the Timer, InterprocessConnection, and MessageQueue in this case, have been deleted (probably from another thread in this instance), then the lock either tries to lock/unlock and kaboom it’s a bad access.
I recently came across this in the Timer (although I couldn’t reproduce it) and we pushed some fixes to hopefully resolve that one.
Looking again at the InterprocessConnection I think I can see something there too so I’ll push something for that.
In regards to the third one, that thread isn’t started and managed by JUCE so I think the thread created in that plugin is probably outliving the MessageQueue, so some other object on the main thread probably needs to notify the thread to stop and then wait for it to stop to ensure that never happens.
I guess what’s interesting about these crashes is that I can only imagine them happening when the plugin is being destroyed. Maybe you could share the whole crash logs with me? we might well see that on another thread.
If you’re just running these plugins in the background I wonder if Logic is doing something that FinalCut has done for years. Spawning up instances to process small chunks of audio in parallel. In FinalCut they do this in order to generate the waveforms. Maybe they are doing that in Logic now? If I’m right it’s worth noting that even touching a parameter or changing some automation for a plugin on the same channel as the plugin in question could trigger the issue. So you may not always being doing something with the plugin in questions but if your doing anything in Logic it would be useful to know.
I’ve looked at this a little more and the IPC thread does the same thing the Timer thread was doing, it gives it 4 seconds to stop the thread and if that fails it tries to forcibly kill the thread. The problem with trying to forcibly kill a thread is that it’s really just a request and I have no idea how much more code the thread might try to run after that point. I’m assuming in all these cases a lock just holds for much longer than expected. In the case of the IPC thread I’m also a little more concerned that it could actually get stuck so I’ll need to dive into that, but that’s not what I find interesting.
@Toddler-Boy very kindly shared some crash reports they were seeing come up relatively regularly with their customers in the TimerThread that now I look back seem remarkably similar. What’s interesting is that almost all of the crashes shared with me were in Logic Pro X or MainStage 3 running on Apple Silicon (I’ve double checked and two were in FL Studio).
I can’t help but think these issues are general issues but for some reason they are more likely to occur in Logic for one reason or another.
I’m going to let my brain linger on the issue for a little longer.
One thing we could maybe try is creating a plugin we share with you, (@ddv) as you seem to be able to reproduce this issue relatively easily, that has all the symbols and some better logging so hopefully we can see what’s going on. I probably won’t be able to get to that today unfortunately though.
I just tried opening another case with Apple “pro apps” support, however this time they were far less cooperative. They were quick to blame “third-party” plug-ins and refused to accept any log/crash files.
Is it worth using my (#dayjob) Enterprise Support route to try to get this to Apple engineering, or is this, as the annoying tech support person said, “typical scenario where plug-in developers fail to update their apps in a timely manner when macOS updates are released.”? — I responded aghast with “there are beta programs which these developers use to prevent this kind of problem.”
What a joke Apple support was today. I’m still all worked up! If you think it’s worth escalating with Apple please let me know. I’m happy to use whatever tools I have to get this fixed.
To be honest I’m not surprised.
I think we need more data and a smoking gun before pointing all the blame at Apple.
Apple may well be doing something that makes this more likely to occur but arguably it does seem like this is likely a problem both in JUCE and our customers code that should be fixed.
Thank you for your help, @anthony-nicholls and for setting my expectations 
I appreciate any assistance, whenever time permits, and in the meantime I am working with SSL and Eventide to provide reproducible test-cases for the crashes I’m seeing. Interestingly (and not surprisingly), SSL has said they have yet to reproduce this under macOS 13.5.1 (M1) and Logic 10.8.1 (I’m using macOS 13.6.1, and as of Monday, 13.6.3)
I also have this problem and a lot of crash reports. I’ve seen it with a lot of different plugins and almost always when logic is sitting idle for a few minutes.
All Apple Silicon AUs are run within the AUHostingServiceXPC_arrow process (or AUHostingServiceXPC if they are Intel AUs). This sadly can mean: if a user has many AUs installed and any one of them smashes the memory within the process, any other AU can go down. The exact cause of that can not be found via a crashlog.
We’ve seen these “unexplainable” crashes a lot in the past within Logic Pro. Since we forced all AUs out-of-process when releasing the Apple Silicon version, all these unexplainable crashes are gone and we are left with our own crashes. I wrote scripts to parse thousands of crashlogs and find patterns, sometimes you are lucky and find a match: your code crashes, but in every single instance a specific AU was also used.
Said that: this obviously does not have the be the case in this situation. Another extremely common issue with crashes happen during the instantiation or the destruction of AUs. Typically race conditions of e.g. a timer firing “too early” or “on last time” after the destructor was already done. Xcode has debug options to find these race conditions – and it was horrible the first time I used them. Took a while to fix all bugs.
Yes, this. I can reliably spin-lock LOGIC (not the AUhostingService) by opening a project with 16 SSL channel strip 2 AUs and deleting tracks from logic (not from mixer, from track list). I have sent this to Solid State Logic support and they can reproduce this, FWIW.
I guess I’ll go bug my vendors again as this sh*t is getting really old.
I encounter this problem multiple times a day, across a wide variety of AU plugins (today alone it has happened using Toontrack EZ Keys 2, Output Arcade, Eventide Recirculate, and Karanyi Cloudmax). For me, it does not happen while the Mac is idling; it happens when I click to take some action, usually I/O related. I get the spinning beachball, other processes (like trying to save a Sample report from Activity Monitor) are blocked, but if I wait several minutes (yes, minutes, not seconds), the beach ball goes away and I can pick up where I left off.
In each case, the crashed thread is JUCE Timer, with the following info:
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000000
Exception Codes: 0x0000000000000001, 0x0000000000000000
Termination Reason: Namespace SIGNAL, Code 11 Segmentation fault: 11
and the offending command always seems to be this:
0 libsystem_pthread.dylib 0x18ac30600 pthread_mutex_lock + 12
Not being a Mac developer, I am not sure how best to seek some resolution. It seems the problem is in JUCE code, not Logic or the individual AU plugins’ code, no? And notifying the dozens of individual developers whose plugins suffer also seems relatively pointless since they are powerless to fix the code in JUCE libraries, no? Apologies if I’ve misunderstood the situation, and thanks for any help anyone can offer.
Any more word on this? Is the only solution to disable all plugins that use JUCE?
Well I decided to test this again and dug a little deeper in this time, and I think I have found something like a smoking gun.
@anthony-nicholls mentioned that something is likely destroying the plugin without notifying the main thread (?)
Regardless, whilst observing this message appearing: An Audio Unit plug-in reported a problem which might cause the system to become unstable. Please quit and restart Logic Pro., I also saw this in the System logs:
| err_level | date_time | app | err_message |
|---|---|---|---|
| default | 16:27:32.072624-0700 | kernel | memorystatus: killing_idle_process pid 9070 [AUHostingService] jetsam_reason->osr_code: 9 |
| error | 16:27:32.075124-0700 | Logic Pro X | AUHostingServiceClient.mm:227 [AUHostingService Client] connection interrupted. |
| default | 16:27:32.075738-0700 | WindowManager | Connection invalidated |
| default | 16:27:32.080029-0700 | runningboardd | XPC connection invalidated: [xpcservice<com.apple.audio.AUHostingService.arm64e([app<application.com.apple.logic10.3602314.3602320(501)>:9061])(501)>:9070:9070] |
| default | 16:27:32.083873-0700 | mDNSResponder | [R282813] DNSServiceCreateConnection STOP PID9070 |
| default | 16:27:32.084055-0700 | mDNSResponder | [R282811] DNSServiceBrowse Cancel domain enumeration for WAB and mDNS PID9070 |
| default | 16:27:32.088387-0700 | mDNSResponder | [R282811] DNSServiceBrowse(0, 0, <mask.hash: ‘n5Idhb710/ZAw8YrVV2M9g==’>) STOP PID9070 |
| default | 16:27:32.087057-0700 | runningboardd | [xpcservice<com.apple.audio.AUHostingService.arm64e([app<application.com.apple.logic10.3602314.3602320(501)>:9061])(501)>:9070:9070] termination reported by launchd (1, 9, 9) |
| default | 16:27:32.089047-0700 | runningboardd | Removing process: [xpcservice<com.apple.audio.AUHostingService.arm64e([app<application.com.apple.logic10.3602314.3602320(501)>:9061])(501)>:9070:9070] |
| default | 16:27:32.089272-0700 | runningboardd | removeJobWithInstance called for identity without existing job [xpcservice<com.apple.audio.AUHostingService.arm64e([app<application.com.apple.logic10.3602314.3602320(501)>:9061])(501)>:9070:9070] |
| default | 16:27:32.089318-0700 | runningboardd | Removing assertions for terminated process: [xpcservice<com.apple.audio.AUHostingService.arm64e([app<application.com.apple.logic10.3602314.3602320(501)>:9061])(501)>:9070:9070] |
| default | 16:27:32.104243-0700 | loginwindow | -[PersistentAppsSupport applicationQuit:] |
So this would appear to indicate that a “kernel memorystatus” process is “cleaning up” (?) and killing AUHostingService sub-processes (plugins).
Does that seem like a reasonable assessment?
Does JUCE need to better handle these events or do I need to go yell at Apple again?
Digging even deeper (Apple source code) reveals jetsam_reason->osr_code: 9:
#define JETSAM_REASON_MEMORY_IDLE_EXIT 9
(from this kernel source header)
The actual function that kills these idle process is defined in this kernel C source code.
This is the if statement that decides if a process is to be a “victim”:
3815 if ((p->p_memstat_dirty & (P_DIRTY_ALLOW_IDLE_EXIT|P_DIRTY_IS_DIRTY|P_DIRTY_TERMINATED)) == (P_DIRTY_ALLOW_IDLE_EXIT)) {
3816 if (current_time >= p->p_memstat_idledeadline) {
3817 p->p_memstat_dirty |= P_DIRTY_TERMINATED;
3818 victim_p = proc_ref_locked(p);
3819 break;
3820 }
3821 }
So maybe these AUHostingService subprocesses need to not have PROC_DIRTY_ALLOW_IDLE_EXIT set (as defined earlier in the same source file):
3210 if (pcontrol & PROC_DIRTY_ALLOW_IDLE_EXIT) {
3211 p->p_memstat_dirty |= P_DIRTY_ALLOW_IDLE_EXIT;
3212 }
Or maybe these processes shouldn’t be “tracked”:
3078 /* Idle exit requires that process be tracked */
3079 if ((pcontrol & PROC_DIRTY_ALLOW_IDLE_EXIT) &&
3080 !(pcontrol & PROC_DIRTY_TRACK)) {
3081 return EINVAL;
3082 }
