After a bit more investigation, I’ve managed to trigger a similar issue in GarageBand 10.3 and Logic 10.6 with macOS 10.15.7 (intel).
I created a new “app” project in Xcode, then added an Audio Unit extension target to the project.
In the generated swift view controller I added the following function:
public override func viewDidAppear() {
for (i, alpha) in [0.7, 0.8].enumerated() {
let w = NSWindow(contentRect: NSRect(x: 100 + 50 * i,
y: 100 + 50 * i,
width: 200,
height: 200),
styleMask: NSWindow.StyleMask.borderless,
backing: NSWindow.BackingStoreType.buffered,
defer: true)
w.isOpaque = false
w.backgroundColor = NSColor.cyan.withAlphaComponent(CGFloat(alpha))
w.hasShadow = true
w.level = NSWindow.Level.popUpMenu
w.isReleasedWhenClosed = true
w.isExcludedFromWindowsMenu = true
w.ignoresMouseEvents = false
w.setIsVisible(true)
windows.append(w)
}
}
Here, we’re creating two cyan windows, one with an alpha of 0.7
and another with an alpha of 0.8
. We set some other properties on the windows, and then make them visible.
When I open the generated AUv3 in GarageBand, I see the following:
The window with an alpha of 0.7 is displayed in pink, whereas the window with an alpha of 0.8 renders properly. It seems that any window with an alpha value which is below a certain threshold renders in pink instead, which is why JUCE’s completely clear popupmenus don’t work as expected.
Given that I can produce the above error in a completely JUCE-free app, I’m confident that this is an issue in GB/Logic’s hosting.
Interestingly, on this setup, transparent menus in a JUCE AU plugin work properly. I also tried downloading Apple’s demo AU project, and adding the extra windows to that project - again, both windows rendered in the correct colour. After a bit of debugging, I realised that the working plugins (Apple’s demo, and JUCE AUv2 plugins) were being loaded in-process, while the broken plugin was an AUv3 being loaded out-of-process. If I updated my broken AUv3 plugin to support in-process loading instead, then it also started to render correctly.
To sum up: it looks like rendering of transparent windows is broken in both Logic and GB, specifically for plugins loaded out-of-process.
It’s probably best to report the issue to apple via the offical feedback pages for GarageBand and Logic.
Edit: I tried running GarageBand on a DTK while also viewing hierarchically-grouped processes in the Activity Monitor. If I add a JUCE AU with broken popup rendering (not an AUv3, just an AU) to a track, GarageBand starts a new “AUHostingService” process, and when I remove the plugin, that process quits. This is consistent with the theory that this bug affects only plugins which are loaded out-of-process.