Finally got round to taking a look at this. I think that the problem stems from isWindowAtPoint in the NSViewComponentPeer. This function calls through to windowNumberAtPoint:belowWindowWithWindowNumber:, which is supposed to return the topmost window that would receive a mouse-down at a given screen position. The docs for this function say:
Because this method uses the same rules as mouse-down hit-testing, windows with transparency at the given point, and windows that ignore mouse events, will not be returned.
But this doesn’t seem to be true in AUs hosted in Logic on Arm. If I add some code like this into isWindowAtPoint, I see that it sometimes returns a window that ignores mouse events (the drag-image window), even though it’s not supposed to!
const auto windowAtPoint = [NSWindow windowNumberAtPoint: screenPoint belowWindowWithWindowNumber: windowToCheck];
auto* mouseWindow = [NSApp windowWithWindowNumber: windowAtPoint];
const auto ignoresEvents = [mouseWindow ignoresMouseEvents];
I tried to continue searching ‘below’ our drag-image window to see whether we eventually reach the main plugin window, but unfortunately we hit another window first. At the moment, I can’t think of a way of determining whether this inserted window is a ‘fake’ window inserted by the out-of-process hosting layer, or a real window that should block the mouse.
To sum up, I think this is another instance of the out-of-process hosting in macOS behaving in an unexpected way. Apple hints at this on their page about debugging AUs.
If custom, free-floating dialogs don’t work as you expect, add them as a subwindow to the main window — this includes custom tooltip and drop-down menu implementations.
Although not satisfying, this is probably the most robust solution. I’ll look at adding an option to the ListBox to disable dragging to external components.