Unfortunately I don’t think any of the suggestions would resolve the problem.
This won’t allow popup windows to have completely transparent rounded corners.
This initially seemed like a viable solution, but it seems that any part of the child window that is not on top of the parent window is still painted with a magenta background. To test, I applied this patch:
childwindow.patch (2.9 KB)
Then, I built the DSPModulePluginDemo AU and loaded it in Logic on an M1 mac. Popup menus which are not entirely contained by the editor window sill have partially magenta backgrounds.
This is not a general solution, it only fixes the case where the desired corner radius matches that set by the OS. It doesn’t fix the issue for windows with a translucent background.
This would force the menu to use OS styling, which may not be acceptable to some developers. It would also only provide a solution for PopupMenus. This solution would not work for arbitrary floating widgets such as dragged items, tooltips and so on.
All this is to say that I still think the solution that will yield the most consistent results across platforms is to avoid opening new top-level windows entirely, and to draw widgets such as menus, tooltips, and dialogs directly into the editor window wherever possible.