Code review: Workaround to get true modal dialogs on Windows and OSX

I make plugins for a host who only expects me to show a true modal dialog (Premiere Pro). So its “Show settings” callback is called with the host already fully disabled; I get a native handle to the host window to parent myself to, I’m expected to show a modal dialog to do my thing, blocking execution on the dialog, and only to return once the dialog is dismissed. So none of that semi-modal async embedded panel VST plugin stuff :slight_smile:

That said I had the hardest time getting true modal dialogs to work properly with Juce. And not only do I need to show modal dialogs on the host, but I also need modals on my modals, since that one main modal dialog is a switchboard to all sort of functionality of my plugin. I just want to build a multi-tier dialog-based app in my plugin.

The problems with using the default Juce modal functionality:

  • I can specify the parent dialog on OSX, but it then makes the modal dialog a subview of the host’s view, effectively embedding the dialog as a subcontrol in the host window.
  • Not specifying a parent on OSX makes the dialog disconnected from the host, and the user can just click on and/or Ctrl-Tab to the disabled host and my dialog will go behind it (forgot which one it was). This can be solved by making my dialog a topmost window, but then it’s topmost for all other apps in the system, which is also undesirable.
  • Specifying another Juce dialog as the parent of another Juce dialog also has this sub-can-go-behind-parent nuisance, on both Windows and OSX.

I now however have some code ready that seems to play nicely on both Windows and OSX, with one remaining minor nuicance on Windows. What I came up with is a C_ModalDialog class, together with a helper C_ParentWindow class which allows the C_ModalDialog to be parented to both Juce components/dialogs and native window handles. It wraps a Juce::DialogWindow, and I’m using a modified version of the innards of Juce’s LaunchOptions to get it going. As a solution for the OSX topmost hack I just parent the new Juce window to the host window, and as a solution to Juce subdialogs being able to go behind their Juce parent dialogs I just cut the link between; I get the Juce parent dialog’s native window handle and treat it just like I do the host window handle.

The C_ParentWindow is a simple wrapper that enables me to transform both native handles and Juce dialog pointers into the same “native parent window handle + possible parent Juce component” data, so that I can parent my dialogs to both Juce dialogs and native handles the exact same way.

class C_ParentWindow {
    // the native handle to use for this parent window
    void* m_handle = nullptr;

    // the associated Juce component, if any
    juce::Component* m_componentPtr = nullptr;

    // sets the parent window to be either the given native handle or the given Juce component
    C_ParentWindow(void* parentWindowHandle) {
        m_componentPtr = nullptr;
        m_handle = parentWindowHandle;
    }
    C_ParentWindow(juce::Component* parentComponentPtr) {
        m_componentPtr = parentComponentPtr;
#ifdef __APPLE__
        m_handle = [(NSView*)parentComponentPtr->getWindowHandle() window];
#else
        m_handle = parentComponentPtr->getWindowHandle();
#endif
    }
};

class C_ModalDialog : public juce::DialogWindow {
    int LaunchModal(juce::Component* dialogContentPtr, C_ParentWindow& parentWindow);
        // set our options
        setUsingNativeTitleBar(true);
        setAlwaysOnTop(false);
        setContentNonOwned(dialogContentPtr, true);
        centreAroundComponent(parentWindow.m_componentPtr, getWidth(), getHeight());
        setResizable(false, false);
        
        // bring us to life, ensuring we're parented under the parent
#ifdef __APPLE__
        addToDesktop(getDesktopWindowStyleFlags(), nullptr);
        NSWindow* parentWindowHandle = (NSWindow*)parentWindow.m_handle;
        NSWindow* usAsWindowHandle = [(NSView*)getWindowHandle() window];
        [parentWindowHandle addChildWindow:usAsWindowHandle ordered:NSWindowAbove];
#else
        addToDesktop(getDesktopWindowStyleFlags(), parentWindow.m_handle);
#endif
    
        // transit to a modal state
        enterModalState(true, nullptr, true); // last 'true' means Juce will delete us on exit of dialog

        // show the dialog
        int result = runModalLoop();

        // decouple the dialog from the host
#ifdef __APPLE__
        [parentWindowHandle removeChildWindow:usAsWindowHandle];
#endif

        // ensure our parent has focus now
        // workaround for Windows -- if not done so, Juce effectively performs an alt-tab if we return to the host
        // window when we stacked 2 or more subdialogs on each other
#ifdef _WIN32
        HWND parentWindowHandle = (HWND)parentWindow.m_handle;
        SetForegroundWindow(parentWindowHandle);
#endif
        
        // and return the result
        return result;
    }

    // gets the window style flags to use
    int getDesktopWindowStyleFlags() const override {
        // for windows: ensure the 'add to taskbar' flag is not set so we don't end up there and in the alt-tab list
        // FIXTHIS: has other undesired side effect of the content not fitting in the window anymore &&
        // the window not having our icon anymore
        return juce::DialogWindow::getDesktopWindowStyleFlags();// & ~juce::ComponentPeer::windowAppearsOnTaskbar;
    }
};

The above code is a stripped down version of what I have (it is a base class for all my modal dialogs, and handles all sorts of commonalities for my modal dialogs), but the code should work as adverised I think.

This seems to work nicely on OSX, and on Windows the only beef I still have with it is that it shows a taskbar icon for every modal opened, but I can live with that if need be. However, is the above code good enough? I’m not a true OSX developer, so have I dropped the ball on something there? Any caveats I overlooked or OS versions I’m now incompatible with? If anyone could add their thoughts about it, that would be great!

I found a snag also on OSX: while the modal dialog is now always on top of the parent window, and while the parent window is nicely disabled since it calls us from it’s GUI thread (as by design – we’re expected to block here), on OSX the host’s menu is still fully active and responsive… Users can therefore trigger host functionality like a modal “Start new project” dialog while my modal dialog is up. This of course leads to certain deadlock.

Is this a matter of one or more of:

  • Bad design of the host – it should have disabled it’s menus first before calling us?
  • Bad coding from me – I should disable the host’s menu myself? But isn’t this a no-no on OSX?
  • A badly thought out workflow – the host calls my callback function and wants me return a value, for which I must show a dialog for user interaction, so the dialog must effectively be modal, but the host is not properly prepared for it?
  • A mismatch between the way Juce works and the way the host works on OSX?

I’ve already posted a “What to do?” post in the host’s (Adobe Premiere) forums, but haven’t gotten any answer there yet (nor do I expect an answer there given past experience with these kind of questions).

I’ve also tried dabbling with

[[NSApplication sharedApplication] runModalForWindow:myJuceNativeWindowHandle]

but that (expectedly) also only led to grief; I expect that when executing that line a private message loop is started, but I cannot call Juce’s runModalLoop() before that (since it has it’s own message loop) nor after (since then Juce isn’t yet active to handle the messages)?