Turns out the Window issue seems to be related to Kontakt when loading previously saved MULTI data using AudioProcessor::getStateInformation().
I've isolated this down to just Kontakt. If I have my app load an instance of Kontakt but don't load any previously saved plugin state data, my windows and popups operate fine.
If I also call getStateInformation() to load the saved state - which in my case consists of MULTI data where during the loading process Kontakt displays some temporary windows briefly ( indicating its loading and converting nki instrument data ) it is I think this display of windows temporarily that puts the JUCE or OSX/Cocoa framework into this funny modal state.
I've also noticed that simply opening a modal non-native JUCE dialog once this happens - behaves as if we're on "modal" mode yet no window is shown. However the dialog window ( a custom one by me ) IS actually being instantiated and opened as it would.
This is very weird but needs sorting.
I am putting together a cut down test app to send to you to illustrate this bug.
I did check to download the latest version of Kontakt in case it was a known bug sorted now. But since i can open up K5 in multi output mode and load saved data in Ableton fine - this is something pertaining to JUCE and Kontakt interaction.