Your modal window doesn’t block the main window at all, you can interact with both at the same time and your modal window doesn’t seem to have modal behavior at all?
How do you link the .dll and the .exe with JUCE? Do you staticlink each with a static version of JUCE or dou you dynamiclink them against the same juce.dll?
Because JUCE handles modal state on its own it keeps a stack of modal components. If you link your .dll and .exe against a static version of JUCE your .dll and .exe will have a seperate version of this stack, which is a possible cause for your problem. If you link against a common juce.dll there should be only one instance of the modal component stack.
So if you link static against JUCE you should compile JUCE as DLL and link against it from your .dll and .exe and see if that helps.