I just went through the documentation of DialogWindow::LaunchOptions::runModal and stumbled upon the following remark:
"Note that running modal loops inline is a BAD technique. "
I have to admit, that I do not quite understand, why modal loops are bad. I would be happy, if someone could shed light on this :-)
Are they generally bad or only in plugins?
I suppose they are an unnecessarily complicated legacy method to implement dialogs that prevent the user from using other parts of the application/plugin before they close the dialog...Qt seems to have this problem too. Opening a widget as a modal dialog comes with caveats they warn about.
All that said, a modern GUI probably shouldn't be using modal dialogs anyway. Something is probably wrong in your code if you can't apply/cancel changes made by the user on-the-fly while the dialog is open and the user also works with the application/plugin at the same time...Immediate changes to options seem to be quite consistently used for example in Mac OS-X. The model where an options/preferences dialog is opened modally and changes applied only on pressing "OK" seems to be a Windows thing and not a good one for modern GUIs.
The central reason is that if you're running a modal loop, there's a whole stack full of code underneath you. Anything can happen during the modal loop, including the deletion of objects which are relied upon when that stack gets unwound later. It's particularly bad in plugins, where the host could delete your plugin before it has had a chance to unwind its stack, which is an unrecoverable situation.