Hang in Windows UI call

1

Ok, I’m a bit lost here, maybe someone has an idea: I have an app (let’s call it app A) that loads plugins and communicates with another app (app B) via IPC. On Windows, app A attaches the editor of a loaded plugin to a HWND it obtains from app B. So far, so good.
Now, after a few seconds of turning knobs on the plugin UI, app A hangs, and I’ve narrowed it down to peerWindowProc (HWND h, UINT message, WPARAM wParam, LPARAM lParam) in juce_win32_Windowing.cpp, and specifically to the case when message == WM_SETCURSOR, in which case said function calls DefWindowProcW (h, message, wParam, lParam) which never returns (and since this is the message thread of app A, it also brings the interprocess connection to a halt). The call stack starts within the loaded plugin, so moving the mouse over its UI leads to passing that event to app A (the plugin host).
When I just add a line to JUCE to ignore the case “message == WM_SETCURSOR” everything works fine, but I don’t really want to do that without knowing what might be going on here. Any ideas?

2

I can’t see why DefWindowProc would hang as it just does the default processing for messages that aren’t handled by JUCE in peerWindowProc, such as WM_SETCURSOR. I’d be hesitant to just ignore the WM_SETCURSOR message and not pass it on to DefWindowProc but I don’t really know enough about the Windows message loop to say whether it could be catastrophic or not.

Can you try adding

 case WM_SETCURSOR:
    return 0;

to the bottom of the switch statement in peerWindowProc in juce_win32_Windowing.cpp before the default case is handled and see if you app no longer hangs? This should narrow down whether it is the DefWindowProc call that is the issue.

3

Thanks for answering… I have now found out how the deadlock occurs: app B does an IPC sendMessage on its message thread and is waiting for an answer from app A. App A should receive the message on its message thread, but when a cursor move occurs over the editor of app A, Windows sends that WM_SETCURSOR message, also on the message thread, which, and here it comes, is being rerouted to app B because the editor of app A is attached to the window handle of app B. So both apps are stuck waiting for each other in their message thread.
It can be resolved by letting app A receive its IPC callbacks on a separate thread. So all is fine with JUCE, it’s just a peculiar situation, but in case anybody should ever stumble upon this, that’s how it works.

2 Likes