Crash on HWNDComponentPeer::destroyWindowCallback

Hello, I’ve run into a trouble that my program crashes everytime at “HWNDComponentPeer::destroyWindowCallback” when the main window is shutting down.
Here is the call stack:

00007ffa945dd5b6() unknown
user32.dll!00007ffae0e663ed() unknown
user32.dll!00007ffae0e65fbc() unknown
user32.dll!00007ffae0e72753() unknown
ntdll.dll!00007ffae10dfe14() unknown
win32u.dll!00007ffade1923e4() unknown
CloudMusicianServer.exe!juce::HWNDComponentPeer::destroyWindowCallback(void * handle=0x0000000000050b96) line 2199 C++
CloudMusicianServer.exe!juce::HWNDComponentPeer::callFunctionIfNotLocked(void ()(void *) callback=0x00007ff69443abd2, void * userData=0x0000000000050b96) line 3444 C++
CloudMusicianServer.exe!juce::HWNDComponentPeer::~HWNDComponentPeer() line 1292 C++
[external code]
CloudMusicianServer.exe!juce::Component::removeFromDesktop() line 709 C++
CloudMusicianServer.exe!juce::Component::~Component() line 481 C++
CloudMusicianServer.exe!juce::TopLevelWindow::~TopLevelWindow() line 154 C++
CloudMusicianServer.exe!juce::ResizableWindow::~ResizableWindow() line 60 C++
CloudMusicianServer.exe!juce::DocumentWindow::~DocumentWindow() line 80 C++
CloudMusicianServer.exe!CMServerAPPEntry::MainWindow::~MainWindow() line 191 C++
[external code]
CloudMusicianServer.exe!CMServerAPPEntry::~CMServerAPPEntry() line 57 C++
[external code]
CloudMusicianServer.exe!juce::JUCEApplicationBase::main() line 266 C++
CloudMusicianServer.exe!WinMain(HINSTANCE__ * __ formal=0x00007ff694430000, HINSTANCE__ * __formal=0x0000000000000000, char * __formal=0x000001f528c14a85, int __formal=10) line 233 C++
[external code]

It is weird that my last version can shutdown safely, and the current version cannot, but I can’t find any of my modification to the project may have impact on this sort of things…

The only suspicious modification of my code is that I start a Thread to make a async cleanup on the “systemRequestedQuit” method, and wait for all the cleanup things to finish before calling the quit() method, because my cleanup code waits for jobs on message thread, this request message thread to not be blocked, so I change the cleanup method from calling directly on message thread to calling on a background thread.

And even with these modification and this crash, I found that if I hide the main window (via setVisable(false)) just before starting the cleanup thread, then everything’s fine, this crash won’t appear any more…

But of course, this is not an ideal solution.
And this is a crash that I’ve never seen before, so I got really confused about what have gone wrong…
And this problem contains many variables, so I just couldn’t write a minimum reproduction code…

Is there something that I did wrong? Or is this really is JUCE bug?
Anyone have any ideas on this?

update: this is not a 100% recurring crash, as I was hosting some plugins on my program, and if I just switch to another bunch of plugins without changing any of my code, this crash would not happen.
But in the last bunch of plugins, this crash was 100%, and if I clean them up using the message thread directly, they would not crash as well…

Can this be a bug in the plugin?
But since the plugins are in seperate dll modules, if this crash is really inside the plugin, there should be a plugin dll module in the call stack.
However, as you can see, the call stack goes directly from the exe module to the windows runtime dll module. The entire call stack does not seem to have anything to do with the plugin, which makes me even more confused …

I’m really frustrated…
Can anyone who has experienced a similar situation come to help me?

Do you get any more information in the VS output window about the crash, like an error message of some kind? Can you see which line of the destroyWindowCallback() method it’s crashing on?

I suspect there may not be any particularly useful output.
When shutting down, all output is only:

The thread 0x9220 has exited with code 0 (0x0).
‘xxx.exe’ (Win32): Unloaded ‘C:\Program Files\Common Files\VST3\xxx.vst3’
The thread 0x9e0c has exited with code 0 (0x0).
‘xxx.exe’ (Win32): Loaded ‘C:\Windows\System32\sspicli.dll’. Cannot find or open the PDB file.
‘xxx.exe’ (Win32): Loaded ‘C:\Windows\System32\OnDemandConnRouteHelper.dll’. Cannot find or open the PDB file.
‘xxx.exe’ (Win32): Loaded ‘C:\Windows\System32\winhttp.dll’. Cannot find or open the PDB file.
‘xxx.exe’ (Win32): Loaded ‘C:\Windows\System32\mswsock.dll’. Cannot find or open the PDB file.
‘xxx.exe’ (Win32): Loaded ‘C:\Windows\System32\IPHLPAPI.DLL’. Cannot find or open the PDB file.
‘xxx.exe’ (Win32): Loaded ‘C:\Windows\System32\nsi.dll’. Cannot find or open the PDB file.
‘xxx.exe’ (Win32): Loaded ‘C:\Windows\System32\winnsi.dll’. Cannot find or open the PDB file.
‘xxx.exe’ (Win32): Loaded ‘C:\Windows\System32\dnsapi.dll’. Cannot find or open the PDB file.
‘xxx.exe’ (Win32): Loaded ‘C:\Windows\System32\rasadhlp.dll’. Cannot find or open the PDB file.
‘xxx.exe’ (Win32): Loaded ‘C:\Windows\System32\FWPUCLNT.DLL’. Cannot find or open the PDB file.
‘xxx.exe’ (Win32): Loaded ‘C:\Windows\System32\bcrypt.dll’. Cannot find or open the PDB file.
Exception thrown at 0x00007FFC38E9B621 in xxx.exe: 0xC0000005: Access violation executing location 0x00007FFC38E9B621.

And it crashes on xxx.exe!juce::HWNDComponentPeer::destroyWindowCallback(void * handle=0x00000000001b0c38) Line 2199.

What commit of JUCE are you using? It’d be useful to know the exact point at which it’s crashing. Without a reproducable crash though it’ll be hard to track down what the issue is, it could be any number of things especially if you’re hosting plug-ins within the program.

Okay, I now have a VERY simple reproduction code.

First, I built a dummy audio plugin by simply building the factory audio plugin code template provided by JUCE without any changes, and copied it to “C:\Program Files\Common Files\VST3\AudioPluginTest.vst3”.

Then, I wrote the following very simple host code (MainComponent is the factory code of the JUCE GUI project, without any changes. In order to reduce the post layout, I will not post it here):

/*
  ==============================================================================

    This file was auto-generated!

    It contains the basic startup code for a JUCE application.

  ==============================================================================
*/

#include <JuceHeader.h>
#include "MainComponent.h"

//==============================================================================
class PluginHostTestApplication  : public JUCEApplication, public AsyncUpdater, public Thread
{
public:
    //==============================================================================
    PluginHostTestApplication() : Thread("Plugin Async Destroyer") {}

    const String getApplicationName() override       { return ProjectInfo::projectName; }
    const String getApplicationVersion() override    { return ProjectInfo::versionString; }
    bool moreThanOneInstanceAllowed() override       { return true; }

    //==============================================================================
    void initialise (const String& commandLine) override
    {
        // This method is where you should put your application's initialisation code..

        mainWindow.reset (new MainWindow (getApplicationName()));
		triggerAsyncUpdate();
    }

    void shutdown() override
    {
        // Add your application's shutdown code here..

        mainWindow = nullptr; // (deletes our window)
    }

	void handleAsyncUpdate() override
	{
		VST3PluginFormat vst3;
		OwnedArray<PluginDescription> descs;

		vst3.findAllTypesForFile(descs, "C:\\Program Files\\Common Files\\VST3\\AudioPluginTest.vst3");
		pluginInstance.reset(vst3.createInstanceFromDescription(*descs[0], 44100.0, 1024).release());
	}

	void run() override
	{
		pluginInstance = nullptr;
		quit();
	}

    //==============================================================================
    void systemRequestedQuit() override
    {
        // This is called when the app is being asked to quit: you can ignore this
        // request and let the app carry on running, or call quit() to allow the app to close.
		startThread();
    }

    void anotherInstanceStarted (const String& commandLine) override
    {
        // When another instance of the app is launched while this one is running,
        // this method is invoked, and the commandLine parameter tells you what
        // the other instance's command-line arguments were.
    }

    //==============================================================================
    /*
        This class implements the desktop window that contains an instance of
        our MainComponent class.
    */
    class MainWindow    : public DocumentWindow
    {
    public:
        MainWindow (String name)  : DocumentWindow (name,
                                                    Desktop::getInstance().getDefaultLookAndFeel()
                                                                          .findColour (ResizableWindow::backgroundColourId),
                                                    DocumentWindow::allButtons)
        {
            setUsingNativeTitleBar (true);
            setContentOwned (new MainComponent(), true);

           #if JUCE_IOS || JUCE_ANDROID
            setFullScreen (true);
           #else
            setResizable (true, true);
            centreWithSize (getWidth(), getHeight());
           #endif

            setVisible (true);
        }

        void closeButtonPressed() override
        {
            // This is called when the user tries to close this window. Here, we'll just
            // ask the app to quit when this happens, but you can change this to do
            // whatever you need.
            JUCEApplication::getInstance()->systemRequestedQuit();
        }

        /* Note: Be careful if you override any DocumentWindow methods - the base
           class uses a lot of them, so by overriding you might break its functionality.
           It's best to do all your work in your content component instead, but if
           you really have to override any DocumentWindow methods, make sure your
           subclass also calls the superclass's method.
        */

    private:
        JUCE_DECLARE_NON_COPYABLE_WITH_LEAK_DETECTOR (MainWindow)
    };

private:
    std::unique_ptr<MainWindow> mainWindow;
	std::unique_ptr<AudioPluginInstance> pluginInstance;
};

//==============================================================================
// This macro generates the main() routine that launches the app.
START_JUCE_APPLICATION (PluginHostTestApplication)

When closing the main window, this host code will 100% crash in exactly the same place as I said at the beginning.
As a supplement, the JUCE version I use is the latest commit version on the master branch.

Thanks for the example code.

The underlying issue here is that destroying plug-in instances on threads other than the main thread is going to be problematic. In this case, I think what is happening is that the destructor of the plug-in is tearing down the internal Windows message loop whilst there is a message already in flight so we get a crash when it’s delivered. To fix this we need to always enforce destruction of the plug-in instance on the main thread (we already do something similar for VST2s). This has been added to the juce6 branch here:

1 Like

Thanks for fixing! This solves the crash problem of asynchronous destruction here.

But there is still a problem here. The reason why I wanted to destroy the plugin instance on a background thread is because when I load the plugin on a background thread (If there are a lot of plugins to load, and you don’t want to block the main thread for a long time, loading all of them on a background thread is more convenient to implement and manage than loading one at a time on the main thread, and then sending an asynchronous callback to the main thread to load the next. After all, JUCE will automatically dispatch the “create plugin instance” procedure to the main thread to complete), and then I want to cancel the operation and destroy the plugin instance when the plugin is not fully loaded, performing this “cancel” operation on the main thread will cause the main thread to have a certain chance (not 100%) of being stuck.

My way to implement this “cancel” operation is to send a stop signal to the loading thread and then wait for the thread to exit. The loading thread will continuously determine whether threadShouldExit () is true. When it is found to be true, it will return immediately, and will not continue the subsequent loading process, so that a “cancel” operation can be achieved.

This should work normally, but performing such a “cancel” operation on the main thread does sometimes cause the main thread to freeze. After doing some debugging, I found that when the AudioPluginFormat::createInstanceFromDescription method is not called on the main thread, it will call the createPluginInstanceAsync method to send an asynchronous callback to the main thread to create the plug-in instance, while waiting for the plug-in instance creation operation on the main thread to complete. If this “cancel” operation happens to be triggered after the AudioPluginFormat::createInstanceFromDescription call, before the AsyncCreateMessage asynchronous message is sent to the message queue, the main thread will fall into an infinite waiting state similar to a deadlock, because it is waiting for the return of the loading thread which is waiting for the main thread to become idle.

One possible solution I can think of is to start a background thread to complete the “cancel and destroy” operation. But as you already know, this is also problematic, and there comes this post. Now this problem has been solved, but since this is only a workaround for the “main thread stuck” problem, and starting a background thread is also expensive, and it is not very convenient to manage, is there any way to solve the root cause of this problem, that is, when the current thread is asked to quit, can the operation that needs to wait for the completion of the task on the main thread not continue to wait?

I don’t quite understand what you are trying to achieve by creating plug-ins on a background thread as the potentially time-consuming operations will always be done on the main thread. AudioPluginFormat::createInstanceFromDescription() enforces this by posting a message which will be picked up on the main thread and calls createPluginInstance(), so even if you are using a background thread to call createInstanceFromDescription() the heavyweight work will always be handed off to the main thread. It will be much less complicated and error-prone if you only do the plug-in creation/deletion on the main thread.

In my experience in reality you have to create and destroy plugins on the message thread as some (very popular) plugins run modal loops and popup dialogs that just cause a dead lock if you run them from a background thread.

Let’s say we have 10 plugins to load, if we load all of them synchronously on the main thread, the main thread will be blocked for a long time. If we want to avoid this blocking while still using the main thread, then we need to continuously send asynchronous lambda callbacks to the main thread, and in each callback we need to check whether the loading task has been cancelled before this callback arrives, as there is currently no mechanism for JUCE to cancel an asynchronous callback that has been posted to the main thread (Perhaps using AsyncUpdater is a way, but if we use it, we need to create an additional data structure to store the index and other information that should be processed for each callback, and this can not be achieved simply by “function parameter passing” mechanism). But if we use a background thread to complete the loading of all plugins, we only need a loop to solve everything.

Of course I can do the trade-off mentioned above, but do you think this kind of trade-off has to be done, or can there still be a better solution?

Thanks for your advise. Do you think that plugin instance creation is the only method that is unsafe to call on a background thread, or are there many other initialization-related methods that are not safe to call on a background thread? such as setting the state information, buses layout, etc.

Honestly in my experience pretty much the only thing you can do on a non-message thread is to get plugin values, set plugin values and process.

I wish plugins were more stable and thread-safe but over the years we’ve had to be more and more restrained with our threading to accommodate flaky plugins. Now if only everyone used pluginval I could add tests to hammer all this stuff on background threads…

2 Likes

Okay, it looks like I’d better go back and use AsyncUpdater to refactor the plugin loading / unloading logic so that they can all be dispatched to the main message thread for execution. Thanks again.

Okay, I finally spot that using a Timer is the safest way to load a large number of plugins, as calling triggerAsyncUpdate continuously will also block the main thread.
I wrote a very simple class to illustrate this approach, so that people who encounter similar problems in the future can also refer to this implementation:

class PluginPool : private Timer
{
public:
	PluginPool()
	{
		OwnedArray<PluginDescription> descs;

		vst3.findAllTypesForFile(descs, "C:\\Program Files\\Common Files\\VST3\\AudioPluginTest.vst3");
		pluginDesc = *descs[0];

		startTimer(1);
	}

	~PluginPool()
	{
		stopTimer();
	}

private:
	void timerCallback() override
	{
		pluginInstance.reset(vst3.createInstanceFromDescription(pluginDesc, 44100.0, 1024).release());
	}

	VST3PluginFormat vst3;
	PluginDescription pluginDesc;
	std::unique_ptr<AudioPluginInstance> pluginInstance;
};

As long as it is on the main thread, it is safe to create and destroy instances of this class at any time.

Update, @dave96 you can also take a look:
The above code works fine for all plugins we have tested except Kontakt.
Kontakt will crash if a message is sent to the message thread to stop loading just after the instance is created and before setting the instance state.
A quick way to reproduce is to use the following modified PluginPool class:

class PluginPool : private Timer
{
public:
	PluginPool()
	{
		OwnedArray<PluginDescription> descs;

		vst.findAllTypesForFile(descs, "path\\to\\your\\Kontakt.dll");
		pluginDesc = *descs[0];

		startTimer(1);
	}

	~PluginPool()
	{
		stopTimer();
	}

private:
	void timerCallback() override
	{
		pluginInstance.reset(vst.createInstanceFromDescription(pluginDesc, 44100.0, 1024).release());

		if (firstStateInfo.getSize() == 0) pluginInstance->getStateInformation(firstStateInfo);
		else
		{
			MessageManager::callAsync([] { JUCEApplicationBase::getInstance()->systemRequestedQuit(); });
			pluginInstance->setStateInformation(firstStateInfo.getData(), (int)firstStateInfo.getSize());
		}
	}

	VSTPluginFormat vst;
	PluginDescription pluginDesc;
	MemoryBlock firstStateInfo;
	std::unique_ptr<AudioPluginInstance> pluginInstance;
};

And then, in your app entry, create an instance of PluginPool in the initialise method, and destroy it in the systemRequestedQuit method, just like this:

class PluginHostTestApplication  : public JUCEApplication
{
public:
    //==============================================================================
    PluginHostTestApplication() {}

    const String getApplicationName() override       { return ProjectInfo::projectName; }
    const String getApplicationVersion() override    { return ProjectInfo::versionString; }
    bool moreThanOneInstanceAllowed() override       { return true; }

    //==============================================================================
    void initialise (const String& commandLine) override
    {
        // This method is where you should put your application's initialisation code..

        mainWindow.reset (new MainWindow (getApplicationName()));
		pluginPool.reset (new PluginPool);
    }

    void shutdown() override
    {
        // Add your application's shutdown code here..

        mainWindow = nullptr; // (deletes our window)
    }

    //==============================================================================
    void systemRequestedQuit() override
    {
        // This is called when the app is being asked to quit: you can ignore this
        // request and let the app carry on running, or call quit() to allow the app to close.
		pluginPool = nullptr;
		quit();
    }

private:
    std::unique_ptr<MainWindow> mainWindow;
	std::unique_ptr<PluginPool> pluginPool;
};

And then, BOOM, a crash occurred in Kontakt ’s internal module.
I think this is obviously Kontakt’s bug, but is there any way we can workaround it? Or do you have any suggestions on how to avoid this crash?

By the way, we are using Kontakt 6.2.2, but I think this bug may exist in all Kontakt releases.

Tbh I’m not sure I fully follow this thread. I’m not exactly sure what you’re trying to achieve with the PluginPool and Timer… Why don’t you just use MessageManager::callAsync with a WaitableEvent to stop the current thread from continuing until the lambda body has executed?

I’m not sure what’s causing Kontakt to crash. Maybe they’re pumping the message loop in their setState call? Can you just check the MessageManager::hasStopMessageBeenSent return value and avoid setting the state in that case?

I just want to implement a cancelable plugin loading job. If I have a lot of plugins to load, I hope that I can cancel the loading task at any time in the middle, instead of having to wait for all plugins to be loaded before doing any other operations. All my attempts above are trying to achieve this function, but all these methods seem to have some problems. Do you have any good ideas on how to implement this function?

As a supplement, I don’t want to exit the program after cancellation, so it doesn’t make sense to rely on MessageManager :: hasStopMessageBeenSent .