The problem is caused when juce::Timer is used as part of a singleton instance. The problem is triggered during unloading of the VST3 DLL.
Attached is a minimal reproducible example. The README.md gives some details and steps on how to reproduce (although it’s only a matter of building).
What happens if you use the the Singleton in the JUCE way:
Use JUCE_DECLARE_SINGLETON and JUCE_IMPLEMENT_SINGLETON, along with inheriting from DeletedAtShutdown?
Using JUCE_DECLARE_SINGLETON and JUCE_IMPLEMENT_SINGLETON doesn’t give the same problem.
However, the singleton thing is just one way to reproduce the problem. In my project I’m not using juce::Timer as singleton (or as part of a singleton) and I have yet to find the exact conditions in which this problem is triggered.
EDIT: I found the singleton which holds a juce::Timer. Now that I’ve reworked the code to use juce::SharedResourcePointer instead, the problem is solved.
I hope we can all agree that using juce::Timer as a singleton should be supported.
I’m not using a Juce timer as singleton and i have the same issue, running on CI (problem on windows only). I wasn’t able to find out why it hangs (not using singletons ever, that i know of).
So after a day of unsuccessful digging i took a look at what the vst3 helper does, removed it, and wrote my own that does the same stuff copying the files around etc and my CI is running again… I’d love this to get resolved too, though, so I can build vanilla again on CI
Interesting, I’ve got a fix for this internally but I want it to go through a few pairs of eyes before I commit to anything.
What I found on windows was that a dll appears to be go through the following sequence after main() has finished (although I only tested with the vst3 helper)
Any remaining threads are killed by force, meaning they may be stopped at absolutely any point during their execution and never return
After this the destructors of any static objects are called
As far as I could tell in the case of the Timer what was happening was this…
A static object that is a or has a Timer is created
startTimer() is called which creates a singleton TimerThread object
main finishes
The TimerThread’s thread (i.e. the run() function) is killed by force by the OS, there is as far as I can tell no way for the thread to react to this
The static object which is a or has a Timer is destroyed
As it’s the last Timer object it also triggers the destruction of the TimerThread object
In the TimerThread destructor it waits for the thread to finish, which is determined by checking the value of a threadHandle pointer
As the thread was killed by force the threadHandle has not been set to nullptr meaning the destructor waits forever
Why is this a problem now? There’s at least two reasons
TimerThread used to inherit from DeletedAtShutdown JUCE will call delete on any DeletedAtShutdown objects still alive when the MessageManager is shutdown
TimerThread didn’t use to wait indefinitely for the thread to shutdown
Based on what I found any Thread object that waits in its destructor indefinitely for the thread to finish that has a static instance may not finish if the OS kills the thread in this way.
I have to say however this behaviour surprised me and I feel like there is more to it.
Does it load the plugin to get the all the module info? If so interesting that it doesn’t suffer the same issue. I hadn’t gotten around to trying to load a dll from a different executable.
Nope, honestly I just ignore that part right now because it broke our production CI, while I needed other JUCE updates for other production bug fixes
I’m currently just copying the dll to the right location, rename it to vst3, and my installer build then grabs that binary instead of the entire bundle - which works well enough but definitely is a bandaid fix.
Until I found this thread here, I thought it was about the strangely formatted paths in the VST3 VS2022 project file, which produced some double-// in paths within the Pre/Postbuild scripts for me (I’m not sure about the details anymore, but the script paths passed to vst3helper didn’t seem correct at the time - i injected my own “vst3helper.exe” dummy and just printed all args passed in during build, which lead me to find broken-looking paths in the VS2022 file… but, again I’m not sure about this at all).
I thought all of this was happening to me, because I’m running the whole thing on a custom Git Bash setup on Windows, which is always a bit iffy with path-related stuff.
Now, reading all your comments regarding Timer usage during construction, could usage of Timer::callAfterDelay during construction be an issue?
This is not any specific code I’m using but rather a technique that I sometimes use when async resources may or may not be ready during construction and I have no way of getting better signals for some reason.
void MyClass::doSomethingAsync ()
{
if ( ! tryTheThing () )
{
Timer::callAfterDelay (100, [ &, that = WeakReference<MyClass> (this) ]() {
if (that != nullptr)
that->doSomethingAsync ();
});
}
}
That’s really the only thing I do, that’s not just a Timer-derived class that I can think of, would that affect the vst3helper and get stuck the same like your examples?
No I don’t think so, the object it creates inherits from DeletedAtShutdown which should be guaranteed to be deleted before any threads get killed. I think it will only only be things with static lifetime duration. However, as I said above there could be more to this I don’t understand yet.
Just keep in mind that
Creating the moduleinfo.json should speed up scanning your plugin in some DAWs
If the plugin hangs in the helper it might also hang in other DAWs!
I think it could be any Thread inherited class that calls stop (-1) in it’s destructor (note Thread calls this in the base destructor but it would jassert first if it thinks the thread is “still running”.
Note you could probably use MessageManager::callAsync() for this? The Timer only happens on the message thread anyway. Unless there is a good reason to wait 100ms between attempts.
I’m not sure what causes the Timer issue but I just wanted to say that you can also attach a debugger to juce_vst3_helper and run it with the exact command arguments JUCE is running which has helped me solve similar issues when I was doing suspect things with object lifetimes.
I’m testing all plugins with pluginval during the build process, both before building an installer, as well as after installing them from the built installer - Testing strictness 10, 3 repeats, validate-in-process. Never had a single issue (especially not after the vst3helper issues started) and also haven’t gotten a single report on several thousand installations of more than 10 different plugins experiencing this vst3helper-freezing issue during our CI builds.
I never do that, I properly stop threads always, I’d never use -1 to stop and I also haven’t hit that assertion in years (I know the one you’re talking about - “bad karma if you reach this point”…).
This was only an abstract example. I do my Timer thing only where appropriate, when i don’t want the MessageThread to immediately call again, but when I DO want to wait first. I also only mentioned that technique with Timer up there because I do that in one place and it somewhat fit the issues being Timer-related
I’ll see if I can put together a test project going from a vanilla JUCE plugin over the weekend, that reproduces the issue and maybe I’ll also find what causes it on my end while doing that.
As I said, I “solved” it for our case by just not using the vst3helper for now and I have 0 issues just shipping the actual vst3 binaries without the moduleinfo; so I can’t dedicate a lot of time to it right now, but I’ll do my best to get you a project going where it’s reproducible.
Running the helper in the debugger solved my issue a few weeks back in no time.
The quick succession of creating and destroying is something pluginval is unlikely to catch.
YMMV
Really? I was under the impression that’s what the cold open test does:
{
beginTest ("Open plugin (cold)");
StopwatchTimer sw;
deletePluginAsync (testOpenPlugin (pd));
logVerboseMessage ("\nTime taken to open plugin (cold): " + sw.getDescription());
}
The async deletion only grants the plugin an extra 150ms to delete on a thread, but i thought that’s only done for the frontend/timer to not block. The plugin itself is still immediately deleted after opening, the thread just waits for that to finish while the timer in this block can get done measuring the open-time.
Anyways, I’ll hook up a debugger to the vst3helper soonish and see what that yields, thanks for the tip.
So we kept the solution simple in the end hopefully this works for everyone.
However, word of caution, always stop a thread before any global or static object destructors run. Also prefer to wait for all global or static objects to be created before starting any threads.
It seems that this commit is causing compatibility problems when using PluginDoctor with any VST3 plugin that utilizes an AudioProcessorValueTreeState.
The specific assert that is triggered can be found on line 70 in the juce_Singleton.h file:
if (createdOnceAlready)
{
// This means that the doNotRecreateAfterDeletion flag was set
// and you tried to create the singleton more than once.
jassertfalse;
return nullptr;
}
Prior to this commit, this issue did not occur with PluginDoctor.
I wanted to add that the issue can also arise in PluginDoctor with plugins that don’t utilize the APVTS class.
PluginDoctor initially loads the VST3 DLL to memory, instantiates it, deletes that instance, and subsequently creates a new instance. However, it does not unload the VST3 DLL from memory after deleting the first and only instance, which is why the createdOnceAlready assertion gets triggered even if there were no instances prior.
With PluginDoctor, the VST3 DLL persists in memory even in the absence of any instances. On the other hand, DAWs like FL Studio unloads the VST3 DLL from memory if there are no remaining instances, thus avoiding the assertion. Additionally, with the Timer Thread now being a singleton instance with doNotRecreateAfterDeletion set to true, any host that behaves similarly to PluginDoctor will trigger the same assertion.
Below is the sequence of events that leads to the assertion with PluginDoctor:
This commit resolves the initial issue I described with PluginDoctor, but it introduces a new assertion when a host unloads the VST3 DLL from memory. I’ve also tested this commit with both FL Studio and Studio One on Windows 11, and the assertion occurs in these DAWs when the DLL is unloaded. This new assertion also occurs when closing standalone applications.
This specific assertion can be found on line 50 in the the juce_Singleton.h file:
~SingletonHolder()
{
/* The static singleton holder is being deleted before the object that it holds
has been deleted. This could mean that you've forgotten to call clearSingletonInstance()
in the class's destructor, or have failed to delete it before your app shuts down.
If you're having trouble cleaning up your singletons, perhaps consider using the
SharedResourcePointer class instead.
*/
jassert (instance.load() == nullptr);
}
Here is the call stack from a blank standalone project:
TestProject.exe!juce::SingletonHolder<juce::ShutdownDetector,juce::CriticalSection,0>::~SingletonHolder<juce::ShutdownDetector,juce::CriticalSection,0>() Line 50
TestProject.exe!`dynamic atexit destructor for 'juce::ShutdownDetector::singletonHolder''()
.