Timer: Always ensure the timer thread is started is causing juce_vst3_helper.exe to hang indefinitely

After upgrading to JUCE 7.0.12 the build was not longer passing (in ci) because the build was hanging and interrupted after 2 hours.

Looking into it, I found that juce_vst3_helper.exe hangs indefinitely. By running with a debugger I arrived at the following stack trace:

<unknown> 0x00007ffb2c36f3e4
<unknown> 0x00007ffb29bc421e
juce::Thread::waitForThreadToExit(int) juce_Thread.cpp:227
juce::Thread::stopThread(int) juce_Thread.cpp:247
[Inlined] juce::Timer::TimerThread::{dtor}() juce_Timer.cpp:41
juce::Timer::TimerThread::`scalar deleting destructor'(unsigned int) 0x00007ffaa0be825f
[Inlined] std::_Ref_count_base::_Decref() memory:1178
[Inlined] std::_Ptr_base::_Decref() memory:1403
[Inlined] std::shared_ptr<juce::Timer::TimerThread>::{dtor}() memory:1687
[Inlined] juce::SharedResourcePointer<juce::Timer::TimerThread>::{dtor}() juce_SharedResourcePointer.h:102
juce::Timer::~Timer() juce_Timer.cpp:304
<unknown> 0x00007ffb29713a53
<unknown> 0x00007ffb296e042e
<unknown> 0x00007ffb296ddddd
dllmain_crt_process_detach(const bool) 0x00007ffaa13acbd1
dllmain_dispatch(HINSTANCE__ *const,const unsigned long,void *const) 0x00007ffaa13acd1a
<unknown> 0x00007ffb2c2f869f
<unknown> 0x00007ffb2c3210a6
<unknown> 0x00007ffb2c320c9d
<unknown> 0x00007ffb2b9e7fab
<unknown> 0x00007ffb296dbed8
<unknown> 0x00007ffb296dc099
__scrt_common_main_seh() 0x00007ff76ef5a1bf
<unknown> 0x00007ffb2b9e257d
<unknown> 0x00007ffb2c32aa48

This commit seems to be causing this issue. When checking out one commit before the problem disappears.

This seems to be related to:

and

EDIT 1: Only Windows is affected.

The problem is caused when juce::Timer is used as part of a singleton instance. The problem is triggered during unloading of the VST3 DLL.

Attached is a minimal reproducible example. The README.md gives some details and steps on how to reproduce (although it’s only a matter of building).

juce_vst3_helper_hang.zip (5.4 KB)

Github repo: GitHub - ruurdadema/juce_vst3_helper_hang

What happens if you use the the Singleton in the JUCE way:
Use JUCE_DECLARE_SINGLETON and JUCE_IMPLEMENT_SINGLETON, along with inheriting from DeletedAtShutdown?

Using JUCE_DECLARE_SINGLETON and JUCE_IMPLEMENT_SINGLETON doesn’t give the same problem.

However, the singleton thing is just one way to reproduce the problem. In my project I’m not using juce::Timer as singleton (or as part of a singleton) and I have yet to find the exact conditions in which this problem is triggered.

EDIT: I found the singleton which holds a juce::Timer. Now that I’ve reworked the code to use juce::SharedResourcePointer instead, the problem is solved.

I hope we can all agree that using juce::Timer as a singleton should be supported.

2 Likes

I’m not using a Juce timer as singleton and i have the same issue, running on CI (problem on windows only). I wasn’t able to find out why it hangs (not using singletons ever, that i know of).

So after a day of unsuccessful digging i took a look at what the vst3 helper does, removed it, and wrote my own that does the same stuff copying the files around etc and my CI is running again… I’d love this to get resolved too, though, so I can build vanilla again on CI

Interesting, I’ve got a fix for this internally but I want it to go through a few pairs of eyes before I commit to anything.

What I found on windows was that a dll appears to be go through the following sequence after main() has finished (although I only tested with the vst3 helper)

  • Any remaining threads are killed by force, meaning they may be stopped at absolutely any point during their execution and never return
  • After this the destructors of any static objects are called

As far as I could tell in the case of the Timer what was happening was this…

  • A static object that is a or has a Timer is created
  • startTimer() is called which creates a singleton TimerThread object
  • main finishes
  • The TimerThread’s thread (i.e. the run() function) is killed by force by the OS, there is as far as I can tell no way for the thread to react to this
  • The static object which is a or has a Timer is destroyed
  • As it’s the last Timer object it also triggers the destruction of the TimerThread object
  • In the TimerThread destructor it waits for the thread to finish, which is determined by checking the value of a threadHandle pointer
  • As the thread was killed by force the threadHandle has not been set to nullptr meaning the destructor waits forever

Why is this a problem now? There’s at least two reasons

  1. TimerThread used to inherit from DeletedAtShutdown JUCE will call delete on any DeletedAtShutdown objects still alive when the MessageManager is shutdown
  2. TimerThread didn’t use to wait indefinitely for the thread to shutdown

Based on what I found any Thread object that waits in its destructor indefinitely for the thread to finish that has a static instance may not finish if the OS kills the thread in this way.

I have to say however this behaviour surprised me and I feel like there is more to it.

Does it load the plugin to get the all the module info? If so interesting that it doesn’t suffer the same issue. I hadn’t gotten around to trying to load a dll from a different executable.

1 Like

Thank you for the update! Looking forward to the fix!

Nope, honestly I just ignore that part right now because it broke our production CI, while I needed other JUCE updates for other production bug fixes :melting_face:

I’m currently just copying the dll to the right location, rename it to vst3, and my installer build then grabs that binary instead of the entire bundle - which works well enough but definitely is a bandaid fix.

Until I found this thread here, I thought it was about the strangely formatted paths in the VST3 VS2022 project file, which produced some double-// in paths within the Pre/Postbuild scripts for me (I’m not sure about the details anymore, but the script paths passed to vst3helper didn’t seem correct at the time - i injected my own “vst3helper.exe” dummy and just printed all args passed in during build, which lead me to find broken-looking paths in the VS2022 file… but, again I’m not sure about this at all).

I thought all of this was happening to me, because I’m running the whole thing on a custom Git Bash setup on Windows, which is always a bit iffy with path-related stuff.

Now, reading all your comments regarding Timer usage during construction, could usage of Timer::callAfterDelay during construction be an issue?

This is not any specific code I’m using but rather a technique that I sometimes use when async resources may or may not be ready during construction and I have no way of getting better signals for some reason.

void MyClass::doSomethingAsync ()
{
    if ( ! tryTheThing () )
    {
        Timer::callAfterDelay (100, [ &, that = WeakReference<MyClass> (this) ]() {
            if (that != nullptr)
                that->doSomethingAsync ();
        });
    }
}

That’s really the only thing I do, that’s not just a Timer-derived class that I can think of, would that affect the vst3helper and get stuck the same like your examples?

No I don’t think so, the object it creates inherits from DeletedAtShutdown which should be guaranteed to be deleted before any threads get killed. I think it will only only be things with static lifetime duration. However, as I said above there could be more to this I don’t understand yet.

Just keep in mind that

  1. Creating the moduleinfo.json should speed up scanning your plugin in some DAWs
  2. If the plugin hangs in the helper it might also hang in other DAWs!

I think it could be any Thread inherited class that calls stop (-1) in it’s destructor (note Thread calls this in the base destructor but it would jassert first if it thinks the thread is “still running”.

Note you could probably use MessageManager::callAsync() for this? The Timer only happens on the message thread anyway. Unless there is a good reason to wait 100ms between attempts.

I’m not sure what causes the Timer issue but I just wanted to say that you can also attach a debugger to juce_vst3_helper and run it with the exact command arguments JUCE is running which has helped me solve similar issues when I was doing suspect things with object lifetimes.

The arguments are:

-create -version “${target_version_string}” -path “${product}” -output “${product}/Contents/Resources/moduleinfo.json”

With ${product} being something like:
“/Users/username/Library/Audio/Plug-Ins/VST3/MyPlugin.vst3”

2 Likes

I’m testing all plugins with pluginval during the build process, both before building an installer, as well as after installing them from the built installer - Testing strictness 10, 3 repeats, validate-in-process. Never had a single issue (especially not after the vst3helper issues started) and also haven’t gotten a single report on several thousand installations of more than 10 different plugins experiencing this vst3helper-freezing issue during our CI builds.

I never do that, I properly stop threads always, I’d never use -1 to stop and I also haven’t hit that assertion in years (I know the one you’re talking about - “bad karma if you reach this point”…).

This was only an abstract example. I do my Timer thing only where appropriate, when i don’t want the MessageThread to immediately call again, but when I DO want to wait first. I also only mentioned that technique with Timer up there because I do that in one place and it somewhat fit the issues being Timer-related :melting_face:

I’ll see if I can put together a test project going from a vanilla JUCE plugin over the weekend, that reproduces the issue and maybe I’ll also find what causes it on my end while doing that.

As I said, I “solved” it for our case by just not using the vst3helper for now and I have 0 issues just shipping the actual vst3 binaries without the moduleinfo; so I can’t dedicate a lot of time to it right now, but I’ll do my best to get you a project going where it’s reproducible.

1 Like

Running the helper in the debugger solved my issue a few weeks back in no time.
The quick succession of creating and destroying is something pluginval is unlikely to catch.
YMMV

Really? I was under the impression that’s what the cold open test does:

{
    beginTest ("Open plugin (cold)");
    StopwatchTimer sw;
    deletePluginAsync (testOpenPlugin (pd));
    logVerboseMessage ("\nTime taken to open plugin (cold): " + sw.getDescription());
}

The async deletion only grants the plugin an extra 150ms to delete on a thread, but i thought that’s only done for the frontend/timer to not block. The plugin itself is still immediately deleted after opening, the thread just waits for that to finish while the timer in this block can get done measuring the open-time.

Anyways, I’ll hook up a debugger to the vst3helper soonish and see what that yields, thanks for the tip.

So we kept the solution simple in the end hopefully this works for everyone.

However, word of caution, always stop a thread before any global or static object destructors run. Also prefer to wait for all global or static objects to be created before starting any threads.

It seems that this commit is causing compatibility problems when using PluginDoctor with any VST3 plugin that utilizes an AudioProcessorValueTreeState.

The specific assert that is triggered can be found on line 70 in the juce_Singleton.h file:

if (createdOnceAlready)
{
    // This means that the doNotRecreateAfterDeletion flag was set
    // and you tried to create the singleton more than once.
    jassertfalse;
    return nullptr;
}

Prior to this commit, this issue did not occur with PluginDoctor.

OK thanks I’ll take a look.

I wanted to add that the issue can also arise in PluginDoctor with plugins that don’t utilize the APVTS class.

PluginDoctor initially loads the VST3 DLL to memory, instantiates it, deletes that instance, and subsequently creates a new instance. However, it does not unload the VST3 DLL from memory after deleting the first and only instance, which is why the createdOnceAlready assertion gets triggered even if there were no instances prior.

With PluginDoctor, the VST3 DLL persists in memory even in the absence of any instances. On the other hand, DAWs like FL Studio unloads the VST3 DLL from memory if there are no remaining instances, thus avoiding the assertion. Additionally, with the Timer Thread now being a singleton instance with doNotRecreateAfterDeletion set to true, any host that behaves similarly to PluginDoctor will trigger the same assertion.

Below is the sequence of events that leads to the assertion with PluginDoctor:

When using the APVTS class:

  • Plugin DLL Loaded
  • startTimer Called by APVTS class
  • AudioProcessor Created
  • AudioProcessor Deleted
  • Timer Thread Deleted
  • startTimer Called by APVTS class
  • JUCE Assertion failure in juce_Singleton.h:74

Without APVTS class:

  • First Instance Created:

    • Plugin DLL Loaded
    • AudioProcessor Created
    • AudioProcessor Deleted
    • AudioProcessor Created
    • startTimer Called
    • Editor Created
    • Editor Deleted
    • startTimer Called
    • Editor Created
    • startTimer Called (4x)
  • First Instance Deleted:

    • Editor Deleted
    • AudioProcessor Deleted
    • Timer Thread Deleted
  • New Instance Created:

    • AudioProcessor Created
    • AudioProcessor Deleted
    • AudioProcessor Created
    • startTimer Called
    • JUCE Assertion failure in juce_Singleton.h:74
2 Likes