Any way to speed up scanning VST3 bundles?

Scanning VST3 bundles can be incredibly slow if the bundle contains a large number of plugins. A large bundle of plugins can take 15 minutes or more to scan.

Something I’ve noticed is that despite being time-consuming, the process consumes only a small amount of CPU.

The time-consuming call is VSTComSmartPtr<Vst::IComponent>::initialize() in DescriptionFactory::findDescriptionsAndPerform()

Now, this can be optimised in non-bundle plugins by parallelising the scanning process. This is done in the JUCE AudioPluginHost example by creating a ThreadPool and scanning plugins in parallel.

However, with bundles, the scanning loop is internal to KnownPluginList::scanAndAddFile() so the caller has no control over it, and cannot parallelise.

Does anyone have any suggestions for optimising this? Would the JUCE team consider adding some parallelization inside of scanAndAddFile() to deal with bundles?

First, I wouldn’t advise to scan plugins in-process. While it may work for some/most plugins, you’ll inevitably get users reporting that your app crashes while scanning. So it’s better / safer to sandbox the scan of each plugin.

Sandboxing still allows you to parallelize multiple scans by spawning multiple child processes. That being said, I couldn’t see any significant speed improvement doing so, so I personally scan 1 plugin/shell at a time.

Now about the main issue, that is speeding up the scan of shells, I’m curious what shell are you talking about? Personally when I scan the Waves shell for example, some plugins in the shell can be slow, but most are lightning fast. By default Juce does not let you know what plugin is being scanned in the shell, so it can appear to be stuck, but if you tweak Juce to give you an update every time it goes to the next plugin, you’ll see how fast it goes. Of course it has to be ran in release mode. Debug will slow everything down (and also prevent some plugins to be properly scanned).

2 Likes

Have you checked out:

Rail

1 Like

Good points. Medium-term plan is to do the scan in a separate process as you suggest.

Very good point about debug builds, it hadn’t occurred to me that the difference would be significant. That said, in debug, when I timed it some plugins in the Waves bundle take some seconds to initialise. It doesn’t seem like much, but when you have 400 or so plugins to scan it really adds up.

It’s hard to imagine how scanning in parallel wouldn’t speed things up. I might profile it.

Yeah, I already have the fix in that thread, but I think that optimization relates to not rescanning the bundle for each plugin rather than the time spent scanning individual plugins inside the bundle.

I even had trouble to scan in the debugger at all for some plugins, I assume counter measures for crackers.
For me all woes went away in a release build and a plugin scanner in an extra process.

But that is of course anecdotal and of no scientific value.

Yes I also experience some plugins in the Waves shell taking seconds although others are super fast. But as far as I remember, this is across DAWs.
Also, the idea of scannings plugins in parallel is fine to explore, but you won’t be able to parallelize plugins that are in the same shell without hacking Juce.

When I did it, sure it allowed to scan other plugins while the shell scan was running but my conclusion was that on some machines it was actually slowing down the system & the user experience was poor, especially when plugins pop up auth dialogs or other crap. Then it’s all over the place with modal dialogs hiding each other etc. I would advise against it.

It’s not that anecdotal, you’re totally right. Some plugins will simply popup a message when a debugger is attached and won’t run. Others will just crash on you as a punishment.

And running multiple plugins in the same process increases the risk of crashes because it’s sadly quite common for plugins to corrupt memory. After all, there’s a reason why major DAWs are running plugins out of process nowadays :slightly_smiling_face:

The only optimization I added was to skip plugins which have already been scanned in the bundle if one of them fails and a rescan is started.

Rail