Can anyone explain what’s happening here. It’s a bit unclear what’s actually failing and why! The first copy step that claims to have an error seems to actually work from the command line.
edit: found it - this actually a steinberg tool wrapped up with the juce_ prefix and apparently has some error conditions which don’t seem to display an error message … arrrrgh
edit2: so I’ve got the debugger on Steinbergs tool which apparently exits with no error message if it can’t load the VST3 DLL. So now I’m debugging that problem, we are getting error 126 but without any obvious reason.
Right, so we use a copy protection system and cannot load the VST3 DLL until after a postbuild step has been completed.
Unfortunately the new build system tries to load the DLL during the build during the creation of the module json file for the VST3. How can we disable this step and do it much later?
Yeah, we’ve rolled back JUCE for now. But will need to fix this somehow. Will our post-build command run before the manifest generateion? It doesn’t look like it from your code example, looks like it’s appended?
Might be good to be able to turn this off and we can just do the VST3 construction in our wrapper script that does everything else?
We are considering the options, but generally think that there is a strong reason for supporting the manifest feature of VST3, and if developers saw the added benefit it brings, they would want to utilise it even if it requires modifications to the build process.
[1]: The manifest contains all the information that a DAW would want to find out when scanning a plugin first. This way, instead of scanning the plugin, it can just read a .json file, which is many times faster. In order to generate this .json file, we are scanning the plugin right after building it to populate the contents of the manifest.
Oh yeah - I’m in favour of the manifest generation, I just need to move it to after we do some pre-processing of the DLL. I think we can move the virtualization step into the projucer for our projucer based projects. Future projects are all CMAKE so will need to look at that too.
Is pkgScript the script we specify in the projucer?
The line with pkgScript… etc. are part of the “extra build steps”, and they all get appended after the post build script, which is not part of that excerpt.
This appending is done in the getPostBuildSteps() function in the same file.
This way, instead of scanning the plugin, it can just read a .json file, which is many times faster. In order to generate this .json file, we are scanning the plugin right after building it to populate the contents of the manifest.
I have doubts about this. In theory, the operating system should only be loading the minimal amount of data from the DLL in order to execute a single entry point to query for plug-in information. This is all operating system code which has been optimized to an extreme over many decades. Switching to JSON should load similar amounts of data, but now it relies on enumerating the folder contents, loading and parsing JSON which is relatively slow. Not to mention the complexity of people having to implement this all across the industry.
Are there benchmarks somewhere from a trusted source where this is actually verified to be faster in the real world?
Sorry, but do you have any sources for that? How would that even work? When loading a DLL, how would the operating system know which parts are needed to execute specific functions? I’ve never heard such a claim.
The operating system typically memory maps segments of the DLL into virtual address space (there is no disk access at this point to the associated segments). Page faults are triggered in response to access to these regions of memory which causes the associated data to be read from disk. This access could be either instruction pointer (jump to address) or read/write access. It’s all transparent from the perspective of the user mode program.
Source is just computer architecture and operating system experience. For example, I wrote an Xbox emulator decades ago which involved emulating the PE file format which is what is used by Windows DLLs/EXEs. Also had to use memory access faults to emulate various GPU instructions.
Again, operating systems have been along for a long time and they are very well optimized. It’s hard to out-do them with this sort of thing, so that’s why I’m curious to see if there are benchmarks and a deeper explanation for why this is an improvement.
What happens in practice is when loading plugins (dlls), all the static/global data, along with fun stuff like resource loading, copy protection, etc, gets initialized.
That part takes a whole lot longer than parsing JSON, which you will clearly see if you have a lot of plugins loaded, some of them even bring up copy protection windows/etc during scanning sometimes.
The Clap plugin format went that route and while it generally does result in fast scanning of the plugins, it is also possible for plugins to negate the benefits of the system.
For example, a more complicated plugin may scans its own subplugins with the associated copy protection/network access etc when calling the Clap entry point function that should just return the main plugin’s metadata, resulting in potentially several seconds of slowdown. (Not a theoretical scenario, I already encountered that with a plugin while making my Clap host.)
Thanks for the feedback. We’ve added an update to the CMake integration to allow postponing the manifest generation step so that other custom steps can run first:
The manifest generator is driving me absolutely crazy. Really hard to debug errors with it.
Maybe this is a code signing issue again, whichI thought we’d squashed, but there’s no way of telling! And it’s fairly tricky to stop it deleting the vst3 after it fails which is half of the problem with debugging this.
Sorry, being an idiot - we are using that, but then calling enable_vst3_manifest step function, so if I drop that i’ll get a working plugin but without the speedy manifest. Let me try it.
