I’ve basically removed pluginval from my pipeline at the moment. That assert triggers every time, so it’s not usable for my needs.
1 Like
After posting I Googled on to check if I was not missing something simple.
But the results were disappointing:
- Last version of Pluginval is from September 2021
- The assert has been introduced in Juce in April 2021
The “Verified by Pluginval” page on Tracktion’s site puzzles me… is this for a different audience?
I think the idea is to create trust in JUCE by having a chance to pinpoint problems correctly. When you get trouble in a DAW there is always the blame game going on, is it the host or the plugin at fault.
By getting as many developers verifying their plugins with pluginval, the chances are good that a pluginval verified plugin is not the one at fault.
That’s the theory. So I think the audience is all three of them:
- plugin producers have a good indication that their plugin is correct
- plugin users get better/validated plugins
- host developers can point to plugins that fail pluginval
Also pluginval allows to validate AU plugins from any location without going through the whole install/auval cycle
I had a more pessimistic feeling about the audience: investors and other third parties (not developers) that need to get a very good impression of what Tracktion is up to…
But for developers: no maintenance in the last 9 months - really not sure what to think of this
Are you currently using it, @daniel ? If so, what required tweaks am I missing?
I think making a good impression towards all stake holders is due diligence and not necessarily a bad thing 
I used it and integrated it in a clients pipeine. But didn’t hear back from that client in a while, so I don’t know if they still use that pipeline. In my own projects I still called it manually from time to time, but lately more busy with two hosts…
1 Like
@johnplanetz I’ve been experiencing something similar, also with vst3 and the message manager lock that is taken in getText, please see BlockingMessage accessing MessageManager::Lock that went out of scope.
@reuk Can you point me to the fix that you made for this?
The underlying issue here is a general uncertainty about which calls are done from which threads not only normatively on each format, but also effectively on each host. It would be good to have this information gathered somewhere, as not everyone is fluent in all formats, let alone all hosts. There’s been many commits the last year or so to fix non-compliant corners of the VST3 wrapper and plugin format. These asserts make sense if you’re making a host, but I’m not sure they make sense for pluginval if some relevant host happens to be non-compliant. For example, normatively in VST3 get/setStateInformation and prepareToPlay are message-thread calls, but if you search the forum you’ll find some good uncertainty about it.
1 Like
There seems to be a lot of FUD appearing in this thread so please let me try and clear some things up.
Motivation for pluginval and tracktion.com Plugin Developer’s Page
pluginval was created for two main reasons, the first was to validate and stress test our own Tracktion plugins as there didn’t seem to be a suitable tool for this available. I wanted something that would create stack traces for problems in our CI pipeline.
The second was to provide a tool in Waveform that could catch compatibility issues with plugins and report the details to us. Replicating plugin incompatibilities is extremely difficult so using a tool that increases the chances of this should greatly reduce the amount of time required to debug.
Once I’d done this I realised this could be a useful tool for others so decided to open source it. We added the tracktion.com plugin developer’s page for two reasons:
Firstly, to give some promotion to companies that test with pluginval. I’ve always been a keen advocate of testing and wanted to promote it in the community and try and push a few sales towards the companies that embraced it. I’m not ashamed of this and it has nothing to do with investors or third parties.
Secondly, lofty ideal was that if there was a single tool that everyone in the community was using to validate plugins, it would be easier to create a “standard” for plugin development. One of the huge problems plugin APIs have suffered from historically is lack of specification so the idea was to create this spec using a tool.
pluginval Development
I primarily work on pluginval in my spare time. In September 2021 I had a baby so my spare time has dropped to almost 0. Additionally, we launched Waveform 12 in early 2022, several plugins and continue to work on a major new hardware product and a huge new version of Tracktion Engine 2.0. Unfortunately pluginval was the project with the lowest priority so received the least dev time.
pluginval is an open source project so if people have issues with it they can always contribute fixes. That is one way to expedite things. I should warn you though that as a project maintainer, I can’t just accept any PR that is opened. I have to review it and make sure it works, is in the style of the project, is a valid fix/feature and doesn’t break anything else. That’s hard to do when you only have a few minutes to look at something. I did this recently with a PR and it broke a lot of things as a side effect which @reuk spotted and had to create a new PR as a fix. I’ve been more cautious about PRs as a result since.
Project Structure
pluginval was always designed to only be built with the version of JUCE in the git submodule. The JUCE API changes too rapidly to ensure it always builds with the latest. Furthermore, as it’s a standalone app/tool, there shouldn’t be any reason you need to integrate it in to an existing project so shouldn’t get JUCE clashes. Yes, it could be updated to the most recent JUCE. It’s on my list.
Finally, assertions are tricky things to spot as they don’t trigger in release builds (and hence the pre-built binaries). This means they don’t trigger on our CI where the bulk of our pluginval usage happens and the above VST3 assertion has gone undetected for a while.
Additionally, pluginval was always created to stress test plugins and those grey areas where no threading is specified by the formats. I’m pretty sure this JUCE assertion appeared after I originally wrote pluginval and I hadn’t had time to go back and change the behaviour of the test. The main thing I haven’t been able to decide is if it should be called on the message thread for all plugin formats or specifically for VST3.
Additionally, I know the comment in JUCE says setupProcessing should only be called from the message thread but before this comment was added, the VST3 docs simply say this:
/** Called in disable state (setActive not called with true) before setProcessing is called and processing will begin. */
virtual tresult PLUGIN_API setupProcessing (ProcessSetup& setup) = 0;
Which doesn’t say anything about threading. (As an aside this is one of the huge benefits of clap, it actually specifies the threading contracts).
Finally
I’m going to try and spend a bit of time on pluginval this week to rectify the above issue and also change the internal architecture a bit. I have an idea that I’m hoping will reduce the complexity a bit.
12 Likes
Thank you, very much appreciated!
The Steinberg docs are a bit of a byzantine labyrinth, but this is actually specified and it’s quite simple -all functions are UI-thread except for setProcessing and process. It’s stated here, and graphically pictured in this set of diagrams. I suspect there may be more variety in the wild though…
1 Like
I’m running 0.3.0 successfully on MacOS/Windows in CI.
Linux is broken, due to this issue, which is unfortunate as it’s the cheapest to run regularly in CI…
I’ve mentioned before, but I’m more than willing to invest regular time to help keep the project maintained. Pluginval is a critical piece of my infrastructure and I appreciate it very much!
Happy to chat after you’ve had a chance to look at architecture and I can help more with CI and maintenance issues.
Yikes, hopefully this wasn’t my last one… 
This is why I hate out-of-code docs.
2 Likes
Hi @Sudara, your log says “pluginval v0.3.0 - JUCE v6.1.0”
I will try Pluginval with that Juce version
Thanks
That’s the JUCE version that pluginval uses on the develop branch (which is version 0.3.0). Your plugin doesn’t have to match that version, it’s just what the pluginval tool is built with.
-all functions are UI-thread except for setProcessing and process.
I remember having a long argument years ago with the author of Steinberg’s own Wavelab because I couldn’t initialise the JUCE message manager, because their host never called anything on the message thread!
Unfortunately when it comes to VST, there has been so little enforcement of even the few specs that they published that it’s pretty much the wild west in terms of how the DAWs behave. Your plugin just needs to be able to deal with whatever gets thrown at it.
4 Likes
unpopular opinion: In some cases when I discover that a DAW is violating the spec in some manner and causing my plugins to crash. I email the DAW vendor about it, and leave the plugins crashing. This makes the issue the DAWs problem (so it’s more likely to get fixed for good). Granted, I can get away with this because I’m responsible for 1000’s of plugins, so I have more leverage than many small developers.
Because every time I patch a bug that is really the DAW’s fault, I’m only passing the problem on to every other plugin vendor, who is gonna have to waste their time on the same issue.
Given enough time and apathy, the bug in the DAW might even become a de facto part of the standard! This is called ‘protocol decay’.
“Be liberal in what you accept, and conservative in what you send” - Jon Postel’s famous statement about APIs, and why it’s wrong.
2 Likes
Right, but in a lot of cases a crash might happen when neither the DAW or plugins is doing anything wrong, but where they’ve just both interpreted things slightly differently.
e.g. I remember cases where a DAW calls function foo() and function bar(), but if most DAWs happen to always call foo() before bar(), then you’ll inevitably get plugins that crash when they’re called in the opposite order, despite this seeming totally reasonable. Is that a fault of the plugin for expecting all DAWs to behave the same way, or the DAW for not following an unwritten rule about ordering? I’ve seen lots of cases where there’s no obvious mistake on either side, but both blame the other.
2 Likes
I think this kind of reasoning is entirely dependant on the bug, the plugin and how popular the host is. I’m sure hosts generally want to be as compatible as possible but as Jules says in some cases this is just impossible as plugins behave in a mutually exclusive way.
This leads to all kinds of “if plugin X do Y otherwise do Z” code which is just impossible to maintain. Worse, if the plugin ever changes behaviour to be more in line with others you’re now breaking that plugin explicitly in your code. Enter “if plugin X and version 123, do Y otherwise do Z” which is just ridiculous (and very real).
As I’ve said before this really stems from historical lack of well defined APIs, a historical lack of tooling, where there are conventions or rules, these are documented outside of code, not enforced by the code and one of the problems people forget: that mosts hosts or plugins aren’t written several times for each format, they all use some kind of wrapper on each end. If the APIs or rules don’t fit nicely in to the same wrapper in the same call order there will be inherent problems.
I honestly cant think of a way around this without making both hosts and plugins as robust as possible.
2 Likes
When testing a VST3 plugin from the command line, we occasionally get a crash with this stack trace:
Starting test: pluginval / Editor Automation...
*** FAILED: VALIDATION CRASHED WHILST VALIDATING /Library/Audio/Plug-Ins/VST3/CenterOne.vst3
0 pluginval 0x00000001008721a0 _ZN4juce11SystemStats17getStackBacktraceEv + 64
1 pluginval 0x00000001006bd9e6 _ZN12_GLOBAL__N_119getCrashLogContentsEv + 38
2 pluginval 0x00000001006bd98b _Z11getCrashLogv + 107
3 pluginval 0x00000001006b8122 _ZN20CommandLineValidator14connectionLostEv + 50
4 pluginval 0x00000001006d26e8 _ZNSt3__110__function6__funcIZN9Validator16ensureConnectionEvE3$_1NS_9allocatorIS3_EEFvvEEclEv + 56
5 pluginval 0x00000001006d2444 _ZN22ValidatorParentProcess20handleConnectionLostEv + 68
6 pluginval 0x000000010090f379 _ZN4juce12MessageQueue21runLoopSourceCallbackEPv + 57
7 CoreFoundation 0x00007ff80880f35a __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
8 CoreFoundation 0x00007ff80880f2c2 __CFRunLoopDoSource0 + 180
9 CoreFoundation 0x00007ff80880f040 __CFRunLoopDoSources0 + 242
10 CoreFoundation 0x00007ff80880da50 __CFRunLoopRun + 892
11 CoreFoundation 0x00007ff80880d014 CFRunLoopRunSpecific + 562
12 HIToolbox 0x00007ff8119215e6 RunCurrentEventLoopInMode + 292
13 HIToolbox 0x00007ff81192134a ReceiveNextEventCommon + 594
14 HIToolbox 0x00007ff8119210e5 _BlockUntilNextEventMatchingListInModeWithFilter + 70
15 AppKit 0x00007ff80b1831fd _DPSNextEvent + 927
16 AppKit 0x00007ff80b1818ba -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1394
17 AppKit 0x00007ff80b173f69 -[NSApplication run] + 586
18 pluginval 0x0000000100903274 _ZN4juce19JUCEApplicationBase4mainEv + 164
19 pluginval 0x0000000100903183 _ZN4juce19JUCEApplicationBase4mainEiPPKc + 83
20 dyld 0x000000020129851e start + 462
21 dyld 0x0000000201293000 dyld + 0
Binary Images:
0x1006ad000 pluginval
0x7ff808790000 CoreFoundation
0x7ff8118f3000 HIToolbox
0x7ff80b144000 AppKit
libc++abi: Pure virtual function called!
The failure appears to only happen from the command line, I’ve never had it happen when running from the GUI app (though this may be coincidental).
When I run pluginval --validate <plugin> --strictnessLevel 10 all tests pass, but when I add any of the randomize, repeat, samplerates, or blocksize options I get this crash.
Does this appear to be a bug in pluginval itself or in our plugin code? If it’s a bug in our code, does anyone have any suggestions for where to start looking?
Me and @sudara are actually in the middle of updating pluginval and the codepath in question has been removed. Running on the command line is always in-process now (because it only make sense to validate a single plugin at a time on the command line) so you’ll get more informative stack traces and no problems with child processes.
We’re tying up the final bits but should have a release ready in the next week or so. If you want an early look check out the feature/pluginval2 branch and build from source.
6 Likes