Yes, it does and that’s intentional. Essentially this acts in two ways:
- For plugin developers to ensure compatibility with JUCE based hosts (and also as a general testing/validation tool, used in addition to the format-specific tools)
- For JUCE based host developers (possibly including JUCE itself ) to test compatibility with plugins from all formats and built with all frameworks
It might be easiest if I run over some problem areas/workflows which we will be putting in to practice at Tracktion:
• For Host Developer QA Teams:
Testing plugin compatibility with your host can be an extremely tedious process. Simply getting hold of, updating, registering etc. plugins is extremely time consuming. At Tracktion, we have a contracted QA company (which I can highly recommend if anyone’s interested) so they have a huge database of plugins etc. already installed. I saw an opportunity here for them to periodically run
pluginval on their installed plugins, across multiple formats on multiple OSes all essentially automated. They can then give us the reports and let us know if any plugins fail tests which likely mean there will be compatibility problems with Tracktion/Waveform.
We can then get them to do targeted testing with the actual DAW.
Originally, I had this plan to automate the DAW in this way but it seemed better to do it as an external tool and open source it so it can be used by everyone.
• For Host Developer CI:
Similarly to above, if you have a CI setup, you can add a job to periodically run
pluginval on any installed plugins and find out compatibility problems early. This will most likely be used to spot regression testing.
I’ve tried very hard to make
pluginval as simple to clone or download binaries for precisely for these kind of CI workflows.
The main advantage of having QA run tests like these is that they will already have to spend time updating plugins etc. The set on the build server is likely to be much reduced simply due to time constraints/expiring licences (iLok anyone?) etc. The larger the company, the more likely you are to have a dedicated person to manage this. Unfortunately at Tracktion we’re not quite there yet so we’ll use a combination.
• For Plugin Developer CI:
If you’re a plugin developer, you’ll likely know that testing with the dozens of hosts out there is astoundingly time consuming.
pluginval aims to speed up that process by offering an automated way to check for silly mistakes, regressions and even things that are stricter than most hosts will require.
This should lead to a better overall level of software development and increased visibility in the space. Remember, with an open source tool we can add comments and log messages to explain to developers (particularly those new to the domain) why tests are failing and point them in the right direction.
I imagine fuzz testing here will come in extremely useful, particularly for things such as race conditions that can happen rarely and are very difficult to reproduce. Run some fuzzing for a few hours over your plugins and see what happens…
The other reason I like this CI-tooled approach is that developers (myself included) often respond better to machines telling them they’ve done something wrong. Getting a Jenkins email telling you you’ve broken the build due to some failing tests seems easier to take than a person telling you you’ve broken something. (Jenkins then nagging you every time you commit reinforces this message). You can then tidy it up, push it and no one else may even notice…
pluginval should be used in conjunction with existing tools, particularly on automated CI as they don’t take any extra development time to run.
• For Plugin Developers:
Again similarly to above but having an open source project that you can actually run with a debugger attached to see where tests are failing is super useful.
The other side to this which I’ve not discussed yet is essentially some kind of “recognised” or “compatible with” acknowledgement status. At Tracktion, we love to promote plugin developers making cool software and particularly if you take the time to make sure your plugins work with our DAW. We want to add a compatibility table to our own website in order to do this (our user base can then look at this to hopefully find interesting new plugins) but we know testing with hosts is time consuming. If you can send us a passing
pluginval report, we’re 95% sure your plugin will be compatible with us and we can add you to this list.
Who knows, in the future we might even add a GitHub badge!
• For Host End Users:
For users of our host, we wanted to provide a tool that they can run on problematic plugins to generate a log file which they can then send to us and plugin developers in order to give us a head start on fixing the problem.
We could even automate the collection of these so we can find the most popular, problematic plugins.
• For Plugin End Users:
Again, similarly to above but end users can use this tool to test their own installed plugins and generate reports to send to the plugin devs. This can save a lot of time and there are many users out there willing to do this providing the process is easy and quick enough.
I think that covers the main ways I see this being used but I’m sure there are many more which I’m excited to hear about!