Hi everyone,
I just spent the last 4 hours hunting an extremely non-deterministic memory corruption bug in my plugin. The root cause was that Pro Tools 12 will call setParameter on an automation thread while processBlock is called on one of their audio processing threads, which in my case lead to memory corruption inside the parameter class.
The resolution is super easy: Make sure you lock the callback lock in setParameter, e.g.:
const ScopedLock sl (getCallbackLock());
That does not sound like a good solution; locking the audio processor like that does not scale at all.
It is also against the basic guidelines given by JUCE regarding gaining locks that could block processing, i.e. don’t do that.
I am curious, though, to see what the JUCE team suggests to deal with this
It seems the best solution would be to keep two sets of parameters around. One for the setParameter getParameter functions and one for the use inside processBlock. Or maybe one should make a copy of all current parameter values for processBlock and then use that copy while processing. But in any case, it seems that setParameter, getParameter and processBlock interactions depend highly on the plug-in standard and also on the host.
For example, AAX calls everything in parallel using multiple automation and processing threads, so the GUI redraw, setParameter and processBlock might all run concurrently. On the other hand, some VST2 hosts call setParameter exclusively from the real-time thread, meaning it can run concurrently with the GUI, but not with processBlock. Other VST2 hosts call setParameter exclusively from the GUI thread, so that then it can again run concurrently with processBlock, but not with the GUI redraw.
But my non-representative testing shows that just locking out setParameter calls while processBlock is running works just fine for Cubase, Nuendo, Pro Tools, Logic and Ableton. So while I agree that this solution won’t scale to millions of automation parameters, I also don’t detect any performance degradation.
In the long run, both VST3 and AAX aim to completely decouple GUI from processing. So in the long run, I guess the JUCE framework should get support for an independent “plugin parameter object” and every PluginProcessor then gets split up into the GUI component writing into the plugin parameter object and the real-time component reading from the plugin parameter object. This architecture is needed especially if you want to support hardware DSP execution in AAX. But AFAIK, even then one would need to have some sort of synchronization for the plugin parameter object to prevent the real-time component from reading a partially updated state.
It isn’t really about performance degradation as much as it is about drop-outs and priority inversion. If you haven’t yet experienced being on a stage, and having some plugin lag even once, be sure as hell it will never be used again (live). Or during tracking of a perfect take.
If you keep your plugin’s mutable state in pieces of 8 bytes, you are guaranteed lock free atomic reads and writes on all x86-compliant architechtures by default since the first 32-bit Intel processor. This means that any thread can write and read concurrently to the value without problems. I cannot speak for all other platforms.
Are you sure the automation runs off another thread? In that case, Pro-tools is unable to provide deterministic rendering and sample-accurate automation.
The description in the last paragraph is what I do.
Thanks for bringing up the priority inversion problem. I agree that I’ll need to consider that. Upon closer examination, it now seems that I would need 3 copies of my parameter data:
I could of course lock out getParameter and setParameter while the GUI is redrawing, but that leads to noticeable stuttering. So as-is, locking out processBlock has less negative impact than locking the main GUI thread.
And yes, I just checked again. Both on Windows and on Mac I get concurrent calls to processBlock and setParameter from different threads. I tested with Pro Tools 12.4.0 on Mac and 12.5.2 on Windows.
BTW thanks for the hint with the 8-byte memory access. But my problem isn’t so much that I get partially written parameters, but that I need to interpolate between old and new parameter values to prevent audible klick noises. My plugin does 3d panning with reverb, so just repositioning an object halfway through the audio buffer without any correction or interpolation leads to sound problems. So at the beginning of processBlock, I need to precalculate the object trajectories and apply the appropriate panning power law. And of course, those trajectories break if the parameters change while I’m doing my calculation.
On the GUI side I get a similar problem if I don’t use explicit synchronization, in that parameters will flicker in the GUI, for example when the redraw happens after the LR coordinate has been updated but before FR was written.
Also, my impression is that they are not even trying to make automation precise. For example, here’s a quote from the AAX spec: “any parameter updates from automation will be reflected in this context a little bit ahead of time (~21 ms at 44.1 kHz.)” So as the plugin developer, should I just assume stuff always arrives 21ms too early? Or is that my hint that nobody really cares that much about sample-precise automation?
Thank you for bringing that up. I have a similar problem, I also use the parameter for geometric calculations, but I have only one parameter to synchronize. So I do an averaging over the last few values (16 blocks…), which works ok. I just can’t program abrupt movements, but it’s good enough for this specific problem.
I was about to write a test plug to meassure accuracy, I think I can skip that with this information. To bad.
Did you raise this question in the avid forum? I will have a look there too…
Well, there should only be one value, that the processor holds. The GUI sends a message to change it, and recieves a notification back if it changes, thus syncing itself. The host automation, if run in another thread, has to send messages as well.
The processor either chooses to consume all changes once it starts rendering or to read directly from the value the others write to. The latter works ok for simple stuff like gain values.
The former model essentially creates two views of the data, one being the parameter values that can be concurrently updated and some internal inaccesible view to the processor. This method is completely safe and scales without problem, and it basically works like a messaging system actually.
[quote=“fxtentacle, post:6, topic:18216”]That, of course, leads to really annoying synchronization problems. What if my user changes a parameter in the GUI concurrently to Pro Tools updating that parameter with setParameter.[/quote]This is handled in every host and plugin format I know of by turning off automation after the host has been notified of a parameter changing from the outside, thus no problem.
Have an internal variable representing the current, and on each process() call (or each 32th sample or whatever) you update a target value from the outside, that you constantly interpolate “current” towards. Usually the automation is lagged one update behind, such that no “guessing” has to be performed.
[quote]On the GUI side I get a similar problem if I don’t use explicit synchronization, in that parameters will flicker in the GUI, for example when the redraw happens after the LR coordinate has been updated but before FR was written.[/quote]To solve this problem you need to use a lock free queue/ringbuffer, where the processor produces values that the GUI can consume as whole.
[quote]Also, my impression is that they are not even trying to make automation precise. For example, here’s a quote from the AAX spec: “any parameter updates from automation will be reflected in this context a little bit ahead of time (~21 ms at 44.1 kHz.)” So as the plugin developer, should I just assume stuff always arrives 21ms too early? Or is that my hint that nobody really cares that much about sample-precise automation?
[/quote]
Just remembered this topic: Automation firing early in Pro Tools 11 - #9 by chkn
