Multithreading oddities with millisecond counter in ProTools

We hade a bug report that our RTAS plugin didn’t work reliably in ProTools, giving the infamous 6101 error - even at large block sizes.

The problem occurred on an 8 core Mac Pro System, and we werent’t able to reproduce it on a CoreDuo Whitebook.

But the errors showed up as soon as we tested it in ProTools 9.0.5 on a i5 Macbook Pro, though only if we were using strict settings for the Playback Engine (64 samples buffer size, report errors when cpu sage hits 40%). We did a quite a few builds with different part of the plugin enabled and had some stunning results:

  • the error message often came up at the same time where an overload indicator in the gui was repainted
  • the plugin runs stable if the plugin GUI is never opened
  • the plugin runs mostly stable if the ::processNextBlock() of the plugin is totally disabled, but the GUI is opened and rapidly repainted with random values

So that looked like some concurrency issue with the audio and gui thread, so I carefully stepped through every single line within the audio callback to check if I missed a lock, allocation or change-message.

In the end it turned out changeing one line that looked harmless to me made the big difference:
replacing Time::getApproximateMillisecondCounter() with Time::getMillisecondCounter()

What’s going on here? Isn’t the Approximate variant the one that is supposed to return immediately with a somewhat outdated version of the counter? Can anyone give me a reasonable explanation for this? Especially on why this call doesn’t do any bad if the plugin editor is closed.

Time::getApproximateMillisecondCounter() will definitely not cause any threading problems, but perhaps adding a call to Time::getMillisecondCounter() is the thing that’s fixing the problem…? Have you tried adding a dummy call to getMillisecondCounter() in addition to the current call of getApproximateMillisecondCounter() ?

The reason I say this is that you may be in a situation where getApproximateMillisecondCounter is never getting its time updated because nothing ever calls getMillisecondCounter(), so maybe a counter somewhere that relies on getApproximateMillisecondCounter is getting stuck…?

Good point, I was assuming that getMillisecondCounter() was called anyway by the Timers that trigger the gui repaints. I’ll check this out tomorrow.

But I can’t see any way the audio thread might get stuck due to a wrong value of the counter. All it does is to pass it over a a few method calls, and store the value in a member variable (where timers will read it). Still it seems to trigger the same ProTools errors that you usually get when you’re allocating or locking in the audio thread.

Yeah, having the timer thread running should take care of updating it, and I also can’t think of any audio code that would get stuck in that way, but can’t think of anything else to suggest!

This mystery is solved: the actual call to ::getApproximateMillisecondCounter() was indeed harmless. There was a race condition when the gui thread compared its current time to the one stored by the audio thread. Some bit of code was assuming that the current time is always more recent. The actual protools trouble was caused by code that should have be only executed in case the host stopped calling the audio-callback for a while.

This ran fine for years, but with smaller buffer sizes and multiple cores, it went wrong at random times, depending on when other parts of the plugin called ::getMillisecondCounter().

Yeesh, that’s a tricky one! Glad you nailed it!