Pluginval: Real-time safety checking

Audio Plugins
1

Hi pluginval users. I’m in the process of adding real-time safety checks to pluginval.
macOS support is pretty much done but I’m still working on Linux (any support from proper Linux users welcome). You can see the progress and test it out for yourself on the feature/rtcheck branch.

To use it, either select the new "Realtime check mode"s from the “Options” menu in the UI. Or from the CLI add the following: --rtcheck enabled.

Doing this work has brought up a couple of interesting points that are worth a bit of discussion here:

1. We can’t check for mutexes

    #define RTC_REALTIME_CONTEXT_IF_ENABLED(realtimeCheckMode, blockNum)                                        \
      std::optional<rtc::realtime_context> rc;                                                                  \
                                                                                                                \
      if (realtimeCheckMode != RealtimeCheck::disabled)                                                         \
      {                                                                                                         \
          if (realtimeCheckMode != RealtimeCheck::relaxed || blockNum > 0)                                      \
          {                                                                                                     \
              rc.emplace();                                                                                     \
              rtc::disable_checks_for_thread (static_cast<uint64_t>(rtc::check_flags::pthread_mutex_lock)       \
                                              | static_cast<uint64_t>(rtc::check_flags::pthread_mutex_unlock)); \
          }                                                                                                     \
      }

If you look at the code that turns on real-time checking, you’ll see that I have to explicitly disable checks for mutex lock/unlock. The reason for this is the juce::CriticalSection in juce::AudioProcessor.

This is kind of annoying because if this lock is used correctly and never contented, this won’t actually be a system call and would technically be real-time safe. I can’t however disable checks in juce code so have to pessimise here which unfortunately means I can’t check for locks in your own code (which you probably don’t want to do unless you really know what you’re doing).

2. “relaxed” mode

You’ll also notice there are three modes “disabled”/“enabled” and “relaxed”. The first two are obvious.

It’s actually quite common in audio processing for the first block to allocate resources. This could be calculating filter coefficients, queues or even initialising thread-local storage (which can allocate depending on the size and platform).

For this reason I’ve added the “relaxed” option which ignores checks for the first processing block. If you’re non-realtime safe more often than that it’s probably a good indication something might be up.

Feel free to ask any questions (or offer up any Linux help) in this thread whilst I work to get this production ready. And thanks for the support that made this happen!

9 Likes
2

@juce_team, one thing that did come up whilst I was trying to add this to my existing dogfood tests running the juce plugins is that they can do a fair bit of allocating. I just wanted to get your stance on if there’s any chance these might be tightened up or if you just consider them “demos” and I should remove them from my tests?

Here’s an example:

Real-time violation: intercepted call to real-time unsafe function realloc in real-time context! Stack trace:
0   librtcheck.dylib                    0x0000000108292f5c _ZN3rtc14get_stacktraceEv + 80
 1   librtcheck.dylib                    0x0000000108292bdc _ZN3rtc32log_function_if_realtime_contextEPKc + 300
  2   librtcheck.dylib                    0x00000001082935b4 _Z44log_function_if_realtime_context_and_enabledN3rtc11check_flagsEPKc + 64
   3   librtcheck.dylib                    0x0000000108293658 wrap_realloc + 36
    4   DSPModulePluginDemo                 0x0000000123719ddc _ZZN4juce9HeapBlockIPNS_14MessageManager11MessageBaseELb0EE14reallocWrapperEPvmENKUlvE_clEv + 32
     5   DSPModulePluginDemo                 0x0000000123719d94 _ZN4juce9HeapBlockIPNS_14MessageManager11MessageBaseELb0EE7wrapperIZNS4_14reallocWrapperEPvmEUlvE_EEPS3_mOT_ + 56
      6   DSPModulePluginDemo                 0x0000000123719d50 _ZN4juce9HeapBlockIPNS_14MessageManager11MessageBaseELb0EE14reallocWrapperEPvm + 48
       7   DSPModulePluginDemo                 0x0000000123719d0c _ZN4juce9HeapBlockIPNS_14MessageManager11MessageBaseELb0EE7reallocImEEvT_m + 52
        8   DSPModulePluginDemo                 0x0000000123719c90 _ZN4juce9ArrayBaseIPNS_14MessageManager11MessageBaseENS_15CriticalSectionEE24setAllocatedSizeInternalEi + 40
         9   DSPModulePluginDemo                 0x0000000123719c38 _ZN4juce9ArrayBaseIPNS_14MessageManager11MessageBaseENS_15CriticalSectionEE16setAllocatedSizeEi + 176
          10  DSPModulePluginDemo                 0x000000012371a48c _ZN4juce9ArrayBaseIPNS_14MessageManager11MessageBaseENS_15CriticalSectionEE19ensureAllocatedSizeEi + 88
           11  DSPModulePluginDemo                 0x000000012371a378 _ZN4juce9ArrayBaseIPNS_14MessageManager11MessageBaseENS_15CriticalSectionEE7addImplIJRKS3_EEEvDpOT_ + 52
            12  DSPModulePluginDemo                 0x000000012371a338 _ZN4juce9ArrayBaseIPNS_14MessageManager11MessageBaseENS_15CriticalSectionEE3addERKS3_ + 32
             13  DSPModulePluginDemo                 0x000000012371a2b4 _ZN4juce21ReferenceCountedArrayINS_14MessageManager11MessageBaseENS_15CriticalSectionEE3addEPS2_ + 64
              14  DSPModulePluginDemo                 0x00000001236fc31c _ZN4juce12MessageQueue4postEPNS_14MessageManager11MessageBaseE + 36
               15  DSPModulePluginDemo                 0x00000001236f217c _ZN4juce14MessageManager24postMessageToSystemQueueEPNS0_11MessageBaseE + 120
                16  DSPModulePluginDemo                 0x00000001236f1c00 _ZN4juce14MessageManager11MessageBase4postEv + 92
                 17  DSPModulePluginDemo                 0x00000001236f44fc _ZN4juce12AsyncUpdater18triggerAsyncUpdateEv + 156
                  18  DSPModulePluginDemo                 0x0000000122d59924 _ZN4juce18ComponentRestarter7restartEi + 116
                   19  DSPModulePluginDemo                 0x0000000122d5526c _ZN4juce22JuceVST3EditController21audioProcessorChangedEPNS_14AudioProcessorERKNS_22AudioProcessorListener13ChangeDetailsE + 776
                    20  DSPModulePluginDemo                 0x00000001238445d4 _ZN4juce14AudioProcessor17updateHostDisplayERKNS_22AudioProcessorListener13ChangeDetailsE + 132
                     21  DSPModulePluginDemo                 0x0000000123844540 _ZN4juce14AudioProcessor17setLatencySamplesEi + 116
                      22  DSPModulePluginDemo                 0x0000000122db1b10 _ZN19DspModulePluginDemo12processBlockERN4juce11AudioBufferIfEERNS0_10MidiBufferE + 356
                       23  DSPModulePluginDemo                 0x0000000122d5e5c4 _ZN4juce17JuceVST3Component12processAudioIfEEvRN9Steinberg3Vst11ProcessDataE + 532
                        24  DSPModulePluginDemo                 0x0000000122d2b2b4 _ZN4juce17JuceVST3Component7processERN9Steinberg3Vst11ProcessDataE + 864
                         25  pluginval                           0x0000000105293d9c _ZN4juce26VST3PluginInstanceHeadless12processAudioIfEEvRNS_11AudioBufferIT_EERNS_10MidiBufferEN9Steinberg3Vst19SymbolicSampleSizesEb + 480
                          26  pluginval                           0x000000010526ebf4 _ZN4juce26VST3PluginInstanceHeadless12processBlockERNS_11AudioBufferIfEERNS_10MidiBufferE + 192
                           27  pluginval                           0x0000000104f7c940 _ZN14AutomationTest7runTestER11PluginTestsRN4juce19AudioPluginInstanceE + 2028
                            28  pluginval                           0x0000000104f6caa0 _ZN11PluginTests8testTypeERKN4juce17PluginDescriptionE + 2188
                             29  pluginval                           0x0000000104f6c070 _ZN11PluginTests7runTestEv + 1328
                              30  pluginval                           0x000000010513e040 _ZN4juce8UnitTest11performTestEPNS_14UnitTestRunnerE + 148
                               31  pluginval                           0x000000010513ed44 _ZN4juce14UnitTestRunner8runTestsERKNS_5ArrayIPNS_8UnitTestENS_20DummyCriticalSectionELi0EEEx + 424
                                32  pluginval                           0x0000000104f96c00 _Z8runTestsR11PluginTestsNSt3__18functionIFvRKN4juce6StringEEEE + 228
                                 33  pluginval                           0x0000000104f95f48 _Z8validateRKN4juce6StringEN11PluginTests7OptionsENSt3__18functionIFvS2_EEE + 156
                                  34  pluginval                           0x0000000104f95c0c _ZN14AsyncValidator3runEv + 156
                                   35  pluginval                           0x0000000104f95b64 _ZZN14AsyncValidatorC1ERKN4juce6StringEN11PluginTests7OptionsENSt3__18functionIFvS1_EEENS7_IFvS1_jEEENS7_IFvS3_EEEENKUlvE_clEv + 28
                                    36  pluginval                           0x0000000104f95b14 _ZNSt3__18__invokeB8ne180100IZN14AsyncValidatorC1ERKN4juce6StringEN11PluginTests7OptionsENS_8functionIFvS3_EEENS8_IFvS3_jEEENS8_IFvS5_EEEEUlvE_JEEEDTclclsr3stdE7declvalIT_EEspclsr3stdE7declvalIT0_EEEEOSG_DpOSH_ + 24
                                     37  pluginval                           0x0000000104f95af0 _ZNSt3__116__thread_executeB8ne180100INS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEZN14AsyncValidatorC1ERKN4juce6StringEN11PluginTests7OptionsENS_8functionIFvS8_EEENSD_IFvS8_jEEENSD_IFvSA_EEEEUlvE_JEJEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE + 28
                                      38  pluginval                           0x0000000104f95800 _ZNSt3__114__thread_proxyB8ne180100INS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEZN14AsyncValidatorC1ERKN4juce6StringEN11PluginTests7OptionsENS_8functionIFvS9_EEENSE_IFvS9_jEEENSE_IFvSB_EEEEUlvE_EEEEEPvSN_ + 84
                                       39  libsystem_pthread.dylib             0x0000000183bbf2e4 _pthread_start + 136
                                        40  libsystem_pthread.dylib             0x0000000183bba0fc thread_start + 8

Here’s the relevant parts demangled:

                                juce::HeapBlock::realloc<…>(unsigned long, unsigned long) juce_HeapBlock.h:299
                              juce::ArrayBase::setAllocatedSizeInternal(int) juce_ArrayBase.h:432
                            juce::ArrayBase::setAllocatedSize(int) juce_ArrayBase.h:232
                          juce::ArrayBase::ensureAllocatedSize(int) juce_ArrayBase.h:243
                        juce::ArrayBase::addImpl<…>(juce::MessageManager::MessageBase *const &) juce_ArrayBase.h:561
                      juce::ArrayBase::add(juce::MessageManager::MessageBase *const &) juce_ArrayBase.h:273
                    juce::ReferenceCountedArray::add(juce::MessageManager::MessageBase *) juce_ReferenceCountedArray.h:369
                  juce::MessageQueue::post(juce::MessageManager::MessageBase *) juce_MessageQueue_mac.h:67
                juce::MessageManager::postMessageToSystemQueue(juce::MessageManager::MessageBase *) juce_MessageManager_mac.mm:436
              juce::MessageManager::MessageBase::post() juce_MessageManager.cpp:85
          juce::AsyncUpdater::triggerAsyncUpdate() juce_AsyncUpdater.cpp:81
juce::ComponentRestarter::restart(int) juce_VST3Common.h:1651
        juce::JuceVST3EditController::audioProcessorChanged(juce::AudioProcessor *, const juce::AudioProcessorListener::ChangeDetails &) juce_audio_plugin_client_VST3.cpp:1561
      juce::AudioProcessor::updateHostDisplay(const juce::AudioProcessorListener::ChangeDetails &) juce_AudioProcessor.cpp:435
    juce::AudioProcessor::setLatencySamples(int) juce_AudioProcessor.cpp:420
  DspModulePluginDemo::processBlock(juce::AudioBuffer<…> &, juce::MidiBuffer &) DSPModulePluginDemo.h:194

In the DSPModulePluginDemo specifically, the fact that updates are async and can lead to allocations is highlighted:

                                        wrap_malloc(size_t) rtcheck.cpp:287
                                      <lambda>::operator()() const juce_HeapBlock.h:356
                                    juce::HeapBlock::wrapper<…>(unsigned long, <lambda> &&) juce_HeapBlock.h:343
                                  juce::HeapBlock::mallocWrapper(unsigned long) juce_HeapBlock.h:356
                                juce::HeapBlock::allocate<…>(unsigned long, bool) juce_HeapBlock.h:288
                              juce::AudioBuffer::setSize(int, int, bool, bool, bool) juce_AudioSampleBuffer.h:442
                            juce::dsp::DelayLine::setMaximumDelayInSamples(int) juce_DelayLine.cpp:97
                          juce::dsp::DelayLine::DelayLine(int) juce_DelayLine.cpp:52
                        juce::dsp::DelayLine::DelayLine(int) juce_DelayLine.cpp:47
                      juce::dsp::Chorus::prepare(const juce::dsp::ProcessSpec &) juce_Chorus.cpp:103
<lambda>::operator()<…>(auto &, auto) const juce_ProcessorChain.h:91
                    juce::dsp::detail::forEachInTuple<…>(auto &&, auto &&, std::integer_sequence<…>) juce_ProcessorChain.h:48
                  juce::dsp::detail::forEachInTuple<…>(auto &&, auto &&) juce_ProcessorChain.h:57
                juce::dsp::ProcessorChain::prepare(const juce::dsp::ProcessSpec &) juce_ProcessorChain.h:91
              DspModulePluginDemo::prepareToPlay(double, int) DSPModulePluginDemo.h:166
            DspModulePluginDemo::update() DSPModulePluginDemo.h:833
          DspModulePluginDemo::processBlock(juce::AudioBuffer<…> &, juce::MidiBuffer &) DSPModulePluginDemo.h:187
        juce::JuceVST3Component::processAudio<…>(Steinberg::Vst::ProcessData &) juce_audio_plugin_client_VST3.cpp:3853
      juce::JuceVST3Component::process(Steinberg::Vst::ProcessData &) juce_audio_plugin_client_VST3.cpp:3731
    juce::VST3PluginInstanceHeadless::processAudio<…>(juce::AudioBuffer<…> &, juce::MidiBuffer &, Steinberg::Vst::SymbolicSampleSizes, bool) juce_VST3PluginFormatImpl.h:2500
  juce::VST3PluginInstanceHeadless::processBlock(juce::AudioBuffer<…> &, juce::MidiBuffer &) juce_VST3PluginFormatImpl.h:2415
AutomationTest::runTest(PluginTests &, juce::AudioPluginInstance &) BasicTests.cpp:440

Anyway, I thought you might find those findings interesting.

3 Likes
3

Thank you for this amazing work!

Is there any way for the people, who are maintaining a fork of JUCE, to add a bypass around the `CriticalSection` in `AudioProcessor` so that everything else may still hit an assertion?
If so, could this be supplied as a patch file for example?

4

I think ideally we’d fix these up. Unfortunately, we can’t prioritise this work at the moment, as there’s a few other more serious issues that we need to fix, along with working on JUCE 9 in earnest. So, it might be best to remove these from your tests (or at least relax the tests in this area) for the time being, and revisit this once JUCE 9 is out.

5

For sanitizers you can add attributes to functions to make them be ignored, I wonder if there might be a similar solution for this problem

6

Unfortunately it’s not that straightforward for this as I’m using my own rtcheck library, not the RTSan with clang.
It was quicker to do it this way and I have a few features (like being able to disable specific checks) that were required to get it validating juce plugins (with juce hosting).

But yes, if it used RTSan, JUCE could fairly safely mark those uncontested uses of the CriticalSection with attributes.

There is a way this could be supported using rtcheck though and that would be for JUCE to do a

#if __has_include (<rtcheck.h>)
  #include <rtcheck.h>
  #define RTCHECK_NON_REALTIME rtc::non_realtime_context nrtc;
#else
  #define RTCHECK_NON_REALTIME 
#endif

But that would tie it strongly to the library.

1 Like