Problems using Tracktion engine within a plugin

Hi there,

I’m having some problems when trying to run tracktion engine within a plugin. Just to give a bit of context, I’m working on an app based on tracktion engine which I want to deploy on the ELK Audio OS/raspberry pi embedded platform (if interested check their talk at ADC 2019). A requirement for deploying in their platform is to have the app compiled as a plugin so a custom plugin host in ELK’s Audio OS can load the plugin and pass audio/midi in/out using the super low latency drivers they have developed.

I know from this highly related thread Using the Tracktion engine within a plugin? that running tracktion engine inside a plugin is currently possible (albeit with some limitations) and there is example code in the tracktion engine repository (EngineInPluginDemo project). Nevertheless, I’ve been trying to deploy this demo project compiled as VST2 in ELK board and I had no success. To discard issues particular to the ELK platform, I decided to try compiling and running the demo project in a Linux virtual machine and also on my host macOS machine first, but I’m also having problems. My goal is now to fix these issues for the Linux VM/macOS versions and then I’ll continue trying to deploy in ELK.

These are the results of the experiments I did while trying to get EngineInPluginDemo running:

macOS (vst2/3, au, stand alone)

  • all targets compile OK
  • works well as stand alone (I can connect MIDI keyboard and play and hear notes)
  • loads well as vst2 in Carla plugin host, but it does not seem to produce any notes when sending MIDI (I hear nothing)
  • loads well as vst2/vst3 in Bitwig DAW, but similarly to Carla, it does not seem to produce any notes when sending MIDI
  • fails AU validation when trying to load in Logic as AU. Error says something like ERROR: -4 IN CALL MusicDeviceMIDIEvent (link to full log file here)
  • using JUCE plugin host, vst3 fails initialization and can’t be loaded (in fact it makes plugin host crash)
  • using JUCE plugin host, AU version seems lo load fine but I don’t seem to be able to produce any notes when sending MIDI (I hear nothing)

linux VM - ubuntu studio 18.04 64bit (vst2, stand alone)

  • all targets compile OK
  • as stand alone I can run it and see normal plugin output log messages in the console without errors, but it freezes my VM right after engine initialization (most likely when it would start the engine or something like this). Maybe because the audio device in my VM is making JUCE go nuts.
  • as vst2, with Carla I’m not able to load the plugin as as scan operation seems to fail for the plugin and I can’t have it available in the list of plugins. Maybe the plugin is failing some sort of validation?
  • as vst2, when running in ELK’s custom plugin host (called Sushi), I see some errors in the console which end with a segmentation fault. Error says *** ERROR: Rogue call to triggerAndWaitForCallback() *** ERROR: triggerAndWaitForCallback() unable to complete (link to full log file here). This errors are indeed the same that I saw on my ELK board after compiling the plugin for the board and trying to run with sushi.
  • I could not test with JUCE’s AudioPluginHost because even if I can compile it ok on my VM, I can lot launch the app as it freezes my VM

I’ve been discussing with @stez-mind from the ELK team. They’re interested in getting this running so might join the conversation here as well.

Anyone from the tracktion engine team has ideas about what could be happening? Are you indeed able to compile as VST in linux and run the demo project in a host like Carla?

Any help will be much appreciated, thanks!

We have posted a roadmap for the engine here: https://github.com/Tracktion/tracktion_engine/blob/develop/ROADMAP.md

The main limitation currently is that the engine still uses several singletons which we need to remove. If your host is creating multiple copies of the plugin, that could be causing issues.

The second limitation is the PIP file that generates the .jucer file can’t specify the flags like “Is Synth”, “Plugin MIDI Input”, so you’ll need to manually set those flags.

Hi @G-Mon thanks for your answer.
I’m aware of the roadmap and it looks great, so hopefully all “engine inside plugin” issues will get solved soon :slight_smile: Nevertheless I though in its current form I should already be able to do something with it.

I checked the “is synth” and “plugin MIDI input” flags in projucer and recompiled. these are the results:

macOS

  • vst2/3 now work correcly and produce sound as expected when playing notes with the MIDI keyboard (tested Bitwig, Live, Carla)
  • AU still fails validation, although I don’t see errors in specific tests of the validator. Nevertheless, I can use the AU version just fine in Live and in Logic if I force using the plugin even with the validation step failed it still seems to work fine.

Therefore in macOS it almost works all good except for the AU validation fail which seems weird because the plugin actually works.

linux VM

Would be great if someone can try to build the EngineInPluginDemo project in Linux as VST and check if it works properly or you see the same errors I see. Is this what you tried @stez-mind ?

Thank you very much!

Do you have a stack trace for when the triggerAndWaitForCallback assertion triggers?
It looks like whatever is being called is taking more that 50ms to complete which is unusual.

I’m a bit of a newbie in debugging c++, but I run the command with strace -tt xxx and that is what I see: https://www.dropbox.com/s/ufza3hpys8y8p8j/strace_out.txt?dl=0 (not copying it here as it is too long)

I’m not sure how to read all of this but I see timeouts so what @dave96 says makes sense. Maybe I should try changing the 50ms limit?

Just putting a breakpoint in where the assertion is hit will stop the program at that point (assuming you’re running under a debugger like GDB) and allow you to print the stack trace with the bt command.

Thanks for the tip, I’ll look into it, probably not until next week though. To be able to debug with gdb I guess I’ll simply have to compile with make CONFIG=Debug (this will add necessary flags) and then I can set the breakpoint and run within gdb right?

[BTW a bit off topic, I wanted to check the tracktion engine talk @dave96 gave at the London Audio Developers Meetup (https://github.com/drowaudio/presentations#tracktion-engine) but the video seems to be down. Is there any possibility to recover it?]

I’ve been trying but I could not get output from gdb so far as I don’t know how to do it. The error with triggerAndWaitForCallback I only get it from the plugin version of the app, not the standalone. Therefore I guess I’d need to gdb into the plugin but I don’t know how to do that. I guess I need some specific plugin host that would allow me to do that?

EDIT: I did manage to debug the plugin (thanks @stez-mind for the tip), and this is the back trace that I get for the triggerAndWaitForCallback assertion

JUCE Assertion failure in tracktion_AsyncFunctionUtils.h:169

Thread 1 "sushi" received signal SIGTRAP, Trace/breakpoint trap.
0x00007ffff5acb187 in kill () at ../sysdeps/unix/syscall-template.S:78
78	in ../sysdeps/unix/syscall-template.S
(gdb) bt
#0  0x00007ffff5acb187 in kill () at ../sysdeps/unix/syscall-template.S:78
#1  0x00007fffee01a49a in tracktion_engine::MessageThreadCallback::triggerAndWaitForCallback() (this=0x7fffffffcd10)
    at ../../../../../modules/tracktion_engine/utilities/tracktion_AsyncFunctionUtils.h:169
#2  0x00007fffee01a77b in tracktion_engine::callBlocking(std::function<void ()>) (f=...) at ../../../../../modules/tracktion_engine/utilities/tracktion_AsyncFunctionUtils.h:207
#3  0x00007fffee33fe9c in tracktion_engine::PluginList::initialise(juce::ValueTree const&) (this=0x555555a2aa20, v=...)
    at ../../../../../modules/tracktion_engine/plugins/tracktion_PluginList.cpp:85
#4  0x00007fffee06ff1d in tracktion_engine::Edit::initialiseMasterPlugins() (this=0x5555559fce90) at ../../../../../modules/tracktion_engine/model/edit/tracktion_Edit.cpp:910
#5  0x00007fffee06e664 in tracktion_engine::Edit::initialise() (this=0x5555559fce90) at ../../../../../modules/tracktion_engine/model/edit/tracktion_Edit.cpp:683
#6  0x00007fffee06c930 in tracktion_engine::Edit::Edit(tracktion_engine::Edit::Options) (this=0x5555559fce90, options=...)
    at ../../../../../modules/tracktion_engine/model/edit/tracktion_Edit.cpp:538
#7  0x00007fffee06d238 in tracktion_engine::Edit::Edit(tracktion_engine::Engine&, juce::ValueTree, tracktion_engine::Edit::EditRole, tracktion_engine::Edit::LoadContext*, int) (this=0x5555559fce90, e=..., editState=..., role=tracktion_engine::Edit::forEditing, sourceLoadContext=0x0, numUndoLevelsToStore=0)
    at ../../../../../modules/tracktion_engine/model/edit/tracktion_Edit.cpp:561
#8  0x00007fffedf414ff in EngineInPluginDemo::EngineInPluginDemo() (this=0x5555559fcc70) at ../../Source/../../../EngineInPluginDemo.h:56
#9  0x00007fffedf38db7 in createPluginFilter() () at ../../Source/Main.cpp:15
#10 0x00007fffedb41f5a in createPluginFilterOfType(juce::AudioProcessor::WrapperType) (type=juce::AudioProcessor::wrapperType_VST)
    at ../../../../../modules/juce/modules/juce_audio_plugin_client/utility/juce_PluginUtilities.cpp:178
#11 0x00007fffedae9784 in (anonymous namespace)::pluginEntryPoint(Vst2::audioMasterCallback) (audioMaster=0x5555555f8600 <sushi::vst2::host_callback(AEffect*, int, int, long, void*, float)>) at ../../../../../modules/juce/modules/juce_audio_plugin_client/VST/juce_VST_Wrapper.cpp:2350
#12 0x00007fffedae9931 in VSTPluginMain(Vst2::audioMasterCallback) (audioMaster=0x5555555f8600 <sushi::vst2::host_callback(AEffect*, int, int, long, void*, float)>)
    at ../../../../../modules/juce/modules/juce_audio_plugin_client/VST/juce_VST_Wrapper.cpp:2409
#13 0x00005555555e9f1f in sushi::vst2::Vst2xWrapper::init(float) ()
#14 0x00005555555af4ca in sushi::engine::AudioEngine::add_plugin_to_track(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, sushi::engine::PluginType) ()
#15 0x00005555555c4d98 in sushi::jsonconfig::JsonConfigurator::_make_track(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&) ()
#16 0x00005555555c6ba4 in sushi::jsonconfig::JsonConfigurator::load_tracks() ()
#17 0x000055555557cb30 in main ()

In any case I think the error seems to be related to tracktion engine trying to open devices. This might be because of my Linux VM setup in which audio actually does not seem to work. I’ll try to do further experiments with either trying to make the audio work on my VM or testing with a raspberry pi as a native linux machine.

Aaaand this is the gdb trace I get when after all these triggerAndWaitForCallback errors the programs finishes with a segmentation fault:

Thread 1 "sushi" received signal SIGSEGV, Segmentation fault.
0x00007fffee2ad59a in tracktion_engine::InputDevice::isEnabled (this=0x0) at ../../../../../modules/tracktion_engine/playback/devices/tracktion_InputDevice.cpp:114
114	    return enabled;
(gdb) bt
#0  0x00007fffee2ad59a in tracktion_engine::InputDevice::isEnabled() const (this=0x0) at ../../../../../modules/tracktion_engine/playback/devices/tracktion_InputDevice.cpp:114
#1  0x00007fffee2972f3 in tracktion_engine::EditPlaybackContext::rebuildDeviceList() (this=0x555555a48350)
    at ../../../../../modules/tracktion_engine/playback/tracktion_EditPlaybackContext.cpp:106
#2  0x00007fffee29689d in tracktion_engine::EditPlaybackContext::EditPlaybackContext(tracktion_engine::TransportControl&) (this=0x555555a48350, tc=...)
    at ../../../../../modules/tracktion_engine/playback/tracktion_EditPlaybackContext.cpp:37
#3  0x00007fffee3015b9 in std::make_unique<tracktion_engine::EditPlaybackContext, tracktion_engine::TransportControl&>(tracktion_engine::TransportControl&) (__args#0=...)
    at /usr/include/c++/7/bits/unique_ptr.h:825
#4  0x00007fffee29f818 in tracktion_engine::TransportControl::ensureContextAllocated(bool) (this=0x555555a28940, alwaysReallocate=false)
    at ../../../../../modules/tracktion_engine/playback/tracktion_TransportControl.cpp:591
#5  0x00007fffee10b33a in tracktion_engine::Edit::TreeWatcher::valueTreePropertyChanged(juce::ValueTree&, juce::Identifier const&) (this=0x5555559c5d00, v=..., i=...)
    at ../../../../../modules/tracktion_engine/model/edit/tracktion_Edit.cpp:107
#6  0x00007fffedc5b2f4 in juce::ValueTree::SharedObject::sendPropertyChangeMessage(juce::Identifier const&, juce::ValueTree::Listener*)::{lambda(juce::ValueTree::Listener&)#1}::operator()(juce::ValueTree::Listener&) const (__closure=0x7fffffffce10, l=...) at ../../../../../modules/juce/modules/juce_data_structures/values/juce_ValueTree.cpp:101
#7  0x00007fffedc64763 in juce::ListenerList<juce::ValueTree::Listener, juce::Array<juce::ValueTree::Listener*, juce::DummyCriticalSection, 0> >::callExcluding<juce::ValueTree::SharedObject::sendPropertyChangeMessage(juce::Identifier const&, juce::ValueTree::Listener*)::{lambda(juce::ValueTree::Listener&)#1}&>(juce::ValueTree::Listener*, juce::ValueTree::SharedObject::sendPropertyChangeMessage(juce::Identifier const&, juce::ValueTree::Listener*)::{lambda(juce::ValueTree::Listener&)#1}&) (this=0x5555559c5d18, listenerToExclude=0x0, callback=...)
    at ../../../../../modules/juce/modules/juce_core/containers/juce_ListenerList.h:140
#8  0x00007fffedc61913 in juce::ValueTree::SharedObject::callListeners<juce::ValueTree::SharedObject::sendPropertyChangeMessage(juce::Identifier const&, juce::ValueTree::Listener*)::{lambda(juce::ValueTree::Listener&)#1}>(juce::ValueTree::Listener*, juce::ValueTree::SharedObject::sendPropertyChangeMessage(juce::Identifier const&, juce::ValueTree::Listener*)::{lambda(juce::ValueTree::Listener&)#1}) const (this=0x555555a259f0, listenerToExclude=0x0, fn=...) at ../../../../../modules/juce/modules/juce_data_structures/values/juce_ValueTree.cpp:86
#9  0x00007fffedc5ec82 in juce::ValueTree::SharedObject::callListenersForAllParents<juce::ValueTree::SharedObject::sendPropertyChangeMessage(juce::Identifier const&, juce::ValueTree::Listener*)::{lambda(juce::ValueTree::Listener&)#1}>(juce::ValueTree::Listener*, juce::ValueTree::SharedObject::sendPropertyChangeMessage(juce::Identifier const&, juce::ValueTree::Listener*)::{lambda(juce::ValueTree::Listener&)#1}) const (this=0x555555a25aa0, listenerToExclude=0x0, fn=...) at ../../../../../modules/juce/modules/juce_data_structures/values/juce_ValueTree.cpp:95
#10 0x00007fffedc5b354 in juce::ValueTree::SharedObject::sendPropertyChangeMessage(juce::Identifier const&, juce::ValueTree::Listener*) (this=0x555555a25aa0, property=..., listenerToExclude=0x0) at ../../../../../modules/juce/modules/juce_data_structures/values/juce_ValueTree.cpp:101
#11 0x00007fffedc5b7e5 in juce::ValueTree::SharedObject::setProperty(juce::Identifier const&, juce::var const&, juce::UndoManager*, juce::ValueTree::Listener*) (this=0x555555a25aa0, name=..., newValue=..., undoManager=0x0, listenerToExclude=0x0) at ../../../../../modules/juce/modules/juce_data_structures/values/juce_ValueTree.cpp:139
#12 0x00007fffedc54e38 in juce::ValueTree::setPropertyExcludingListener(juce::ValueTree::Listener*, juce::Identifier const&, juce::var const&, juce::UndoManager*) (this=0x5555559fd408, listenerToExclude=0x0, name=..., newValue=..., undoManager=0x0) at ../../../../../modules/juce/modules/juce_data_structures/values/juce_ValueTree.cpp:770
#13 0x00007fffedc54d56 in juce::ValueTree::setProperty(juce::Identifier const&, juce::var const&, juce::UndoManager*) (this=0x5555559fd408, name=..., newValue=..., undoManager=0x0)
    at ../../../../../modules/juce/modules/juce_data_structures/values/juce_ValueTree.cpp:760
#14 0x00007fffedf42ebc in juce::CachedValue<bool>::setValue(bool const&, juce::UndoManager*) (this=0x5555559fd400, newValue=@0x7fffffffd110: true, undoManagerToUse=0x0)
    at ../../../../../modules/juce/modules/juce_data_structures/values/juce_CachedValue.h:252
#15 0x00007fffedf4215b in juce::CachedValue<bool>::operator=(bool const&) (this=0x5555559fd400, newValue=@0x7fffffffd110: true)
    at ../../../../../modules/juce/modules/juce_data_structures/values/juce_CachedValue.h:242
#16 0x00007fffedf41c87 in EngineInPluginDemo::setupInputs() (this=0x5555559fcc70) at ../../Source/../../../EngineInPluginDemo.h:134
#17 0x00007fffedf41617 in EngineInPluginDemo::EngineInPluginDemo() (this=0x5555559fcc70) at ../../Source/../../../EngineInPluginDemo.h:60
#18 0x00007fffedf38db7 in createPluginFilter() () at ../../Source/Main.cpp:15

I may have missed it in the previous messages:
Is the problem due to the fact that

a) triggerAndWaitForCallback() doesn’t return at all (in which case that strongly indicates a deadlock)

or

b) triggerAndWaitForCallback() returns, but it exceeds a time threshold which triggers the jassert?

thanks for the answer. I’d say it is a), triggerAndWaitForCallback() does not return at all, but how could I verify that?

Set a breakpoint at the beginning of it (break triggerAndWaitForCallback in gdb I think, but I’m a bit rusty), then when it is hit, try to step out of the function with finish.

If it hangs there for long (minutes), then it’s most likely a deadlock

Ah no sorry, I have now looked at what triggerAndWaitForCallback() does and my suggestion above will not work to determine whether it’s a deadlock, but certainly setting a breakpoint at its beginning and stepping through it will give you a better understanding of what it tries to do and why it is failing.

I know nothing about the TracktionEngine but looking at your call stack it seems this code is called directly upon initialisation of your plug-in: it all happens inside createPluginFilter() after all.

If that is the case, perhaps the initialisation code is expected to be called by the message thread, but the triggerAndWaitForCallback() implementation fails to detect it?

At line 139 of “modules/tracktion_engine/utilities/tracktion_AsyncFunctionUtils.h”, it says

    if (juce::MessageManager::getInstance()->isThisTheMessageThread())
    {
        handleAsyncUpdate();
        return;
    }

maybe try changing that isThisTheMessageThread() with currentThreadHasLockedMessageManager() ?

Yeah, can you first try just changing the waiter.wait (50); on line 165 to waiter.wait (-1); and see what happens?

I have a feeling that this is happening because you’re running this in a straight main() function without having a JUCE message loop which means the async message can never be dispatched.

If this was running on the main JUCE message thread, callBlocking happens synchronously which would be fine.

Can you put a breakpoint in juce::MessageManager::getInstance() to see if it’s actually been initialised and started at this point in your plugin?

If that were the case, maybe a change in TracktionEngine code could take that into account and also make the call synchronously if MessageManager::getInstanceWithoutCreating() returns nullptr?

But that would assume the code to be single-threaded in that case, which could not be necessarily true…

Honestly, I can’t begin to comprehend what would need to change for Engine to work without a message loop. There’s thousands of places which rely on this behaviour for all sorts of reasons. If you don’t have a message loop, I just don’t think that’s a feasible use-case for us to support at the moment.

Thanks for your answers @dave96 @yfede. These are the results of what @dave96 suggested:

  • I tried compiling with waiter.wait (-1) and in this case the program hangs forever. Hence it looks like triggerAndWaitForCallback() never returns.

  • Setting breakpoint to juce::MessageManager::getInstance() stops the debugger before the app hangs infinitely (because I’m still using waiter.wait (-1)). In fact it calls the function a number of times. This is the stack trace the first time getInstance is called:

Thread 5 "sushi" hit Breakpoint 1, juce::MessageManager::getInstance ()
    at ../../../../../modules/juce/modules/juce_events/messages/juce_MessageManager.cpp:47
47	    if (instance == nullptr)
(gdb) bt
#0  0x00007ffff1d31091 in juce::MessageManager::getInstance() ()
    at ../../../../../modules/juce/modules/juce_events/messages/juce_MessageManager.cpp:47
#1  0x00007ffff1d32046 in juce::initialiseJuce_GUI() ()
    at ../../../../../modules/juce/modules/juce_events/messages/juce_MessageManager.cpp:461
#2  0x00007ffff1bb222d in SharedMessageThread::run() (this=0x5555559fc800)
    at ../../../../../modules/juce/modules/juce_audio_plugin_client/VST/juce_VST_Wrapper.cpp:239
#3  0x00007ffff1c9402f in juce::Thread::threadEntryPoint() (this=0x5555559fc800)
    at ../../../../../modules/juce/modules/juce_core/threads/juce_Thread.cpp:96
#4  0x00007ffff1c9411c in juce::juce_threadEntryPoint(void*) (userData=0x5555559fc800)
    at ../../../../../modules/juce/modules/juce_core/threads/juce_Thread.cpp:118
#5  0x00007ffff1cb821c in juce::threadEntryProc(void*) (userData=0x5555559fc800)
    at ../../../../../modules/juce/modules/juce_core/native/juce_posix_SharedCode.h:923
#6  0x00007ffff71e66db in start_thread (arg=0x7fffd3526700) at pthread_create.c:463
#7  0x00007ffff5bad88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

This I did not try yet. What would be the purpose of this?

Also about the discussion of the message loop I don’t know much of it, but I assume this is something in the JUCE architecture. Here I’m trying to run the EngineInPlugin example demo app of traction as a plugin (VST2 ideally) inside a host, so I’m not modifying internals or hacking the Engine/JUCE in any way. I’m running this in a Linux VM, but I’ve been reported same problem happening in native linux machine. I’m currently also setting up a raspberry pi with raspbian as a native linux machine so I can repeat the same tests there and see what happens.

What Linux distro are you using?

With the wait time set to -1, if you pause the debugger at that point and print out the stack traces of all the other threads, is there something else blocking the message queue?

I’ll be honest, I’m not 100% sure if we’ve tried running the Engine as plugin on a Linux test machine yet. It’s still very much a work-in-progress. We do have some time allocated, hopefully at the at the end of Jan to complete work on the Engine as a plugin though.

For my Linux VM I’m using ubuntu studio (18.04).

However I’ve been finally able to make some experiments with native linux with a raspberry pi and Raspbian buster (latest version). These are my findings:

  • EngineInPluginDemo project works well as standalone. I assume in my VM I could not get it working because my audio was not configured properly.

  • EngineInPluginDemo as plugin (VST2) still produces the same triggerAndWaitForCallback() error. This is testing with the JUCE AudioPluingHost which I was able to compile (with VST2 support).

I guess this is both good news and bad news. Bad news because it does not work. Good news because the problem does not seem to be platform/distro specific (I’ve seen same problem with pi+AudioPluginHost, ubuntu+Carla and ubuntu+Sushi) and is easy to reproduce. Therefore I’m pretty sure the awesome Tracktion team will fix it whenever you have some time to polish the Tracktion-inside-plugin compatibility.

Please let me know if there’s any other thing you think I could try as a workaround solution. Thanks for your answers and your time! @G-Mon @dave96 @yfede