Step sequencer working on macOS build but not when running in ELK

Hi,

I’m building a tracktion engine-based app which runs as a VST plugin in an embedded device (raspberry pi) using the ELK Audio OS system. I have been posting many questions here about issues I’ve been finding so you might know about that :slight_smile: I pretty much got everything up and running, but there are still a few quirks that need to be solved. Here is one:

My app has one track which contains a step sequencer. I use the same code as in the step sequencer demo of the tracktion engine repository. When I build my project on macOS it works as expected, step sequencer sounding normally. However, when I build for the ELK platform the step sequencer does not produce any sound. In the same project I have other tracks playing audio clips at the same time which work normally on both platforms. Also I made sure the problem is not that the wav files can’t be loaded in the sampler because these files are bundled as binary data and one of them I use it as well as an audio clip in another track and it plays normally.

Any ideas of what sort of thing could be causing that? How could I further debug what is going on?

Just a quick update with a couple of experiments that I did, in case it sheds some light…

  • I inspect the StepClip object using a timer that periodically runs after all objects are created and I can see the different channels and patters of the clip set up correctly
  • I inspect the SamplerPlugin object also in that same timer and I can see sounds being loaded correctly, I can print sound sources, duration, the note they’re mapped to and the other different sampler sound properties. This inspection tells me the path of the audio file(s) loaded (e.g. /tmp/temp_eefaf901.kick.wav). I can download this file from the ELK board to my dev computer and play it as expected.
  • I try to trigger a SamplerPlugin note manually (I use the playNotes method and pass a BigInteger with the bits corresponding to the loaded sounds set). This works fine on macOS but again fails (I hear nothing) in my ELK setup. I added a std::cout << "test" in the playNotes method of SamplerPlugin and it is properly reached in both paltforms.
  • I checked that the track that has the step sequencer is routed to the same audio output as all other tracks that work fine on both platforms.

I don’t have an Elk to test with so it’s a bit difficult to know what could be going wrong, especially as this code works on other platforms.

Are you able to use a breakpoint and step through the code? You probably want to step through SamplerPlugin::playNotes to see if the note actually gets added.

The only other thing I can think of is a read-permissions problem trying to read the file?

The read permissions issue I did not think of, but I don’t think it is the problem because in audio clips I can load and play files normally and when I inspect the sampler object I can see loaded sound properties like duration (which I guess would depend on successfully reading the file).

I have not tried setting beeakpoints yet, but I set some debug prints and I know I reach this line https://github.com/Tracktion/tracktion_engine/blob/e7e388efdccd86cb767b72f1532d513d4be94c44/modules/tracktion_engine/plugins/effects/tracktion_SamplerPlugin.cpp#L259

Did not check if note is indeed added to “playingNotes”, but the “ playingNotes.add” function seems to be called according to my prints. I’ll add more checks here.

That would indicate the notes are getting added. Next step is to step through SamplerPlugin::applyToBuffer to see if the playingNotes are used to generate the sound or if they’re getting dropped somewhere due to perhaps a missing MIDI buffer? (I.e. are you sure MIDI is actually getting passed to the plugin?)

That is interesting. However if MIDI was the issue, I whould be hearing the sample when I manually call SamplerPlugin.playNotes right? (bacause I assume midi is used only from stepclip to samplerplugin). When manually playing the notes it works on macos but not elk.

I’ll investigate applyToBuffer and report, many thanks!

You’re manually calling playNotes? Are you sure the audio is passing through the plugin at all?
First thing I’d try is to set a breakpoint in SamplerPlugin::applyToBuffer to see if it’s getting played back at all… It should be fairly obvious to see what’s going on if you can step trough that.

The manual call to “playNotes” is only for debugging purposes. Normally I would let the stepclip handle that. But the thing is it works on macOS but not elk :frowning: I’ll debug the “applyToBuffer” bit, thanks!

I’ve been debugging SamplerPlugin::applyToBuffer calls, here my findings:

Therefore it looks like SamplerPlugin is receiving the notes correctly and playing them as expected. I guess this is the line failing:

sn->addNextBlock (*fc.destBuffer, fc.bufferStartSample, fc.bufferNumSamples);

I guess something is wrong with fc.destBuffer (the AudioRenderContext). Remember that I have other tracks in the same edit with WaveClips (i think it is called like that) that play just fine. Also that in macOS the same code does indeed work fine. Any ideas for checking on AudioRenderContext?

Thanks!

EDIT: just as a quick test, if I print track->getOutput().getOutputName() for all my tracks in the edit, they all show the same name default audio output. Just to discard the idea that step sequencer track is routed elsewhere. Also printed track->isMuted(true) and all show 0 (not muted)

Hah, I did some more debugging and I found something interesting. I set up more debugging prints, this time at the point where samples are actually copied to the outBuffer: https://github.com/Tracktion/tracktion_engine/blob/e7e388efdccd86cb767b72f1532d513d4be94c44/modules/tracktion_engine/plugins/effects/tracktion_SamplerPlugin.cpp#L69-L74

I’m printing the first 20 samples of both the audioData buffer (which should have the step sequencer sample) and the outBuffer (which I guess is what is passed to the next plugin in the chain). On macOS I see non-zero values in both buffers (which makes sense):

...
void addNextBlock - writing channel 1 to output buffer, offset 10856
   inBuffer: 4.44651e-05, 3.96967e-05, 3.71933e-05, 3.23057e-05, 2.75373e-05, 2.2769e-05, 1.80006e-05, 1.32322e-05, 8.46386e-06, 3.69549e-06, 1.07288e-06, -3.69549e-06, -8.46386e-06, -1.32322e-05, -1.80006e-05, -2.06232e-05, -2.53916e-05, -3.016e-05, -3.50475e-05, -3.75509e-05, 
   outBuffer: 2.84397e-05, 2.53898e-05, 2.37887e-05, 2.06626e-05, 1.76128e-05, 1.45629e-05, 1.15131e-05, 8.46328e-06, 5.41345e-06, 2.36362e-06, 6.86212e-07, -2.36362e-06, -5.41345e-06, -8.46328e-06, -1.15131e-05, -1.31905e-05, -1.62403e-05, -1.92902e-05, -2.24163e-05, -2.40174e-05, 
void addNextBlock - writing channel 0 to output buffer, offset 10856
   inBuffer: 4.44651e-05, 3.96967e-05, 3.71933e-05, 3.23057e-05, 2.75373e-05, 2.2769e-05, 1.80006e-05, 1.32322e-05, 8.46386e-06, 3.69549e-06, 1.07288e-06, -3.69549e-06, -8.46386e-06, -1.32322e-05, -1.80006e-05, -2.06232e-05, -2.53916e-05, -3.016e-05, -3.50475e-05, -3.75509e-05, 
   outBuffer: 2.84397e-05, 2.53898e-05, 2.37887e-05, 2.06626e-05, 1.76128e-05, 1.45629e-05, 1.15131e-05, 8.46328e-06, 5.41345e-06, 2.36362e-06, 6.86212e-07, -2.36362e-06, -5.41345e-06, -8.46328e-06, -1.15131e-05, -1.31905e-05, -1.62403e-05, -1.92902e-05, -2.24163e-05, -2.40174e-05, 
...

However, in ELK, both buffers are always 0 (so I hear no sound!):

...
void addNextBlock - writing channel 1 to output buffer, offset 10995
   inBuffer: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
   outBuffer: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
void addNextBlock - writing channel 0 to output buffer, offset 10995
   inBuffer: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
   outBuffer: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
...

Just FYI, the way I do this logging is by adding the following lines after the call to the reampler processAdding method:

auto* inBufferPtr = audioData.getReadPointer (jmin (i, audioData.getNumChannels() - 1), offset);
std::cout << "   inBuffer: ";
for (auto sample = 0; sample < 20; ++sample)
    std::cout << inBufferPtr[sample] << ", ";
std::cout << std::endl;
auto* outBufferPtr = outBuffer.getReadPointer (i, startSamp);
std::cout << "   outBuffer: ";
for (auto sample = 0; sample < 20; ++sample)
    std::cout << outBufferPtr[sample] << ", ";
std::cout << std::endl;

This makes me thing of the idea of the sampler not reading properly the audio data…
I’ll continue testing…

Interesting… What kind of file are you using as the source? Perhaps it’s just not being decoded correctly on Elk?

Maybe step through SamplerPlugin::SamplerSound::setExcerpt and see if the file is read correctly and the data is actually valid when it’s read in to audioData?

So the file is loaded in BinaryData (eg BinaryData::kick_wav), and if I use it in a WaveClip instead of in the sampler it plays nicely. That’s why I thought that the file would not be the issue. But I’ll debug these functions you mention and see what happens.

Getting closer to the issue, problem happens when calling AudioFileCache::Reader::readSamples (https://github.com/Tracktion/tracktion_engine/blob/e7e388efdccd86cb767b72f1532d513d4be94c44/modules/tracktion_engine/audio_files/tracktion_AudioFileCache.cpp#L900), basically it times out and audioData is left as an empty buffer. Should not be a permissions issue because the file is in a folder with read/write permissions by the process running the plugin.

This is where it “hangs” https://github.com/Tracktion/tracktion_engine/blob/e7e388efdccd86cb767b72f1532d513d4be94c44/modules/tracktion_engine/audio_files/tracktion_AudioFileCache.cpp#L355

const LockedReaderFinder l (*this, startSample, timeoutMs);

I guess it has something to do with ELK platform using a Xenomai RT kernel for audio processing and locks. In fact this is something that potentially causes issues in ELK because locks and some other system calls trigger kernel mode switches which are bad (sorry, I’m just learning a bit about these things so I can’t be more precise). In fact, one of the first things I had to do to avoid many mode switches was to make tracktion engine use only 1 CPU for audio processing (I use getNumberOfCPUsToUseForAudio as you advised me in another forum post). Is there a way to completely disable the use of locks in the audio processing code? (maybe I’m asking something which does not make sense…). Any other ideas?

(also I still find it weird that for the case of WaveClips I’m able to read and play files normally… Could it be that in on case I’m streaming from disk while playing the file and actually doing that in the RT thread, and in the other case I’m trying to load files in the non-RT thread?)

Is that happening readSamples happening on the audio thread? I thought that happens on the message thread when the sound is added? Surely it should be fine to lock on the message thread?

Unfortunately no, not at the moment I’m afraid. I understand why they disable any use of locks as technically they’re not real-time safe but for legacy reasons we have some locks in the code that aren’t usually contended. We’re in the process of removing these but it’s a slow and difficult process as we need to make sure we don’t break anything along the way.

It looks like readSamples is happening in the message thread, which makes sense because I’m not doing anything special, just using code from step sequencer sample and not adding anything extra in the processBlock. I added this line before the call to readSamples:

std::cout << "message thread?: " << MessageManager::getInstance()->isThisTheMessageThread() << std::endl;

And it show message thread?: 1, so this should confirm read action is happening in the message thread.

Yes, then I’m not sure why that call is blocking then in Elk. Is there no support for locks even on non-RT threads in Elk?

I’ll go back to the ELK forum and ask there. https://forum.elk.audio/t/juce-tracktion-engine-on-elk-audio-os/147/57
Will get back to you!

ELK guys got back to me, locks should be fine in the non RT thread. I actually double checked whether the loading of the files is happening in the RT thread using a function that ELK provides, and it further confirms that code is not running in the RT thread. Interestingly enough, my logging code revealed something else: when const LockedReaderFinder l (*this, startSample, timeoutMs); is called to load the audio files for the WaveClip (so not the ones for the sampler plugin but for other tracks), in that case it tells me loading is happening in the RT thread (and it works fine, sounds load and play correctly). So apparently this indicates that loading only works when happening in the RT thread (oh dear!).

EDIT: when loading the files in the sampler, I can see that after the timeout is reached for const LockedReaderFinder l (*this, startSample, timeoutMs);, l.isLocked is true. So the file is locked. I again I’m not very knowledgeable about all this, but I assume “someone” should have locked the file before, and I’m not trying to load it anywhere else. Is there anything I can do to prevent this lock? Or how could I debug out when/if the file gets locked?

Hi again, reviving this after some days. I’ve not been able to find out why files are being locked so far. Any ideas what could cause that? Could it be something external to the plugin? Or locks work only at the app level? (sorry for the noobish question if it is)