Latency between adding and playing a midi note

Tracktion Engine
21

So the user writes some code, presses ‘return’ or clicks a ‘compile’ button, and the code churns out a sequence of notes which should then get played at some point in the future?

It doesn’t sound like a particularly hard thing to do… you’d do all the compiling on the message thread, it would create some kind of array of midi notes lasting e.g. a few minutes. Then you want the edit to start playing that new stuff in time with the beat, without stopping, so you’d just insert it as a new clip, on a bar-boundary, maybe 1 or 2 bars ahead of the current time.

Or maybe I’m misunderstanding what you’re trying to do…?

22

As far as I can gather that’s correct.
The problem being reported is if the first calculated note is very soon (< ~0.3s), the time it takes to calculate the new notes, rebuild the playback graph and have that played back means the first note will be lost.

This does sound a bit odd though. In my tests, even with a very complex Edit rebuilding the playback graph and inserting it to play back only takes about 50ms.

Where are you getting this 0.3s from?

23

The ~0.3 second threshold came from the simple test case that I shared in the original post. IIRC I only built it in debug mode, so maybe it would be much lower in release mode. In any case, the problem would still remain, because the user might click on an event that expects to be played “immediately”. The best solution I can think of us to try to schedule everything on the Clip, then mop up everything that the Edit missed and inject it live.

24

I can see that I’m trying to use Tracktion to do something it’s not really designed for; however, I think I can still make it work.

If I understand correctly, the idea in Tracktion is that the MidiClip stores a basic pattern of midi notes, which can then be played and repeated together as a block by the Edit, in sync with the global timeline. The user can change the notes on the MidiClip, but unless the user edits them, they don’t really change.

I am using Tracktion because I need a global timeline which can incorporate midi, VST and sample playback (only midi for now). However, my program doesn’t store notes in Clips–instead, the sequence of notes is determined by the commands that the user has typed out. The commands might say something like, “play note A, then pause for 1 second and repeat”. When the event is clicked, my program will read these commands and then start populating the note A on a MidiClip at one second intervals. It has to call getSequence().addNote() for every single note, because it treats the MidiClip as being an infinitely long timeline.

This obviously isn’t what MidiClip was designed for, but it seems to be working pretty well. The only problem left to solve that I’m aware of is the problem when a new note is added very close to the current playback time. I hope that I can solve this by catching the notes that have been missed, as outlined above.

25

I’ve got it working, using the AudioTrack::Listener to identify dropped notes, as outlined above. However, I am still wondering if there would be a better way for me to approach this whole thing. I’ve been pondering something you said in another post:

It does indeed seem like I’m not respecting the boundary between the model and the playback. Rather, I have created my own model, which I am trying to synchronize with the model inside MidiClip, using addNote() and removeNote().

I’m therefore wondering if it would be possible for me to bypass the MidiClip altogether. Can I redirect the AudioTrack to read from a juce::MidiMessageSequence that I maintain independently of a MidiClip? I’m guessing that the answer is “no”, but I thought I’d check anyway.

Sorry for asking so many daft questions! I really appreciate your help though.

26

No, I think you’re missing the difference between the “model” elements of Tracktion Engine (Edit, Clip, MidiClip, MidiList, MidiNote etc.) where you build your definition of what needs to be played back. And the playback engine (basically the EditPlaybackContext and AudioNode [soon to be tracktion_graph::Node] classes).

When playing back, there is no concept of the “AudioTrack reading from MIDI”, the playback graph abstracts these all away in to MIDI nodes and mixer nodes feeding PluginNodes.

However, a juce::MidiMessageSequence is generated from the MidiList held inside the MidiClip so if you populate that with the sequence you want played back, it will be. I can’t quite see what the problem with this method is as whatever other method you’d come up with to " read from a juce::MidiMessageSequence" would boil down to the same thing.

27

Unfortunately I’m still struggling to make this work. It feels like I am trying to fit a round peg into a square hole. The more I try to fix it, the more complicated the system becomes, without really fixing any of the underlying problems.

I can go into more details with the problems that are arising, but before I do, I want to float a new idea. What would happen if I abandoned the Tracktion scheduling system altogether, and just used injectLiveMidiMessage() for everything? I would use edit.getTransport().getCurrentPosition() to decide when to call the events, so hopefully everything would still stay in sync with the main engine timeline.

What I’m thinking of is something like this:

while(true)
{
    for (auto event : activeEvents)
    {
        if (!event->hasPlayed() && event->getExecutionTime() <= edit.getTransport().getCurrentPosition())
        {
            event->setHasPlayed(true);
            audioTrack->injectLiveMidiMessage(event->getMidiMessage());
        }
    }
    sleep(1);
}

A system like this would be a lot simpler than what I’m currently trying to do. activeEvents is an array that I’m already managing, and I wouldn’t have to worry about interfacing it with the midiSequence (which is what’s causing headaches).

However, I still don’t know much about the inner workings o f Tracktion, so (ab)using injectLiveMidiMessage() like this might be a horrible idea for other reasons. For one thing, I assume that the accuracy of the timing would be reduced, since the timing would no longer be set by the audio thread. Unless it is quite noticeable, I’m prepared to tolerate poorer timing, for now at least. But are there any other side effects that you would expect if I were to use injectLiveMidiMessage() for every note?

28

Just to reiterate a point…anything you do on the message thread will inherently not be in sync with the Tracktion Engine processing thread. And, in fact, other events on the message thread may block for an indefinite period of time. This means that the delay will vary depending on the message thread work load. So, you may find that the midi note plays at the desired point on one attempt, then on another attempt it is delayed, or absent.

Tracktion Engine (and most any other DAW engine) is designed to connect everything together and coordinate playback of clips that are populated with audio, or midi notes, which are time stamped to be played at the appropriate time as the timeline progresses. It does a very good job of this.

Therefore, since the message thread (or your own parsing thread), are not synchronized with the processing thread, it will always be an issue of having some non-zero time offset between the threads. Meaning there will always be some latency with that approach.

If there is a way to streamline your parsing and feed the reduced data directly on the processing thread, you might be able to get down to extremely low latency. But doing anything like that on the processing thread is not recommended…for there lies the realm of pops and crackling noise.

Your own thread might be best because you can at least have some control over the thread work load. The message thread will always be too unpredictable for your use case IMHO.

Just my two cents…

29

Thanks for your response. I do have my own thread dedicated to parsing, so the message thread is off the table right now. The issue I’m facing is that when the parsing thread tries to schedule a note that is very close to the playhead, the note gets missed (or worse yet, the noteOff gets missed). The nature of my program means that this is happening often, so it’s not a problem I can afford to ignore.

The solution I proposed above would essentially be to move the scheduling off of the audio thread and onto the parsing thread (though still keeping in time with the playhead). I’m pretty sure that this would lighten the load on the audio thread, though it might be a bad idea for other reasons.

Another idea would be to ask the audio thread to handle all the scheduling, something like this:

virtual void applyToBuffer(const tracktion_engine::AudioRenderContext& a) override
{
    ignoreUnused(a);

    while (auto e = eventsToBeScheduled.pop())  // eventsToBeScheduled is a lock free fifo
    {
        const double startTime = jmax(e->getStartTime(), edit.getTransport().getCurrentPosition());

        midiClip.getSequence().addNote(e->getNoteNumber(), startTime, ...);
    }
}

As @JeffMcClintock pointed out, this would probably solve the timing issues; however, it might cause the audio thread to freeze, as @bwall says. I currently feel more optimistic about the first approach than the second, but I would like to hear what other people have to say.

30

I think many of us are struggling to see why you require inserting notes so close to the playhead?

As Jules mentioned, since any user interaction is unpredictable, you would need to parse well ahead of the playhead anyway. So, it seems that it is a question of the timing of when the instructions are parsed, and not of where the playhead is, per se’. In other words, perhaps you need to optimize the time between entry of commands and the parsing of those commands?

31

I understand that it must seem like a strange requirement.

As I have explained above, the sequence of midi notes is determined by a set of text commands. The parser parses them and predicts ahead 4 seconds into the future to schedule the notes. But the system needs to be robust enough to handle user input while the audio is playing. If the text commands change during playback, the parser output is destroyed and recalculated. This causes notes on the MidiClip to be removed and then re-added, and if this occurs close to the line, then problems occur.

The result is that if the user changes the text commands right around the time that a note is playing, the system will tend to miss a noteOn or noteOff event. Missing a noteOn is tolerable, but missing noteOff isn’t.

I have been trying to build a system that records which notes have actually been played, and then uses this data to deduce which notes have been missed, and inject them live. But the system is getting really complicated and has yet to solve the problem, so I’m looking for new solutions. The two solutions I’ve proposed above seem horrible, but they are the best I’ve managed to come up with.

32

I think it is possible to state my problem without any reference to parsing and all the other weird things I’m trying to do, which is clearly causing confusion.

Suppose I am making a regular daw, where the user can schedule notes by clicking on a clip. Now suppose that the user cancels a note very close to the line, or in fact after the noteOn message has been sent (but before the noteOff has been sent). The note is now stuck on.

Below there is a very crude working example of this problem. In the example, pressing the ‘a’ key schedules a note 1 second in the future, and pressing the ‘b’ key removes it. If you press ‘b’ after the note has started, the note is stuck on.

Of course, you can easily use injectLiveMidiMessage() to kill the orphaned note, but this isn’t a general solution because you don’t want to kill it if it hasn’t been played yet. So you need to keep track of which notes have actually been played, which is harder than it sounds. (It’s especially hard in my case, since some of the noteOns have to be injected, since the scheduler will miss them for similar reasons).

static String organPatch = "<PLUGIN type=\"4osc\" windowLocked=\"1\" id=\"1069\" enabled=\"1\" filterType=\"1\" presetDirty=\"0\" presetName=\"4OSC: Organ\" filterFreq=\"127.00000000000000000000\" ampAttack=\"0.06000002384185791016\" ampDecay=\"10.00000000000000000000\" ampSustain=\"40.00000000000000000000\" ampRelease=\"0.40000000596046447754\" waveShape1=\"2\" tune2=\"-24.00000000000000000000\" waveShape2=\"1\"> <MACROPARAMETERS id=\"1069\"/> <MODIFIERASSIGNMENTS/> <MODMATRIX/> </PLUGIN>";

class MyMidiSequencer  : public Component
{
public:
    MyMidiSequencer() :
        engine("MyMidiSequencer"),
        edit(engine, tracktion_engine::createEmptyEdit(), tracktion_engine::Edit::EditRole::forEditing, nullptr, 0),
        transport(edit.getTransport()),
        midiClip(getOrCreateMidiClip())
    {
        setSize(200, 300);
        setWantsKeyboardFocus(true);
        const auto newPlugin = edit.getPluginCache().createNewPlugin(tracktion_engine::FourOscPlugin::xmlTypeName, {});

        tracktion_engine::getAudioTracks(edit)[0]->pluginList.insertPlugin(newPlugin, 0, nullptr);
        
        XmlDocument doc(organPatch);
        if (auto e = doc.getDocumentElement())
        {
            auto vt = ValueTree::fromXml(*e);
            if (vt.isValid())
                newPlugin->restorePluginStateFromValueTree(vt);
        }


        transport.setLoopRange(tracktion_engine::Edit::getMaximumEditTimeRange() - 1);
        transport.looping = true;
        transport.position = 0.0;
        transport.play(true);  
    }

private:

    bool keyPressed(const KeyPress& key) override
    {
        if (key.getTextCharacter() == 'a')
            lastScheduledNote = scheduleNewNote();

        else if (key.getTextCharacter() == 'b')
            removeLastScheduledNote();

        return true;
    }

    tracktion_engine::MidiNote* scheduleNewNote()
    {
        const double time = 0.1 + transport.getCurrentPosition();

        return midiClip.getSequence().addNote(45, edit.tempoSequence.timeToBeats(time), 1.0, 127, 127, nullptr);
    }

    void removeLastScheduledNote()
    {
        if (lastScheduledNote)
        {
            midiClip.getSequence().removeNote(*lastScheduledNote, nullptr);
        }
    }



    tracktion_engine::MidiClip& getOrCreateMidiClip()
    {
        edit.ensureNumberOfAudioTracks(1);
        auto firstTrack = tracktion_engine::getAudioTracks(edit)[0];

        if (dynamic_cast<tracktion_engine::MidiClip*> (firstTrack->getClips()[0]) == nullptr)
            firstTrack->insertNewClip(tracktion_engine::TrackItem::Type::midi, {0.0, 10000.0}, nullptr);

        return *static_cast<tracktion_engine::MidiClip*> (firstTrack->getClips()[0]);
    }

    tracktion_engine::Engine engine;
    tracktion_engine::Edit edit;
    tracktion_engine::TransportControl& transport;
    tracktion_engine::MidiClip& midiClip;

    tracktion_engine::MidiNote* lastScheduledNote = nullptr;

    JUCE_DECLARE_NON_COPYABLE_WITH_LEAK_DETECTOR (MyMidiSequencer)
};
33

If the only problem is that you’re getting hanging notes when you change the clip content, then we should probably just fix that in the engine.

We have a pretty complex system to keep track of what notes are playing and to avoid hanging ones, but maybe you’ve just managed to find a way to break it that doesn’t happen in a normal DAW setup.

34

The example code I posted demonstrates the hanging note. Take a look at what I’m doing inside the keyPressed() function. It’s pretty simple–all it really does is call addNote() and removeNote(). But if removeNote() is called at the wrong time, the note hangs.

I would of course love it if this were fixed internally, but I also can’t escape the notion that I am doing something wrong here. Does the user have access to the system you mentioned which tracks notes, or does this all happen internally?

35

You mention that, if the user changes text commands during playback. the parser output is destroyed and rebuilt. That is where the problem arises. Because of inherent time delays already discussed, the changes cannot happen instantaneously, much less retroactively. So, the code needs to be revised to keep those commands that are already in progress, and only rebuild going forward.

At least, that is how it appears to me, if I am understanding correctly.

36

I’ve been meticulous about making sure that the parser only rebuilds the output that has been affected by the particular command change. But there is nothing to stop the user from changing a given command right when its note is being played (indeed, this happens quite often during testing).

I have been trying to “freeze” the events as you suggest so that they can’t be altered once the noteOn occurs, but I am still getting dropped notes. I think that this happens when the parser recalculates an even after it has played but before it knows that it’s been frozen. This is because it’s quite difficult to determine whether a tracktion_engine::MidiNote has been played or not, and hence if it should be frozen. I’ve been trying to achieve this through the recordedMidiMessageSentToPlugins() callback, but it’s difficult to reliably correlate the juce::MidiMessage with the tracktion_engine::MidiNote.

Another factor here is that “play midi note now” is a valid command for my program, so the scheduler has to be able to respond to it (or “play midi note in 0.1 seconds”, etc). Currently how this works is that the note is scheduled for time “now” on the midi sequence. When the note is skipped, the callback is never made. The parser notices that a note was skipped, and then injects it live. This seems to work, but it adds another layer of complexity when trying to hunt down hanging notes.

I know that this must all sound a bit confusing (I mean, I find it confusing and I wrote the damn thing). If I were to summarize, I would say that need either

  1. a reliable way to tell which tracktion_engine::MidiNote has been actually sent to the plugins; or
  2. some way of ensuring that a note is never dropped, even when it’s scheduled right down to the line.
37

This is what your use case looks to me. Correct me if I’m wrong.

You have at every given moment a list of midi events to be played, each with their individual time stamp when they’re expected to be played. Call it a mdi message sequence… :slight_smile:

The playback could be realised either as midi synt plugin (if you want some sound) or a midi effect ditto if you have an external synt/sampler to fix the sound. (A slight difference is that usually these plugins have a midi input, but some, like the Juce demo arpeggiator plugin also creates midi events internally, so this difference is of minor importance).

Now the only unusual thing about your case is that the list of midi events to be played is
expected to change arbitrarily. Like you had a reciter whose text is suddenly switched to
another every once i a while with, possibly, completely different content. Right?

You already have a worker thread that calculates the new sequence of midi events, I believe. Just stuff the result in a MidiMessageSequence and swap that with the current sequence being played by the plugin and voilá there you are!

Use newMidiSequence.swapwith(oldSequence) in a critical section.

The task of keeping track of missing note-off remains of course, but that’s a minor task (Eg. put the note-ons (i.e their note number) just being played in a std::vector and remove them whenever
corr note-off occurs.

After every midisequence swap, you send a note-off for every note that’s left in the vector.

1 Like
38

It seems like @oxxyyd’s approach might be workable for you. So give that a try.

I fear you will always have an issue with trying to change an action that is already in progress. Still, maybe you can get delays down to an acceptable level.

It is an interesting challenge, so I hope you get things working to your satisfaction!

1 Like
39

This sounds very promising. I’m not sure how I would make it work within the tracktion_engine::MidiClip though. The MidiClip seems to work with a tracktion_engine::MidiList, which is different than juce::MidiMessageSequence. I can call tracktion_engine::MidiClip::mergeInMidiSequence(), but that doesn’t seem right.

Am I missing something here? Do you have another idea for how to get the MidiMessageSequence to the plugin without the MidiClip?

40

My suggestion is that instead of using the tracktion engine you put your “non linear sequencer” in a plugin. To be used in a Daw of your choice, in the Juce plugin host or converted to standalone application. If you haven’t done so before, get acquainted with the midi plug-in tutorials, e.g the arpeggiator.

Here’s a short example of using a MidiMessageSequence (possible an output of your parsing thread) as the input to a midi plugin. (It’s a stripped down version of a much larger midi file player, so there can be some details missing, but hopefully it shows how to “play” a MidimessageSequence in a plugin processor.)

/*
* members 
* int nextEventID = 0, pointer into midimessagesequence of next midi event to be played. 
* bool suspended = false;	//optional.
* int samplePos = 0; # of samples since playback started
* int numMidiEvents = midiSequence.getNumEvents();
* double sampleRate. current sample rate read in audioprocessor->prepareToPlay()
* MidiMessageSequence read from midi file or elsewhere...
*/

void MidiSourceProcessor::prepareToPlay(double sampleRate_, int estimatedSamplesPerBlock) noexcept
{
	nextEventID = 0;
	numMidiEvents = midiSequence.getNumEvents();
	samplePos = 0;
	sampleRate = sampleRate_;
}


void MidiSourceProcessor::processBlock(AudioSampleBuffer& buffer, MidiBuffer& midiMessages) noexcept
{
	midiMessages.clear();
	buffer.clear();

	auto numSamples = buffer.getNumSamples();

	if (!suspended)
	{
		if (numMidiEvents && nextEventID < numMidiEvents)
		{
			int waitSamples;

			while ((waitSamples = sampleOfNextMidi - samplePos) < numSamples)
			{
				auto midiEvent = midiSequence.getEventPointer(nextEventID++);

				midiMessages.addEvent(midiEvent->message, waitSamples);

				if (nextEventID < numMidiEvents)
					sampleOfNextMidi = (int)(midiSequence.getEventTime(nextEventID) * sampleRate);
				else
					break;
			}
		}
	}

	samplePos += numSamples;	//we're advancing samplepos even if suspended (by mute or solo) to maintain sync
}