Adding / removing MidiNotes from a playing MidiSequence in a MidiClip without restarting playback

Tracktion Engine
1

I’m working on a sequencer using Tracktion Engine, and I have the code below which creates a MIDI output and loops over 1 bar allowing the user to toggle notes on and off using an external controller. It all works, but when I modify the sequence, either adding notes, removing notes, clearing the sequence, etc. the playback starts from the beginning, sometimes after a small delay. How can I get it to continue playing and not restart when the MidiSequence is modified? I tried creating a separate sequence and clearing the current one then copying the sequence using addFrom() but that didn’t work.

        void createMidiOut()
        {
            track = tracktion_engine::getAudioTracks(*edit)[0];
            if (track)
            {
                track->getOutput().setOutputByName("IAC Driver Bus 2");
                const te::EditTimeRange editTimeRange(0, edit->tempoSequence.barsBeatsToTime({ 1, 0.0 }));
                track->insertMIDIClip("MIDI Clip", editTimeRange, nullptr);

                if (auto clip = dynamic_cast<te::MidiClip*> (track->getClips()[0]))
                {
                    midiClip = *clip;
                }
            }
        }

        void loop()
        {
            auto& transport = edit->getTransport();
            transport.setLoopRange({ 0.0, edit->getLength() });
            transport.looping = true;
            transport.play(false);
        }

        void toggleNote(int col, int row)
        {
            double beats = edit->tempoSequence.timeToBeats(col * .125);
            Note note = {
                    75 + output->getCols() - row,
                    beats,
                    .25,
                    127
            };

            auto& transport = edit->getTransport();
            double pos = transport.getCurrentPosition();

            auto& seq = midiClip->getSequence();
            if (notes.find(note) == notes.end())
            {
                auto midiNote = seq.addNote(note.pitch, note.beats, note.length, note.velocity, 0, nullptr);
                notes[note] = midiNote;
            }
            else
            {
                seq.removeNote(*notes[note], nullptr);
                notes.erase(note);
            }

            transport.setCurrentPosition(pos);
        }
2

I guess you want prevent this behavior by adding this line?

transport.setCurrentPosition(pos);

But I can not see, why your loop is starting at the beginning when you add a note. I can add/remove notes to a sequence but the transport keeps playing. There must be a reason why your transport starts again.

3

Yeah I’m not sure why it starts at the beginning but there will be some delay between reading a position from the message thread and then setting it again (i.e. the audio thread may have advanced in time during this slight gap). So setting the transport position may cause a slight jump.

As @baramgb says, just don’t call transport.setCurrentPosition when modifying the sequence and it should all work.

4

I forgot to mention that without the

transport.setCurrentPosition(pos);

this still happens. That was an attempt to try to prevent the loop restarting. @baramgb So the sequence should be modifiable while the loop plays without restarting. That is helpful to know. Any ideas on what else would cause the sequence to restart? Interestingly enough, the call to setCurrentPosition() doesn’t seem to do anything at all that I can tell as it doesn’t restore either the previous position or any positions close to it and the loop restarts with or without it. @dave96

5

Maybe put a breakpoint in TransportControl::performPositionChange to see what’s causing the change?

6

Actually, I’m wrong about what’s happening. It’s not repeating. It’s sending an all notes off and a reset all controllers message out, then resumes sending midi after a delay. TransportControl::performPositionChange is never called. @dave96

7

Are these getting inserted because a isAllNotesOff message is getting set on the MidiMessageArray?
I.e. line 1190 of ExternalPlugin::prepareIncomingMidiMessages?

That usually happens when the playhead jumps i.e. line 65 or 71 in PlayHeadState::update.
Are you sure PlayHead::setPosition isn’t getting called somewhere from your code?

8

No, prepareIncomingMidiMessages() is not being called and neither is setPosition(). The only time it is and it jumps is when the play/stop on the transport is pressed. Not sure exactly what is happening that’s causing this but when I add/remove notes, none of the above breakpoints are being hit. Maybe it’s some sort of blocking or threading issue or some configuration that is wrong? @dave96

9

Also, the all notes off / reset all controllers doesn’t always happen. If i turn a note on / off by repeatedly alternately calling addNote() and removeNote() when I press a button, then it basically gets delayed indefinitely and nothing plays. Same thing if I call just addNote() repeatedly. But usually only the first time I add a note will the all notes off / reset controllers messages be sent. It also seems to be sent after the call to addNote() / removeNote(). I’m continuing to try different things and ways of triggering these notes to see if I can find a way to avoid this.

10

Sorry, I’m not quite sure I follow what’s happening here.
If you do a similar thing in Waveform, i.e. add/remove a note from the MIDI editor, you’ll see playback just continues so I’m sure you must be doing something else that’s causing the playhead to jump around?

11

I observed Waveform in the MIDI monitor and it seems like the all notes off / reset all controllers messages are a normal part of adding a midi note while the clip is playing. Therefore, I don’t think that is the issue. My app is also not jumping or changing the position of the playhead explicitly either. It’s in sync with the metronome but it seems to skip the rest of the bar the note was added/removed in that was currently playing and start over at the start of the next full bar after that. (It’s looping around one bar.)

I’ve tried clearing the clip and replacing all the notes to the same effect. I’ve tried removing the clip and creating a new clip and adding all the notes also to the same effect. It seems that no matter what I do or how I modify the notes or clip, there is a pause in notes and a continuation only when the time marker hits the start of the next bar.

Is there a better way for me to modify the sequence? Can I create the sequence first and then attach it to the clip or replace the current one in the clip? I’m obviously doing something wrong, but I’m not sure what as I’m farily new to audio programming.

12

When you add the new note, is the playhead currently over (or very near to the start of the note)?
When a note is added, the playback sequence is updated asynchronously and as two distinct events, a note on and a note off. If the playhead has passed the note-on event then the note won’t be started until the playhead moves over it again. Could this be what you’re seeing?

13

There doesn’t seem to be a correlation between the note and the playhead. Whether it’s over the note or in a completely different place, the effect seems to be the same. Also, if I add multiple new notes rapidly, I can basically prevent any notes from playing until I stop adding notes (a partial note or two may be heard if I’m not quick enough) and it makes it to a new bar. Also, all other (existing) notes stop playing too. The entire pattern is silenced. That’s the strange thing. But the metronome is not missing any beats or skipping as far as I can tell. It’s almost as if the midi sequence is removed from playback completely while it is being manipulated (because adding or removing cause the same issue). Then when there are no more changes triggered, the entire sequence is added back and starts on time with the next bar.

14

Have you got a video or screen capture of what you’re doing? It’s a bit difficult to try and visualise.

And did you say you tried something similar in Waveform, i.e. adding/removing notes on a MIDI clip during playback. Did you get the same timing issues?

15

Waveform doesn’t do it. I noticed no issues in it adding/removing notes. I uploaded a clip here with sound: seq1.mp4 - Google Drive You can hear the metronome. The AKAI fire in this case acts as a simple 1 bar step sequencer. Whether I add/remove one note or many, the entire melody is delayed. The more notes, the longer the delay. It seems to always start from the next bar after I stop making changes. I also noticed some slight timing issues with starting / looping, but that is for a different post but I bring it up in case it could be related.

16

Yeah that’s really not how it’s supposed to behave.
Without seeing how your controller influences the code it’s a bit difficult to tell what’s going on.
A couple of things to look in to:

  • Are you sure you’re adding/removing notes at the correct time and not off in to the future?
  • When you add/remove a note, Edit::TreeWatcher should be called which leads to a call to Edit::restartPlayback. Is that happening?
  • If so, is a new MidiNode being created with the new sequence?
  • If you’re only toggling cells though, have you thought about using a StepClip? There’s a pretty thorough example of that in StepSequenceDemo
17

Would it help to post more of the code? I don’t mind, but I don’t want to inundate you with a bunch of code that’s likely unrelated. Most of what I have so far is code that wraps MidiInputDevice and MidiOuptutDevice, listens for events, and adds them to a MidiList. There is a timer then that triggers some minor processing to turn the MidiList events into a generic structure of button presses / knob turns and a very simple event callback that triggers when a pad is pressed which then calls the code above with the x/y coordinates of the pad from which a note is calculated and turned on/off. Could it be some sort of concurrency issue in this part that’s messing with the audio thread somehow? The metronome doesn’t seem to have any issues though.

To answer your questions:

  • Yes, I’m sure the times are correct and are in the first bar. The first pad is location 0, the second .25, etc. And they play/not play as expected except for the “delay” or “dropout” when nothing plays.
  • Yes, Edit::TreeWatcher is called and it does indeed call Edit::restartPlayback. Could this be the issue possibly? Could restartPlayback be causing the delay by waiting to restart at the next bar? It does sound like it is waiting for the start of the next bar and then it plays without issue.
  • Yes, a new MidiNode is being added to the sequence.
  • I will look into that, but eventually, I want to make a complete sequencer. This step sequencing is just a start, for now a proof of concept.
18

I removed my external controller handling code from the picture and now it’s almost working well, however, it is still sending ‘all notes off’ when a note is added which adds a very slight hiccup (note cut off sometimes). My triggering code is in a juce::Timer callback: (x is quarter not position, y is pitch offset from some base pitch)

    void MainComponent::timerCallback()
    {
        static int x = 0, y = 0;
        sequencer->track->toggleNote(x, y);
        ++x;
        ++y;
    }

Now it doesn’t wait a whole bar to restart, but there is this slight delay every time the sequence is modified (5 seconds, that’s how often the callback gets called in this example). I’m wondering why does it do this? I’m also wondering if the juce::Timer that my MIDI controller handling code runs might interfere and be creating the original problem where a whole bar is skipped before the loop is played. The timer callbacks are my suspicion as to what is causing the big delay. I’m basically putting the notes into a MidiBuffer and then transforming them into an std:vector using two 1ms timers. Maybe this is not the best way to do things.

Add message to a MidiBuffer

    void MidiController::handleIncomingMidiMessage(juce::MidiInput* source, const juce::MidiMessage& message)
    {
        auto sampleNumber = (int)(message.getTimeStamp() * deviceManager.getSampleRate());
        inputBuffer->addEvent(message, sampleNumber);
    }

Process them using a timer:

    void MidiController::timerCallback()
    {
        for (const auto message: *inputBuffer)
        {
            processMessage(message.getMessage());
        }

        inputBuffer->clear();
    }

processMessage() then pushes the new messages (no longer MIDI) into an std::vector called controllerMessages, and they are processed further in another juce::Timer (maybe there’s a better way to make such callbacks that are located in various classes?):

    void MultiAkaiFire::timerCallback()
    {
        int fireIndex = 0;
        for (auto& fire: fires)
        {
            while (!fire->controllerMessages.empty())
            {
                auto message = fire->controllerMessages.front();
                fire->controllerMessages.pop();
                processControllerMessage(message, fireIndex);
            }

            fireIndex++;
        }
    }

Even if the controller code is fixed to not cause the delay in the sequencer though, there’s still the problem of the momentary ‘all notes off’ (I think this is causing the issue) when the MIDI sequence is modified that happens when just adding notes using a simple timer callback. Can this be turned off somehow? It happens even when the note is added to the end of the clip, beyond the loop point.

19
  • Do you know where the all-note-off message is being added?
  • Edit::restartPlayback doesn’t change the transport/playhead at all, it just rebuilds the audio graph and sets it to the “currently playing graph” as soon as possible so I doubt this is the problem.
  • What is inputBuffer? What thread does handleIncomingMidiMessage get called on? Is it thread-safe to access inputBuffer between the two methods?
  • I don’t really understand the logic between your two timers and containers of messages. If they don’t run quickly enough it could be that you’re just not dispatching incoming messages to modify the MidiList quickly enough. I’d put some logging in to see the timestamps of your various handleIncomingMidiMessage/timerCallback/timerCallback/processControllerMessage methods to see how they’re separated in time
  • Double check your timers are running at the frequency you expect. Remember they are started either in a period in ms or a frequency in Hz (two different startTimer/startTimerHz calls).
20
  • The all note off message might be coming from calling the addNote() method on the sequence of the MidiClip that’s playing because when I remove that call there is no pause whatsoever with everything else still running, timers, midi input, etc. I changed the code to add/remove notes from a different sequence and there is no all notes off message and no pausing. It’s a pause followed by resuming at the top of the next measure. Therefore I don’t think this is due to threading or timing issues because it always starts precisely at the top of the next measure, something that seems unlikely to be due to threading/timer issues. Also, the metronome has no issues keeping time while the midi is stopped. That’s why I keep wondering if there is another way to modify the midi clip while it’s playing without it restarting from the top at the next measure.
  • As far as I can tell, all my processing is on the message thread. I’m using juce::MessageManager::callAsync() in MidiController::handleIncomingMidiMessage() and timerCallback() to call back my processing functions on the message thread
  • I got rid of inputBuffer altogether but it didn’t change anything.
  • I got rid of the two timers also but that didn’t change anything (added one back in);
  • I set the timers on 1ms originally. Tried other values like 10ms. It doesn’t make any difference.

The only thing that makes a difference is not calling addNote()/removeNote() on the playing clip. I can call all the rest of my code and it doesn’t make the sequence pause. I can call those functions on a seperate MidiList without issues also. But I simply cannot modify the playing sequence while it’s playing without it stopping and restarting at the top of the next measure. I have also tried making the changes in a seperate sequence and replacing the current sequence with the new sequence. This fails in the same way. I’ve even tried removing the clip altogether and rendering the sequence onto a new clip and there’s still the same issue.