Show Midi Data As It Is Recorded

Tracktion Engine
21

Well, I have tried moving everything to the track, and the listener gets added. Using edit.restartPlayback() after all the tracks are built seemed to help here.

However, in my testing, in struct AudioTrack::LiveMidiOutputAudioNode the handleAsyncUpdater never fires, which of course means the callback, recordedMidiMessageSentToPlugins, never happens, and that is precisely the problem I am having.

Is it possible to take a look there and see what might be the issue? Or is there something else I can try?

22

I’d like to get to the bottom of this but it’s not a feature we have in Waveform.
Do you have an example that you’d be able to share so I can check it in that? Perhaps a modification to the Midi Recording Demo?

23

OK. Thank You.

Give me a day or so to put something together.

I took a quick look, and I may be able to do it with the Midi Recording Demo. We shall see…

24

New information!

In the process of putting together a repeatable experience with Midi Recording Demo, I found that the callback fires while midi is being played back! In other words, while recording midi, the callback does not fire even though this is when it is needed. But it does fire when you play back the midi you just recorded.

This is easy to reproduce. Simply make the following modifications to the Midi Recording Demo;

In the Component.h file, add the te::AudioTrack::Listener to the constructor for TrackComponent.

class TrackComponent : public Component,
                       private te::ValueTreeAllEventListener,
                       private FlaggedAsyncUpdater,
                       private ChangeListener,
                       public te::AudioTrack::Listener
{

And in the private: section of trackcomponent add;

    void recordedMidiMessageSentToPlugins(te::AudioTrack&, const juce::MidiMessage&) override
    {
        AlertWindow::showMessageBoxAsync(AlertWindow::InfoIcon,
            "TrackComponent",
            "We Reached the Callback!");
    }

And te::AudioTrack* audioTrack; as a member variable.

Then, over in Component.cpp initialize the audioTrack variable;

TrackComponent::TrackComponent (EditViewState& evs, te::Track::Ptr t)
    : editViewState (evs), track (t), audioTrack(dynamic_cast<te::AudioTrack*>(t.get()))
{

Now, in the constructor add the listener, and in the destructor remove the listener;

audioTrack->addListener(this);

audioTrack->removeListener(this);

You can now run the demo, arm a midi track and record some midi. You will not see the AlertWindow for the callback.

But, if you then rewind and play the notes just recorded, the AlertWindow appears!

And, of course, we need this to happen when recording!

Thank you for your efforts with this! I appreciate it.

25

Ok, I see the problem, the Node that calls this listener is only added if the Track has existing clips and actually before the live inputs so wouldn’t pick them up anyway.

I’m wondering now if this is the correct approach. Perhaps something that works in a similar way to RecordingThumbnailManager for MIDI would actually be better.

This is certainly non-trivial though as MIDI doesn’t have a file to reference it by and there’s a whole background threading system to add recording audio blocks to the thumbnail manager (and also flush the blocks to disk). None of this exists for MIDI so would have to be thought out and written from scratch.

Unfortunately I don’t have capacity to look in to this at the moment. I’m fully consumed with working on the tracktion_graph module and some UI features I have to do for W12. If you look at our Engine roadmap we also have more pressing features on there so I can’t guarantee that I’ll get to it any time soon.

How pressing is this feature for you? As I mentioned, we’ve been shipping a DAW for nearly 20 years without this and it’s only actually been requested a handful of times.

If you wanted to have a go at implementing it in a fully thread-safe way that matches the style of the existing Engine I can probably review and merge it?

26

It is a feature needed for consistency with the audio thumbnail, and a helpful reference when midi is recorded. It will be a nice feature to have, but it is not urgent. Put it on the to-do list at an appropriate spot that will see it done in the not-too-distant future.

In the meantime, if I were to take a look at some sort of implementation, where would you suggest I start? Where do the incoming midi notes initially come in, for example?

27

They come in from the MidiInputDevice as it is a juce::MidiInputCallback.
However, I warn you the internals of this are extremely complicated as there are lots of features such as MIDI channel filtering, note quantising etc. that get applied, different record modes which determine if played notes are heard when looping and the fact that multiple Edits can use the same MIDI inputs so you have MidiInputDeviceInstances for each Edit that deal with recording etc.

Probably putting a break point in RecordingThumbnailManager::Thumbnail::addBlock and working backwards is the best place to start for seeing how it’s done with audio. There are multiple threads involved in this though which complicates the code paths.

28

Thank you! I’ll poke around and see if I can come up with anything.

I know you are actively working on a total re-do of the whole Audio Graph pipeline. Would I be better to wait until you have released the new code?

29

Maybe… It depends on the implementation.
You might have to adapt it to the new code but any changes should be minimal as it’s really all the AudioNode subclasses that are being replaced by tracktion_graph::Node versions and the graph building and playback happens in a slightly different way.

If this goes via the MidiInputDevice/MidiInputDeviceInstance those will largely be staying the same.

30

I had time today to look at this again. Since we only need the Midi notes as a visual guide to show that recording is proceeding smoothly, it will work just fine if there is a listener/callback that can deliver the notes as they arrive. These visual reference notes will only exist until recording stops whereupon they will be replaced by the recorded clip. Is there a point in the architecture where such a listener/callback “hook” could be patched in?

I looked at the MidiInputDevice / MidiInputDeviceInstance code, but it seems like a lot would need to be done to get that to work right. Or am I missing something obvious?

31

I wonder if MidiInputDevice::MidiKeyChangeDispatcher::midiKeyStateChanged actually does what you need?

We use that for MIDI step entry so should forward events from incoming physical devices in a thread-safe way.

If not, you’d probably need a fifo in the MidiInputDeviceInstance class and add any notes that are recorded to that and then have another thread that pulls the notes out and in to a MidiMessageArray suitable for reading on the message thread.

I’d then probably have a way of getting the recording MIDI data via the InputDeviceInstance::state member as that’s unique to the instance.

32

Interesting! I’ll study that code and see what I can come up with.

Thank you!

1 Like
33

Looking at the pure virtual callback midiKeyStateChanged, we have the notes and velocities, but no time stamps. So, it seems that the notes would arrive in clusters, not necessarily relative to each other.

Or am I missing something?

34

Thinking about this further…I know it is a lazy update, but perhaps it is still close enough to be representative of the timing relationships of the notes?

I will put something together and see how it looks.

This may work yet!

35

It will be slightly out but probably ok for a temporary visual reference.
You could time the difference between the incoming timestamp and the callback being delivered. It will probably only be a few ms at most.

36

I’m working on this now and stumbling over adding the listener. How do I expose the listener?

Currently, I have;

class MidiRecordingClip : public Component, public te::MidiInputDevice::MidiKeyChangeDispatcher::Listener
{
public:
	MidiRecordingClip(te::Track::Ptr t) : audioTrack(dynamic_cast<te::AudioTrack*> (t.get()))
	{
		// addListener ???
	}

	~MidiRecordingClip()
	{
          // removeListener ???
	}

and then,

private:

	void midiKeyStateChanged(te::AudioTrack* audTrk, const juce::Array<int>& notes, const juce::Array<int>& vels) override
	{
		if (audTrk == audioTrack)
		{
                 // handle incoming notes here
		}
	}

But, I am not finding the right syntax to add and remove the listener.

I am thinking it comes from the instance state, but what is the correct syntax?

37

Hmm, this looks a bit odd but I seem to remember the way this is done was a bit of a hack when we broke out the Engine in to its own module.

The Listener is actually a singleton so you can just create one of these in your MidiRecordingClip class:

    juce::SharedResourcePointer<MidiKeyChangeDispatcher> midiKeyChangeDispatcher;

Then add your listener to that (and remove it afterwards). E.g. midiKeyChangeDispatcher->addListener (this);

Then your callback should work as expected.

38

Wow. It is pretty unique for it to work that way! No wonder I was having trouble!

I appreciate you explaining how that works!

39

Yeah, it’s not the cleanest way. I can’t remember why but there was a reason for doing it like that at the time.

Those kind of nasty edges are what we’re slowly smoothing off as we go through refactoring things. Adding proper recorded MIDI access similar to audio for thumbnailing is one of those things that would mean this nastiness isn’t required.

1 Like
40

Still no joy. The callback never fires.

Below is the code I came up with to test.

class MidiRecordingClip : public Component, private te::MidiInputDevice::MidiKeyChangeDispatcher::Listener
{
public:
	MidiRecordingClip(te::Track::Ptr t) : audioTrack(dynamic_cast<te::AudioTrack*> (t.get()))
	{
		midiKeyChangeDispatcher->listeners.add(this);
	}

	~MidiRecordingClip()
	{
		midiKeyChangeDispatcher->listeners.remove(this);
	}

private:
	void midiKeyStateChanged(te::AudioTrack*, const juce::Array<int>&, const juce::Array<int>&) override
	{ 
		// handle incoming notes here

		AlertWindow::showMessageBoxAsync(AlertWindow::InfoIcon,
			"MidiRecordingClip",
			"We hit the callback!");
	}

	te::AudioTrack* audioTrack;
	SharedResourcePointer<te::MidiInputDevice::MidiKeyChangeDispatcher> midiKeyChangeDispatcher;
};

I am probably overlooking a detail here. Please advise?