How do you compensate for latency in a MIDI-generating plugin?

Audio Plugins
1

I’ve been developing a MIDI-generating plugin. In it, you can select Direct MIDI outputs and route MIDI to external devices without going through the Track of a DAW. This is mainly for testing, but I might leave it as a feature.

In any case, if I compare the MIDI being sent to the direct MIDI Output with the MIDI that comes out of the VST3, producing the MIDI from the plugin seems to introduce about 11 ms of latency (in Reaper on macOS) - the notes are 11 ms later.

I’ve not attempted to deal with latency at all before.

I see that there is AudioProcessor::getLatencySamples(), whereby you inform the host about latency; and you are supposed to set this yourself with setLatencySamples(). So how do you know what value to use here? What if it’s different in different hosts? In Digital Performer, I’m getting a whopping 150 ms of latency…

And is this actually how you do it? I tried this in my AudioProcessor constructor, just as a test (assuming 48k), and it seemed to make it 11 ms WORSE…

    setLatencySamples(11 * 48);

But negative numbers don’t seem to have an effect. I’m lost on this….

2

What you are dealing with is the latency inside the DAW (not the plugin). i.e. there is a time difference between when your MIDI leaves your plugin - and when it exits the DAW to your MIDI hardware.

[Plugin]→[DAW]→[Hardware]

The MIDI that leaves your plugin directly to the hardware is getting there ‘first’, because it is not subject to the DAWs internal latency.
[Plugin]→[Hardware]

So it will not help to report the plugin as having latency, since it probably has none.

What you need to do is to delay the MIDI from the plugin that is going direct to the hardware.
I’m not sure which plugin standards (if any) have a way to query that latency of the DAW. So an alternative might be to let the end-user manually enter the latency.

A secondary issue is that the plugins processBlock function does not operate in real-time, it typically runs much faster than real time. e.g. if your plugin is reported as using 10% CPU, that means that on average the processBlock function is emitting samples at 10 times faster than the sound card is, with larger pauses between blocks.
So if you want to emit MIDI directly to hardware you’re gonna have to spin up a worker thread that schedules MIDI according to the ‘wall clock’ not the music timing information from the plugins playhead.

TIL: It’s complicated. Hope that makes sense.

3

I’m already doing it according to the wall clock so that’s not a problem. Everything is time-stamped with sample offsets, and it works. The MIDI I emit directly to a port is exactly the same as what comes out of the DAW; just the latter ends up delayed.

In macOS Reaper, it’s only 11 or 15 ms. But for example, in Digital Performer using the VST3, you can select the output of the VST3 plugin as an input to a track, and then route that track to the hardware. This way you can record the output of the VST (i.e. phrases, arpeggios).

If you use MIDI Input to trigger the MIDI generation, the direct output starts instantly (as it should). The output of the VST3 through the DAW comes out 150 ms later. Obviously this is unacceptable and delaying the “direct MIDI” to match this wouldn’t seem to be any solution. Rather, I’m wondering why the output of the VST3 is so delayed.

4

To explain what you are experiencing: In my DAW since the audio output of the DAW incurs latency - the DAW delays MIDI output by the same amount so that the listener experiences hardware and software instruments playing in sync with each other. i.e. the software instruments have latency, so the hardware must be compensated to match.

the direct output starts instantly (as it should)

That’s an incorrect assumption. You have no control over the DAW. You can’t make it output MIDI any sooner. You control only your plugin, so if want everything in sync, it’s up to your plugin to add latency to it’s direct MIDI output.

5

Thanks, I guess we’re talking about two different use cases here. I’m not writing the DAW, so I have no control over what it does. I’m just a plugin that generates MIDI, and I just would like it to come out of the host earlier so it matches the audio of other plugins.

6

I develop MIDI plugins and I’ve never had to deal with any latency.

7

I think this is leading you into a blind alley. Sure, you can leave the direct MIDI outputs in your plugin for testing purposes - but do not use it for metrics in measuring performance. Measure, instead, your plugins real latency through the DAW - and not just one DAW but as many as you can test.