VST3 MIDI events ordering

Hi,

My plugin, when running on Dorico, sometimes receives (when starting playback) an ‘AllNotesOff’ and a ‘NoteOn’ event with the exact same sampleNumber value. Since those two events start on the same sample, their relative ordering is unsure. If the NoteOn is handled before AllNotesOff, then the note is immediately cut, otherwise it is playing normally. The issue with the VST3 stuff is that the noteOn/noteOff stuff events are added to the midiBuffer object with:

MidiEventList::toMidiBuffer (midiBuffer, *data.inputEvents);

And the AllNotesOffs/AllSoundsOff, which are controller events, are added to the midiBuffer object by the call to:

processParameterChanges (*data.inputParameterChanges);

And these two calls currently happen in that order in the juce_VST3_Wrapper.cpp, so when the VST3 timestamp of the NoteOn and AllNotesOff is the same, the AllNotesOff event will be always placed after the NoteOn in the MidiBuffer object. NoteOns are cut off immediately, and users are unhappy.

So I suggest to fix this by moving the

   #if JucePlugin_WantsMidiInput 
    if (data.inputEvents != nullptr)
        MidiEventList::toMidiBuffer (midiBuffer, *data.inputEvents);
   #endif

in the processAudio() function, just after the call to

   if (data.inputParameterChanges != nullptr)
        processParameterChanges (*data.inputParameterChanges);

that way, in case of simultaneous controller and noteon/off events, the controller events will always be placed before the noteon/off events. It seems to me it is better that way than the other.

OK sounds like a reasonable change. However, I moved the processParameterChanges before the MidiEventList::toMidiBuffer so that the processParameterChanges is not in the templated code-path anymore - I think that makes more sense. Also, processParameterChanges was called inside the audio callback lock which I think was unnecessary.

The fix for this is on develop with commit d95edfd.

Great, thanks Fabian !

On this topic, I was not aware that VST3 had separated notes from CC, PitchBend, AfterTouch and ProgramChange messages to separate serialized buffers. That is really problematic for working with many other non-VST3 solutions that depend on a single serial queue.

I guess VST3 basically relies on NoteExpressions for ensuring that related CC, PB, AT and PC messages would be ensured to be in front of the notes. But Cubase and Dorico are the only hosts that do anything with NoteExpressions.

I think JUCE’s hosting probably should be improved so that if incoming midi all intermixed in one midi buffer is split to separate buffers in order to callback to a VST3 plugin, then it should do so using NoteExpressions in order to preserve the proper serial ordering…if and when there is more then one midi event on a given sample timestamp.

This would help any hosts created with JUCE, but other hosts out there that support hosting VST3 would still suffer from this problem unless they all do something similar.

Very troubling. Steinberg dropped the ball on this one.

1 Like

Logically, two events with the same timestamp happen at the same time, regardless of what order they may happen to be stored in a list.

For example, a note-on and note-off event at the same timestamp is ambiguous because it could represent one note ending and a second beginning, or alternately could represent a single very short note. There is no “right” way to handle this situation, the best you can do is apply some heuristic that minimizes unpleasant outcomes like stuck notes.

For this reason, you should never deliberately emit ambiguous MIDI streams, nor write software that depends on any particular interpretation of ambiguous MIDI events.

MIDI events are subjects to all types of sorting, splitting, and merging. There is no way to guarantee the preservation of any ordering other than the timestamp. So It is in no way Steinberg’s fault if your software’s correctness depends on anything as random as what order events happen to be stored in a buffer.

you are missing the point. Midi has for decades been a serial stream and the order of events has been extensively used in many capacities… Now we have sample accurate timestamping, but still internally in plugin processing, there is a serial queue of midi events, so that even midi events with this same sample timestamp do NOT actually happen at exactly the same time, they happening in some serial, atomic order. That order is not defined by the timestamp of course, its only defined by the fact that they have historically been in a single midi event queue.

This is absolutely essential for any situation where you have multiple notes starting on the same timestamp, like a chord…and you need to for some reason provide different expression to different notes of the chord. This is particularly a concern for articulation management systems and approaches being used in all DAW’s to provide keyswitches (which sometimes may be CC keyswitches or PC messages) in front of each note of the chord, for poly-articulation chords, such as divisi for example. This has always been possible with a single serial buffer in the past, but is now broken with the VST3 approach of keeping more than one serial queue for different midi event types. As noted, this makes it impossible to re-correlate the originally intended order.

Any midi coming from a midi device outside of a single host…will produce a single midi stream, in serial order, even if some of the events have the same timestamp, the serial order matters and has been used for decades to ensure this kind of thing can work.

Broken now in VST3. That order is lost the minute that midi stream is separated into separate queues with an expectation that some listening software instrument will know how to read those multiple queues in the originally intended order.

1 Like

as I said before, at least a JUCE based host could improve this situation by interpreting incoming midi buffers(with mixed event types) and when there are multiple messages using this same timestamp, one being a note…then use a NoteExpression when handing it to VST3 with multiple queues.

Of course I don’t know whether NoteExpressions will work properly and automatically with all VST3 plugins…that is a question.

A VST3 expression wrapper would not be a bad idea either. Have the wrapper convert the events with the same timestamp to NoteExpressions…before calling back to the VST3. But that is really a hack solution with numerous drawbacks since the actual VST3 would not be exposed to the host.

Here’s another suggestion for JUCE, since NoteExpressions are not currently supported.

When handling VST3, detect when there are serialized midi events with the same midi timestamp and shift their timestamp by one sample each so that all timestamps are unique with no repeats. This results it not completely sample accurate results, but as an option, still might be useful since what happens now, with all the CC’s coming before all the Notes…although sample accurate in audio render, results in lost serialization for midi event handling.

That’s good and robust IMHO, because it will retain the order of events even with when splitting/merging them.

(this use-case is one reason I am keen on MIDI 2.0, which can handle keyswitching cleanly).

Some people would scream about the loss of sample accuracy I reckon, but I for one would welcome an option to do that.

Note that if you ever merge with anything else, you’ll have the same problem regardless of whether you do this, because two different queues could each have an event with the same timestamp, and no rule about which is supposed to be first.