In our host app incoming MIDI messages are immediately sent to the currently armed instrument (a hosted plug-in) for input monitoring. There is often unbearable latency because messages can only be added to the next block at sample position zero. The current block has already sailed. This is a well-known issue.
Every time I want to record something with precise timing I am so confused by the latency that I have to mute the instrument in order to be able to play coherently. I might be a bit sensitive to this but it is certainly a common issue.
Is there potentially a way (evil hack) to add a message to a block that is already being processed? Like keeping a reference to the last block and adding the event in hindsight at the sample position that corresponds to current wall time?
Time is monotonously increasing, so it should be impossible to add an event in the past (even if so, it would simply not be processed by the plug-in anymore).
MidiBufferIterator could be changed to allow modification under iteration, unless it already does. I have no clue however if all hosted plug-ins will play along (JUCE plugins will probably do fine).
Any ideas? Fixing this could be a boon for everyone.
How large are the blocks you’re currently sending to the plug-in?
Adding messages to a block that is already being processed would be equivalent to rendering shorter blocks, since that allows you to insert MIDI messages earlier.
Mostly on macOS but the app is deployed to Windows, too.
Admittedly the weighted piano keys of a master keyboard add some lag to the mix, but 11.6 ms latency (512 samples) is already a challenge, especially when the sound has a soft envelope (e.g. staccato strings). Every ms that can be shaved off internally will help.
The problem is that the plug-in will probably already be done processing the block by the time you insert additional MIDI events into the buffer (assuming this is even possible).
That should be impossible. You are playing guided by the sound that comes out of the speakers. That has been rendered at least 1 buffer in the past. By the time your message arrives in JUCE the next buffer has just begun processing (if at all).
If the block size is 512 samples it describes about 11 ms of audio. But rendering that block will take a fraction of that, likely less than 1 ms. So if a new MIDI event comes in 5 ms after the plugin started rendering that block, the plugin will already have finished rendering that block and so trying to insert this MIDI event into the current block achieves nothing.
This is just part of any MIDI host when used in real time. The only solution is to reduce your block size until you’re comfortable with the latency AND there’s no CPU issues. Plus of course your audio interface adds it’s own latency. As someone else said ASIO is much lower latency. Personally i find that I’m okay at 512 samples with ASIO and my Behringer UMC 1820 even for playing live on-stage with my band (Pinc Ffloyd), 256 is pretty much imperceptible but I don’t totally trust it in a live scenario. It’s all about compromise. Personally I’d recommend dropping to 256 or get a better audio interface to minimise the additional latency.
But I get what you’re saying. If you have a block size of 512 and whatever is doing the audio processing on a block has only got half way through the job when another note is played why can’t that message just be added in and be factored in during the modification of the buffer. It’s almost like you want the MIDI and the audio being handled separately and when you’re populating the audio buffer just be able to freely read from the midi input. These things we do with VSTs wasn’t really what MIDI was made for. It was made for hardware talking to other hardware in real time, and it’s still superb at that even 40+ years on from when it was created, but audio processing on a computer introduces new challenges.
Actually the more I think about this the more I wonder if it’s doable. Would be awesome if it was doable! I’m building a VST host app at the moment, might have a tinker and let you know if I get anywhere (eventally - I’ve only just started it!)
But rendering that block will take a fraction of that, likely less than 1 ms.
Ouch yes, somehow didn’t think about that. There is no reason for a plug-in to consider absolute time in any way. There is so much other work a host has to do, the less time is consumed by plug-ins the better. I clearly spent too much time coding financial trading platforms.
If it is somehow possible to set a small block size only for the armed/monitored instrument(s), that would be a great solution. Like all other plug-ins at 512 samples and only the one being monitored at 64 samples.
Now the challenge is how to accomplish that. There needs to be a loop in AudioProcessorGraph that sends multiple processBlock() to the plug-in at 64 samples each, while filling the 512 buffer of the graph. It also needs to listen to incoming time-stamped MIDI from outside the block passing mechanism.
That is absolutely doable. Probably not without rewriting AudioProcessorGraph though.
You might try setting up two separate graphs with fifos to handle the bridging. Updates get more tricky though as you need to handle plugins switching between graphs and position syncing and all that.
