Hey Jules and Juce community,
I’m currently working with a custom USB MIDI device that sends SysEx messages as well as note on/off and aftertouch messages.
The SysEx messages are 9 bytes long and being sent as three 3-byte packets. When just the SysEx messages are being sent all is fine, however when being sent along with something like a note on message, the SySex message is interrupted by the note on message and causes an ‘infinite loop’ within the midi input code in Juce.
To be more specific, it will receive the first packet of the SysEx message (which contains the SysEx start byte, and two Manufacturer ID bytes), but then it will receive a note on message and start looping, calling MidiDataConcatenator::pushMidiData() and processSysex() within it where the MIDI data (‘d’ variable) is always equal to the note on message.
Should the juce code be able to handle interrupted SysEx messages? I have contemplated that it could be a problem with the MIDI device itself however other MIDI software doesn’t seem to crash when the initial SysEx message is interrupted. I’m using the latest Juce tip on OS X.
I don’t think that sysEx should be interrupted in a sense that only a few bytes of the message get through and another one get’s inside. There is some fragmentation possible with SysEx when they’re too big, bigger then the buffer size for the MIDI Device (most of the time it’s about 8kb). I’ve never seen a sysex get split into parts by other messages, the MIDI protocol is a Serial protocol you can’t inject one message into the other, they come one after each other, so in that case it might be some driver issue or something else. In my opinion this is not a valid behavior for a MIDI Device.
I should have specified that this is for USB-MIDI which bundles the MIDI data into packets with 3 databytes each. The note on and aftertouch messages are sent from the software and use the USB-MIDI port as a thru port. The SysEx message has 9 bytes and is thus split into 3 USB packets. It seems like an note-on/aftertouch message will get through before the final SysEx packet. I should also note that this is is a problem on OSX and Windows but not on Linux and could have something to do with the MidiDataConcatenator which is used for Core Audio and Windows but not ALSA.
There may be a bug in the JUCE code, but beyond that, if you have any other MIDI command come along prior to receiving the entire sysex (ie. all of it’s packets), then the sysex reception will be aborted.
Are you saying that the ‘incomplete’ sysex message should stop being sent by the MIDI device once it is interrupted? Or should this abortion be handled at the other end within the Juce code?
MIDI is a serial interface if one device is connected it is it’s job to send the sysex fragmented but not put anything between the fragments, otherwise there is no way of knowing witch part is witch. JUCE handles fragmented SysEx just fine (i did program dumps over 32kb and waveform dumps etc and it all works fine). If a SysEx is interrupted, a message is injected between sysex fragments, then it’s either the device’s fault that’s sending those messages, there is more then one device on the MIDI chain and theirs interrupting each other, there is some sort of a MIDI concatenation/hub device (multiple inputs one output) and that’s handling the messages in the wrong way.
I really don’t understand how can SysEx be fragmented into 3 byte fragments, i tried this only once when hosting VST plugins and trying to push SysEx out of them using 3 byte messages (witch is what most of VST hosts only support the CC/NoteOn 3 byte messages), but that didn’t work out well. But i never heard of a MIDI device (USB?) that fragments long SysEx messages in 3 byte packets, the smallest buffer size i saw was 512 bytes.
Yes the partial sysex messages are being handled fine, until they are being interrupted by another type of message. The problem seems to be within MidiDataConcatenator::pushMidiData() where I’m finding the non-sysex ‘interrupt-ee’ message is recognised as a sysex message as processSysex() is called, and then continuously being called thereafter in an infinite loop. This does not occur on Linux though (as the MidiDataConcatenator class is only needed for Windows and OSX) and everything seems to work fine. So could it be an issue with CoreMIDI and the Windows MIDI stuff?
When monitoring the incoming MIDI using a MIDI Monitor app it see’s the messages coming in in the following order; sysex (first 3-byte packet), Note-on, and then two invalid messages which are the the last two packets of the sysex message, however the app doesn’t crash in any way.
The 3 bytes message (actually a 32bit message) is on the USB driver level, JUCE does not interact with the device on that level. It’s the OS/driver job to put that together, you get an API to listen for MIDI (CoreAudio or CoreMIDI on OSX) and you don’t care about USB packaging (what if the MIDI device is not USB but FireWire or it’s connected via a normal RS232 port).
I see Jules made some changes to the data concatenator, but there is a type warning now, probably harmless but i just wanted to mention it
This is VS2010 SP1
1>c:\devel\juce\modules\juce_audio_devices\native\juce_mididataconcatenator.h(81): warning C4244: '=' : conversion from 'int' to 'juce::uint8', possible loss of data
Is this supposed to be fixed? I believe I have the same problem in one of my applications. Sysex data is being sent by a device, but in the middle of any sysex dump I can also get Note data. Once the note data is received, the handleIncomingMidiMessage() callbacks no longer gets called (neither for the Sysex data nor for the Note data), and the handlePartialSysexMessage() callback is called in an infinite loop, providing the same numBytesSoFar on each call and at a very high rate (i.e. it got stuck in the JUCE midi code). Seems like bug.
