Problem with Sysex transfer over USB MIDI on Mac

Moin,

I have a problem sending Sysex dumps of a moderate length of 1330 bytes over USB on a Mac.

Looking into sendMessageNow() I see that Sysex is split into chunks with max 256 bytes that are
sent individually.

On the receiving end, every 256th byte is lost. This is not very surprising:

According to the USB Midi specification, sysex is sent in event packets with three bytes payload each.
There are special codes for ending on 1st, 2nd or 3rd byte, but not for transmitting a single sysex
byte. So as 256%3 == 1, the underlying driver will have a real problem encoding the last byte for
the 256-byte chunk transmission. I think it is simply skipped, exactly what I see on the receiving end.

When I change some code in juce_mac_CoreMidi.cpp very sloppily to this:

if (message.isSysEx())
    {
        const int maxPacketSize = 3256;
...

Everything works fine for me.

So my questions …
Has anyone else noticed this?
Why is sysex split into 256 byte chunks?

Thanks,
Stefan

Hi Stefan,

I’ve also been having problems sending large packet sizes over sysex, and I initially also tried increasing the packet size to a larger number to avoid having the messages split into packets. But I noticed that CoreMidi seems to split the messages into 128 byte packets itself if the messages exceed a certain size and the same problems occur. Having read your message I tried decreasing the packet size instead to 255 and I’m not seeing the problems I saw before… I’m just starting to test though, and I’ll report back to this thread whether that fixes the problems I was having in all contexts…

Best,
Greg

Hi Stefan, Greg and hopefully Jules

I have a very similar problem to what you describe. I have different applications I have developed using older versions of Juce that will send “larger” sysex packages (around 500 bytes) with no problems at all. Now I am developing a new application using the latest Juce version and this application sends out “large” sysex packages as well. As you also experience bytes gets lost! Increasing the 256 bytes maximum package size in the sendMessageNow method fixes the problem. It would be very nice to know why this limit has been added or even better what the problem is when trying to send a package larger that this limit. Any updates from any of you on this topic?

Best regards
John Parbo Andersen

Hi John,

I’ve been testing the sysex behavior with an OSX build of an app that sends firmware updates, presets and smaller control messages over sysex. Like I said in a previous post, I decreased the packet size to 255 rather than increasing it, and now the messages seem to work for a wide variety of message sizes ranging from 20 bytes for certain control messages, 500 bytes for certain preset messages, and 4000 bytes for chunks taken from a firmware update.

We’re going to have a tester come in next week or the following and try the app out on different OSX OS versions. I’ll post back to this thread whether that fixes the problem on Leopard, Snow Leopard, Lion and so on…

Best,
Greg

Over on the midibox.org forums this problem has been noted for longer already.

There are essentially two factors out of control that cause this behaviour:

  1. USB over MIDI protocol implementation
    USB over MIDI is implemented as raw bulk transfers to a single USB device endpoint, as in the specification.
    The segmentation into chunks is caused by the USB chipset driver that will wait for a buffer to fill or timeout before things actually get sent over the wire.

  2. Hardware implementation quirks
    Lots of USB MIDI interfaces tend to overload on long SysEx messages and behave in unexpected ways.
    There are actually only a few interfaces that were found out to be usable for large SysEx transfers at all
    (you can find a maintained list of those interfaces on the midibox wiki pages).

Hi Lucem and Greg

Thank you for answers.

Greg: Nice to know that you have tested it (and still is). We are also doing further testing here and I will post updates here.

Lucem: Thank you for your info. I searced the midibox forum a little but could not find any topics discussing this problem with Juce splitting data into 256 bytes long packages and then losing one byte per package?
As I understand you comments these are related to WHY the data should be split into smaller packages or?

Cheers

John Parbo Andersen

I was actually referring to a general problem in MIDI over USB implementation, which not only affects Juce applications.

The Midibox project uses SysEx transfers for firmware updates and preset data transfer, and this is where the problems surfaced.
Have a look at following links:

http://ucapps.de/mbhp_usb_pic.html
Contains an elaborate description of the quirks with USB over MIDI, not only specific to this device’s problems, but there is a later section
detailing an effect dubbed “blocked pipes”.

http://www.midibox.org/dokuwiki/doku.php?id=midi_interface_blacklist&s[]=usb
http://www.midibox.org/dokuwiki/doku.php?id=midi_interface_whitelist&s[]=usb
A blacklist and a whitelist of USB MIDI interfaces with special attention to their SysEx transfer abilities.

I did not link to the forum itself since usually this kind of information ends up in the project descriptions respective the wiki anyway.

I have done some further investigation on this issue and here are the results.

I have made a test setup as follows:

  • My Mac is sending out midi sysex data from a little test app I have made (in Juce of course).
  • The midi out of the Mac is connected to a midi input on my PC.
  • The data perception on the PC side is done using Midi Ox.
  • Two different midi interfaces has been tested: CakeWalk UM-1G and M-Audio USB Uno

I have then send sysex data from the Mac to the PC and I see that there is no problem with the Uno interface regarding of applying any fix related to the maxPacketSize in sendMessageNow or not.
The CakeWalk interface however do have a problem. At intervals between 264 bytes and 339 bytes two “extra” bytes of value 0x00 is inserted into the stream of data. The intervals between the
wrongly inserted bytes are always the same if I run the same test again. Setting maxPacketSize to 255 fixes this issue.

For this first part of the test I have used the build in USB Midi driver on the Mac. I then installed the dedicated driver for the CakeWalk interface and this fixed the problem (works with no maxPacketSize fix applied).

So to conclude. Some interfaces has a problem when using the build in USB midi driver with the latest version of Juce. Applying the fix Greg posted at the top of this thread (set maxPacketSize to 255) seems to fix this issue.

@Greg: You wrote that you saw that bytes where lost, which also makes sense even though I see the opposite. What interface are you using?

Thank you for your help

Best regards

John Parbo Andersen

Pretty old thread, but looks like I have a similar problem today with the new coremidi implementation. I’m using three different midi Interface, motu 828es (doesn’t work with Sysex > 255), a miditech 4x4 (same), and a cheap ESI (just two cables, and that works fine!). On Windows I could fix it on all three interfaces by splitting the messages to 255 byte packages. The workarround does not work on OSX, and the code referenced above using maxPacketSize is not reached anymore, juce uses newer coremidi implementation now. I tried but don’t really understand what’s going on there…

A USB MIDI interface that transmits received data to serial MIDI can have different implementations as you figured out, one problem is that USB transmits faster than serial MIDI, so sysex needs to be buffered. If the buffer is too small it might send garbage or skip some data.

Also, the interface could ignore USB single byte messages. This is a bit uncommon, but there are some hardware devices that insert those into the receiving sysex stream. For hardwares I made, this worked surprisingly well with the borked 2013 juce midi splitting data into 256 byte blocks:

The underlying OS MIDI system was asked to send a partial sysex messages with data not divisible by 3, which cannot be properly transmitted according to USB MIDI standard for sysex. So the OS does the second next best thing and transmits the orphan byte as a single byte message.

So bottom line, if you use a USB MIDI to serial interface, don’t expect sysex to work reliable with all message lengths, and done blame JUCE for it. If it works on Windows and not on Mac, one possible explanation is that the Windows inserts longer pauses between transmissions and the interfaces have less buffering problems.

If the receiving end is a hardware that expects longer sysex, it should work out of the box, I am confident juce maintainers fixed this by now. Or maybe not…

If you work on the firmware of a USB device, maybe check for single byte messages received from USB.

Stefan,

thanks a lot for coming back on this. I don’t want to blame anyone. Windows didn’t work out of the box too, same problem there, but the workarrpound idea from Greg, splitting the message into 255 byte fragments, applied in my sending application, made it working on the motu and the miditech hardware interfaces (only the motu comes with its on drivers). This is the code for sending:

const int MAX_SEND = 255;
const int SLEEP_PERCENTAGE = 100;
if (sendBuffer[0] == SX_START) {
if (sendBufferLength > MAX_SEND) {
for (int i = 0, j = 0; i < sendBufferLength; i+= MAX_SEND) {
j = std::min(MAX_SEND, sendBufferLength - i);
// juce assert may fail in debug mode if last part is less than 4 bytes
MidiMessage msg = MidiMessage(&sendBuffer[i], j);
SendToOutputs(msg); // calls juce outDevice->sendMessageNow(msg)
// 31250 bits/s, 8 bit +start/stop = 3,125 bytes per ms or 0,96 ms for 3 bytes or 32/100 ms per byte
std::this_thread::sleep_for(std::chrono::milliseconds(SLEEP_PERCENTAGE * j * 32 / 100 / 100));
}
}

I have tried different sleep and fragment size combinations.

The receiving part is a vintage hardware synthesizer with original (serial) midi but without own usb, and the data transmitted is a bank dump - which is a pretty normal thing to do, and most midi interface availlable are USB.

So I’m wondering if I am missing something. or if I may enforce using the old APIs to get it working, or whatever …

I fed the output of the motu interface now to the input of the (working) ESI, and tried sending the sysex in one piece without the mentioned workarround. The first 187 bytes are correct, then there is 0x55 and 0x10 inserted, then the original message continous… strange.