I have a problem sending Sysex dumps of a moderate length of 1330 bytes over USB on a Mac.
Looking into sendMessageNow() I see that Sysex is split into chunks with max 256 bytes that are
sent individually.
On the receiving end, every 256th byte is lost. This is not very surprising:
According to the USB Midi specification, sysex is sent in event packets with three bytes payload each.
There are special codes for ending on 1st, 2nd or 3rd byte, but not for transmitting a single sysex
byte. So as 256%3 == 1, the underlying driver will have a real problem encoding the last byte for
the 256-byte chunk transmission. I think it is simply skipped, exactly what I see on the receiving end.
When I change some code in juce_mac_CoreMidi.cpp very sloppily to this:
if (message.isSysEx())
{
const int maxPacketSize = 3256;
...
Everything works fine for me.
So my questions …
Has anyone else noticed this?
Why is sysex split into 256 byte chunks?
I’ve also been having problems sending large packet sizes over sysex, and I initially also tried increasing the packet size to a larger number to avoid having the messages split into packets. But I noticed that CoreMidi seems to split the messages into 128 byte packets itself if the messages exceed a certain size and the same problems occur. Having read your message I tried decreasing the packet size instead to 255 and I’m not seeing the problems I saw before… I’m just starting to test though, and I’ll report back to this thread whether that fixes the problems I was having in all contexts…
I have a very similar problem to what you describe. I have different applications I have developed using older versions of Juce that will send “larger” sysex packages (around 500 bytes) with no problems at all. Now I am developing a new application using the latest Juce version and this application sends out “large” sysex packages as well. As you also experience bytes gets lost! Increasing the 256 bytes maximum package size in the sendMessageNow method fixes the problem. It would be very nice to know why this limit has been added or even better what the problem is when trying to send a package larger that this limit. Any updates from any of you on this topic?
I’ve been testing the sysex behavior with an OSX build of an app that sends firmware updates, presets and smaller control messages over sysex. Like I said in a previous post, I decreased the packet size to 255 rather than increasing it, and now the messages seem to work for a wide variety of message sizes ranging from 20 bytes for certain control messages, 500 bytes for certain preset messages, and 4000 bytes for chunks taken from a firmware update.
We’re going to have a tester come in next week or the following and try the app out on different OSX OS versions. I’ll post back to this thread whether that fixes the problem on Leopard, Snow Leopard, Lion and so on…
Over on the midibox.org forums this problem has been noted for longer already.
There are essentially two factors out of control that cause this behaviour:
USB over MIDI protocol implementation
USB over MIDI is implemented as raw bulk transfers to a single USB device endpoint, as in the specification.
The segmentation into chunks is caused by the USB chipset driver that will wait for a buffer to fill or timeout before things actually get sent over the wire.
Hardware implementation quirks
Lots of USB MIDI interfaces tend to overload on long SysEx messages and behave in unexpected ways.
There are actually only a few interfaces that were found out to be usable for large SysEx transfers at all
(you can find a maintained list of those interfaces on the midibox wiki pages).
Greg: Nice to know that you have tested it (and still is). We are also doing further testing here and I will post updates here.
Lucem: Thank you for your info. I searced the midibox forum a little but could not find any topics discussing this problem with Juce splitting data into 256 bytes long packages and then losing one byte per package?
As I understand you comments these are related to WHY the data should be split into smaller packages or?
I was actually referring to a general problem in MIDI over USB implementation, which not only affects Juce applications.
The Midibox project uses SysEx transfers for firmware updates and preset data transfer, and this is where the problems surfaced.
Have a look at following links:
http://ucapps.de/mbhp_usb_pic.html
Contains an elaborate description of the quirks with USB over MIDI, not only specific to this device’s problems, but there is a later section
detailing an effect dubbed “blocked pipes”.
I have done some further investigation on this issue and here are the results.
I have made a test setup as follows:
My Mac is sending out midi sysex data from a little test app I have made (in Juce of course).
The midi out of the Mac is connected to a midi input on my PC.
The data perception on the PC side is done using Midi Ox.
Two different midi interfaces has been tested: CakeWalk UM-1G and M-Audio USB Uno
I have then send sysex data from the Mac to the PC and I see that there is no problem with the Uno interface regarding of applying any fix related to the maxPacketSize in sendMessageNow or not.
The CakeWalk interface however do have a problem. At intervals between 264 bytes and 339 bytes two “extra” bytes of value 0x00 is inserted into the stream of data. The intervals between the
wrongly inserted bytes are always the same if I run the same test again. Setting maxPacketSize to 255 fixes this issue.
For this first part of the test I have used the build in USB Midi driver on the Mac. I then installed the dedicated driver for the CakeWalk interface and this fixed the problem (works with no maxPacketSize fix applied).
So to conclude. Some interfaces has a problem when using the build in USB midi driver with the latest version of Juce. Applying the fix Greg posted at the top of this thread (set maxPacketSize to 255) seems to fix this issue.
@Greg: You wrote that you saw that bytes where lost, which also makes sense even though I see the opposite. What interface are you using?
