Large mnidi messages and plugin wrapper


#1

When using the plugn wrapper, long messages don’t come all to the plugin. If a messages is split only the first portion is delivered the rest not, here is an example.
This is a waveshape dump request to a DSI evolver, and it’s response using raw devices (a simple app that opens MIDI IN/OUT) and the same thing using the plugin wrapper
and parsing the messages in the process() loop. This is in standalone build in terms of the plugin wrapper.

[USING RAW DEVICES]
>>> f0 01 20 01 0b 01 0f
<<< f0:01:20:01:0a:01:00:00:00:00:02:00:04:00:00:06:00:08:00:0a:00:0c:00:00:0e:00:10:00:12:00:00:14:00:16:00:18:00:1a:00:00:1c:00:1e:00:20:00:00:22:00:24:00:26:00:28:00:00:2a:00:2c:00:2e:00:00:30:00:32:00:34:00:36:00:00:38:00:3a:00:3c:00:00:3e:00:40:00:42:00:44:00:00:46:00:48:00:4a:00:00:4c:00:4e:00:50:00:52:00:00:54:00:56:00:58:00:00:5a:00:5c:00:5e:00:70:09:00:05:50:0a:0b:60:03:50:0b:00:05:14:70:09:50:07:40:06:20:00:0b:00:04:10:0d:10:04:54:20:0c:60:06:50:08:70:20:0b:20:06:40:0f:10:06:45:10:11:40:08:60:11:50:02:0d:10:0f:40:16:40:0a:51:30:20:10:0d:70:2b:70:62:06:20:25:00:4b:10:74:21:70:7f:00:00:10:00:70:54:0b:00:35:60:5a:10:79:6e:10:54:70:72:50:5f:40:57:75:40:69:70:70:30:72:2b:20:6e:40:77:70:6e:70:5f:79:40:70:60:79:10:74:7a:30:77:20:79:60:73:70:77:7b:70:72:00:7c
<<< 60:74:3a:40:79:30:78:10:76:00:55:7b:30:74:20:7c:30:74:2e:00:7b:10:76:30:78:40:77:79:60:74:00:7c:70:72:2f:70:7b:60:73:20:79:30:7d:77:10:74:60:79:40:70:6a:70:79:70:6e:40:77:20:75:6e:30:72:70:70:40:69:3b:40:75:50:5f:70:72:10:55:54:10:79:60:5a:00:35:08:70:0b:10:00:f7

[USING PLUGIN WRAPPER]
>>> f0:01:20:01:0b:01:0f
<<< f0:01:20:01:0a:01:00:00:00:00:02:00:04:00:00:06:00:08:00:0a:00:0c:00:00:0e:00:10:00:12:00:00:14:00:16:00:18:00:1a:00:00:1c:00:1e:00:20:00:00:22:00:24:00:26:00:28:00:00:2a:00:2c:00:2e:00:00:30:00:32:00:34:00:36:00:00:38:00:3a:00:3c:00:00:3e:00:40:00:42:00:44:00:00:46:00:48:00:4a:00:00:4c:00:4e:00:50:00:52:00:00:54:00:56:00:58:00:00:5a:00:5c:00:5e:00:70:09:00:05:50:0a:0b:60:03:50:0b:00:05:14:70:09:50:07:40:06:20:00:0b:00:04:10:0d:10:04:54:20:0c:60:06:50:08:70:20:0b:20:06:40:0f:10:06:45:10:11:40:08:60:11:50:02:0d:10:0f:40:16:40:0a:51:30:20:10:0d:70:2b:70:62:06:20:25:00:4b:10:74:21:70:7f:00:00:10:00:70:54:0b:00:35:60:5a:10:79:6e:10:54:70:72:50:5f:40:57:75:40:69:70:70:30:72:2b:20:6e:40:77:70:6e:70:5f:79:40:70:60:79:10:74:7a:30:77:20:79:60:73:70:77:7b:70:72:00:7c

the second part of the messages when using raw devices the one that starts with [60:74] comes in the handleIncomingMidiMessage() not the handlePartialSysexMessage() method, witch is weird i think.


#2

No idea why that’d happen, but also can’t really think of anything that could be done to change it. You’re rather at the mercy of the OS when it comes to message delivery.


#3

Perhaps i wasn’t clear :slight_smile: when i’m saying i’m using the devices directly i’m still using JUCE classses.

It doesn’t work only when used in the wrapper (those messages should appear in the process() loop since they appear in the MidiInput callback). I doubt this has anything to do with the OS, it works, just not in the case of the Audio Plugin Wrapper. MidiInput class works fine.


#4

So you mean it might be the host that’s breaking up the messages?


#5

well no, since it’s in standalone mode and i’m using the standalone wrapper part i think it might be the wrapper’s fault. i didn’t check that in VST mode, cause i know that most of the hosts are broken in terms of MIDI SYSEX, but it has to work in standalone mode i need to send large amount of waveform data and download it too.


#6

Well, looking at the standalone wrapper class, it should be fine… There’s really nothing in there, it just takes the message from the device and shoves it into the buffer. Can’t really see how it could go wrong…


#7

There must be something wrong this is how it looks like whem i insert some logging into the juce_AudioFilterStreamer.cpp

EdoMidiDeviceManager::sendMessageNow -----------------------------------------------
f0 01 20 01 0b 01 0f
EdoMidiDeviceManager::sendMessageNow -----------------------------------------------
AudioFilterStreamer::handleIncomingMidiMessage
f0 01 20 01 0a 01 00 00 00 00 02 00 04 00 00 06 00 08 00 0a 00 0c 00 00 0e 00 10 00 12 00 00 14 00 16 00 18 00 1a 00 00 1c 00 1e 00 20 00 00 22 00 24 00 26 00 28 00 00 2a 00 2c 00 2e 00 00 30 00 32 00 34 00 36 00 00 38 00 3a 00 3c 00 00 3e 00 40 00 42 00 44 00 00 46 00 48 00 4a 00 00 4c 00 4e 00 50 00 52 00 00 54 00 56 00 58 00 00 5a 00 5c 00 5e 00 60 00 00 62 00 64 00 66 00 00 68 00 6a 00 6c 00 6e 00 00 70 00 72 00 74 00 00 76 00 78 00 7a 00 7c 28 00 7e 00 00 00 02 00 55 04 00 06 00 08 00 0a 2a 00 0c 00 0e 00 10 00 55 12 00 14 00 16 00 18 2a 00 1a 00 1c 00 1e 00 55 20 00 22 00 24 00 26 2a 00 28 00 2a 00 2c 00 55 2e 00 30 00 32 00 34 2a 00 36 00 38 00 3a 00 55 3c 00 3e 00 40 00 42 2a 00 44 00 46 00 48 00 55 4a 00 4c 00 4e 00 50 2a 00 52 00 54 00 56 00 55 58
AudioFilterStreamer::handleIncomingMidiMessage
00 5a 00 5c 00 5e 2a 00 60 00 62 00 64 00 55 66 00 68 00 6a 00 6c 2a 00 6e 00 70 00 72 00 55 74 00 76 00 78 00 7a 0a 00 7c 00 7e f7
AudioFilterStreamer::audioDeviceIOCallback
f0 01 20 01 0a 01 00 00 00 00 02 00 04 00 00 06 00 08 00 0a 00 0c 00 00 0e 00 10 00 12 00 00 14 00 16 00 18 00 1a 00 00 1c 00 1e 00 20 00 00 22 00 24 00 26 00 28 00 00 2a 00 2c 00 2e 00 00 30 00 32 00 34 00 36 00 00 38 00 3a 00 3c 00 00 3e 00 40 00 42 00 44 00 00 46 00 48 00 4a 00 00 4c 00 4e 00 50 00 52 00 00 54 00 56 00 58 00 00 5a 00 5c 00 5e 00 60 00 00 62 00 64 00 66 00 00 68 00 6a 00 6c 00 6e 00 00 70 00 72 00 74 00 00 76 00 78 00 7a 00 7c 28 00 7e 00 00 00 02 00 55 04 00 06 00 08 00 0a 2a 00 0c 00 0e 00 10 00 55 12 00 14 00 16 00 18 2a 00 1a 00 1c 00 1e 00 55 20 00 22 00 24 00 26 2a 00 28 00 2a 00 2c 00 55 2e 00 30 00 32 00 34 2a 00 36 00 38 00 3a 00 55 3c 00 3e 00 40 00 42 2a 00 44 00 46 00 48 00 55 4a 00 4c 00 4e 00 50 2a 00 52 00 54 00 56 00 55 58

my manager sends the dump request, the handleIncomingMidiMessage callback gets called twice with both the messages, but the audioDeviceIOCallback sends only one message to the process() loop for the filter, the second message gets lost somewhere in the midiCollector.removeNextBlockOfMessages (midiBuffer, numSamples); call.

This is the code block i inserted in the IO callback

if (midiBuffer.getNumEvents() > 0)
			{
				Logger::writeToLog (T("AudioFilterStreamer::audioDeviceIOCallback"));
				MidiBuffer::Iterator i(midiBuffer);
				MidiMessage m(0xf0);
				int p;

				while (i.getNextEvent (m, p))
				{
					Logger::writeToLog (String::toHexString (m.getRawData(), m.getRawDataSize()));
				}
			}

            filter.processBlock (output, midiBuffer);

and this is the midi callback part

void AudioFilterStreamer::handleIncomingMidiMessage (MidiInput* source, const MidiMessage& message)
{
	Logger::writeToLog (T("AudioFilterStreamer::handleIncomingMidiMessage"));
	Logger::writeToLog (String::toHexString (message.getRawData(), message.getRawDataSize()));

#if JucePlugin_WantsMidiInput
    midiCollector.addMessageToQueue (message);
#endif
}

maybe the numSamples is causing the midiCollector to discard the second message, or maybe that the first message is 255 bytes long (i guess the 255 is the driver limit on one midi message).


#8

I can’t figure out what’s going on there, and don’t have a device to test that sends out such large blocks. It must just be a buffer limit somewhere - maybe you could trace it through and spot where it’s getting cut off?


#9

findActualEventLength() returns 0 on the second part, so it doesn’t get added to the queue in the MidiMessageCollector, looks like MidiBuffer does not like those messages, it should be treated as a partial sysex message and appear in the second callback handlePartialSysexMessage(). that looks to be the problem.


#10

Ah, I see. Looks like the MidiMessageCollector hasn’t got anything to handle multi-part messages. I’ll make a note to take a look, but am a bit busy on other code areas at the moment.


#11

sure thing, for now i’ll deal with raw devices for some of this large midi data stuff.


#12

The problem I have now seems to be related to the same issue.

A long sysex message (267 Bytes) doesnt come through completely and handlePartialSysExMessage doesn’t get called at all!

I am on Windows and not using VST here, it’s in Stand Alone.

I’ve update to last GIT - still the same but I remember a previous juce build where it did work.

Any updates here?

Joerg


#13

handlePartialSysExMessage is implemented on the Mac, but I don’t think that was ever handled on windows, was it…?


#14

I think it was… As said, somewhere in the darkest areas of my brain I remember it working, so to say.
But I could be wrong…

Anyway, how to deal with that issue?


#15

No… pretty sure that it didn’t. I certainly wouldn’t have removed something like that. If you can find an old version that works, I’d love to be proved wrong!

I guess the answer would be for me to investigate, but I don’t have any kind of midi equipment set up to test with right now, and have a million other pressing things to do… If you want to do some digging I’d be glad to help with ideas!


#16

Jules,

I misread atoms post. Its actually the opposite thing. If i compile a VST plugin I do receive the complete sysex message (>256 bytes)
w/o any problems (Cubase 5.1). As a stand alone app, I receive both parts of a splitted message via handleIncomingMidiMessage
using juce midi input.
First call includes sysex start byte F0 and the second call includes sysex stop byte F7 cause the message length is only 267 bytes…

Is this the normal behaviour? If yes, what if messages are bigger than 512 bytes?

I’d expect to receive parts of a message >512 bytes within the handlepartialsysex callback if these parts neighter
contain F0 nor F7.

I could live with the above behavior but I want to also compile a MAC version and you said that the handlepartialsysex is only for MAC.
That makes me a bit nervous cause I thought I can use same code for both, MAC and Windows.

Joerg


#17

Joerg,

I have some code that handles incoming sysex for both Windows and Mac, and while I can’t say that I’ve completed tested it, it has worked on both platforms in all of my testing. I hope this is helpful. Here is a code snippet:

[code]
void DK10EditorJuceComponent::receiveDk10Data( int sysexBufferLen, unsigned char* sysexBuffer )
{
// if the data fits in the internal buffer
if( mSysexCurOffset + sysexBufferLen < kSysexBufferSize )
{
// if it the start of the msg, and it’s the start-of-sysex, or any byte after the start
if( ((mSysexCurOffset == 0) && (*sysexBuffer == 0xf0)) ||
(mSysexCurOffset != 0) )
{
memcpy( &mSysexBuffer[ mSysexCurOffset ], sysexBuffer, sysexBufferLen );
mSysexCurOffset += sysexBufferLen;

        if( mSysexBuffer[ mSysexCurOffset - 1 ] == 0xf7 )
        {
            if( mDk10Data.putSysexData( mSysexCurOffset, (unsigned char *)&mSysexBuffer[ 0 ] ) )
            {
                // send a message to update display
                //displayCurrentData();
            }

            // reset the buffer
            mSysexCurOffset = 0;
        }

    }
    else
    {
        // ERROR
        // wrong data
        // reset the buffer
        mSysexCurOffset = 0;
    }

}
else
{
    // ERROR
    // too much data for the buffer
    // reset the buffer
    mSysexCurOffset = 0;
}

}

//
// Midi callbacks
//
void DK10EditorJuceComponent::handleIncomingMidiMessage( MidiInput source, const MidiMessage &message )
{
unsigned char
sysexData = (unsigned char*)message.getRawData();
int sysexDatalen = message.getRawDataSize();

receiveDk10Data( sysexDatalen, sysexData );

};

void DK10EditorJuceComponent::handlePartialSysexMessage( MidiInput *source, const unsigned char *messageData, const int numBytesSoFar, const double timestamp )
{
receiveDk10Data( numBytesSoFar, (unsigned char *)messageData );
}[/code]


#18

Hi cpr,

thank you for the code snipped!
I do have code that works perfect in Windows but I didn’t realize that MAC uses different
methods then Windows. I am talking about handlePartialSysexMessage.

That is my concern because I thought that if I strictly use juce classes (and that is what I do),
then it will work on both, MAC and PC using the same code.

Anyway, I’ll implement the handlePartialSysexMessage code as you did.

Thank you very much for your help!
Joerg