Read and write Midi files

Hi Jules,

I hope you are well.

You may remember my posting some month ago about problems in reading Midi files.

http://www.rawmaterialsoftware.com/viewtopic.php?f=2&t=6049&hilit=Joerg

Now I got faced with a problem doing the opposite thing, writing a Midi file.
I am using the below code and it does create a Midi file which I can easily open using my juce reader code
but it is not readable for other host like Cubase and Ableton.
I am creating Midi files containing Sysex messages only!

MidiMessage msgToPutInFile(*appManager->createSysExMessage(*MemoryBlock blablabla));

MidiMessageSequence seq;
seq.addEvent(msgToPutInFile);
            
MidiFile midiFile;           
midiFile.addTrack(seq);

ScopedPointer <FileOutputStream> fileOutput(fileToWrite.createOutputStream());           
midiFile.writeTo(*fileOutput);

Do I miss something here. The reader code in short is like this:

ScopedPointer <FileInputStream> fileInput(new FileInputStream(file));
				     
MidiFile midiFile;
midiFile.readFrom(*fileInput);
				
for (int i = 0; i < midiFile.getNumTracks(); i++)
{
	for (int j = 0; j < midiFile.getTrack(i)->getNumEvents(); j++)
	{
		if (midiFile.getTrack(i)->getEventPointer(j)->message.isSysEx())				
		{
	                MidiMessage m(midiFile.getTrack(i)->getEventPointer(j)->message);

                        //do something with that message here
                }
        }
}

Thank you for having a look at…

Joerg

My best suggestion is that you look at the output file with a hex editor to see if it’s exactly what you expect.

If it’s what you expect, then either the target program doesn’t read SysEx (but I’m pretty sure Live does) or your idea on what a MIDI file format is, is wrong.

If it is NOT what you expect (my guess), then there’s a problem in the program that isn’t outputting what you expect and we can help you if we see the output…

Yes, that will be the next step. I just thought there might be something missing in my code
like time stamp information I would have to put. Obviously there is nothing wrong so
I’ll investigate in searching the problem using a hex editor.

Reading a MIDI File with JUCE

File myFile = File::createFileWithoutCheckingPath (/*Some juce::String path to a valid MIDI file*/); //Not suggested to use this method exactly - just for show
FileInputStream myStream (myFile);

MidiFile myMIDIFile;
myMIDIFile.readFrom (myStream);

Writing a MIDI File with JUCE

File myFile = File::createFileWithoutCheckingPath (/*Some juce::String path to save a new, hopefully valid, MIDI file*/); //Not suggested to use this method exactly - just for show
FileOutputStream myStream (myFile);

myMIDIFile.writeTo (myStream);

Flow of Code
String path → File → FileInputStream → MIDI File::readFrom
String path → File → FileOutputStream → MIDI File::writeTo

Just an FYI: I’ve no idea why you’re using ScopedPointers - they are unnecessary in these particular situations.

What’s with the MemoryBlock thing? Try adding a message from the pre-created methods first! I suggest to keep things simple to understand it all first, then try adding your own stuff.

Check this method out in juce_MidiMessage.h: static MidiMessage createSysExMessage (const uint8* sysexData, int dataSize) Doesn’t it already do what you need to do? :?

No, I don’t need ScopedPointers here, you’re right. The MemoryBlock is used to create the content of the SySex message.
Juce doesn’t supply such a function to put together my bits and pieces to a MidiMessage.

The file gets written, so I’ll use a hex editor to find the culprit. Must have something to do how
I put the SysEx messages in the file, maybe invalid time stamp???

Thank you anyway.
Joerg

Stop, I do need a ScopedPointer at least for the outputstream because File::createOutputStream() return a pointer to an object
and this has to be deleted unless it returns just a pointer to an object with the same life time like the file it was created from…
But I doubt this because Jules would name it File::getOutputStream().

Unless you’re appending data to the file, then yea - doing it that way is perhaps optimal… But you’re dealing with MIDI (kB of data) - so you could just load everything into memory, perform your operations, and overwrite the file with what you got (like in the example I gave?).

May I miss something but I am not using “FileOutputStream myStream (myFile)” because Jules is not recommending to
use it this way. The doc’s says:

File::createOutputStream() creates an object on the heap.

The Midi files I am creating could reach several Megabytes as they could contain a huge amount of large SysEx Midi messages.
I am not dealing with standard Midi messages.

Yeah… in fact that comment no longer exists, and I’d recommend just creating a local FileOutputStream object on the stack if you only need it temporarily. You can still use the createOutputStream method and a ScopedPointer if you need to keep the pointer somewhere, though that’s probably quite unusual.

Oh, good to know. I was refering to the current docs from the website.
In this case I can do everything on the stack of course.

Thank you guys,

Joerg

Okay, back to the point we’ve started from…

I did some investigation using a hex editor to find the reason why the Midi files I generate
using juce are not readable in i.e. Cubase or Ableton. Just to make it very clear, I am not talking
about Midi files containing standard Midi messages, I am talking about Sysex messages.

I found some differences to readable files:

  1. If I only put 1 track in the Midi file, the first 2 bytes after the 4 track header bytes should stay a 0
    because the file is then a single track file. Juce set it to 1. That might be correct but Midi specs say it has to stay at 0.
    But this is not the reason for failing to read, just a hint!

  2. The following 2 bytes of the status byte “F0” are variable length bytes to tell the reader the length of the data block.
    These 2 bytes are missing if I use juce to write such a Midi file.
    As we still have the same problem while reading such Midi files, (http://www.rawmaterialsoftware.com/viewtopic.php?f=2&t=6049&hilit=Joerg)
    I would like to know Jules whether you have plans to check in a fix or should I do my own class to do it?
    I am just asking because I assume you’re very busy.

I have attached an archive with 2 of such files. The first one ***VC3 is the proper one, the ***VC4 is not readable. May that helps.

Thank you!
Joerg

I’d very much appreciate it if you could give me a hint as to what I should change to fix it - it’d save me the time it’d take to get my head around what’s happening, and you already seem to have a clear idea of what’s going on…

I’ll do but it will take its time as I am moving to a new flat next week and have to finish painting the walls and all this
things…

Thanks

[quote]1) If I only put 1 track in the Midi file, the first 2 bytes after the 4 track header bytes should stay a 0
because the file is then a single track file. Juce set it to 1. That might be correct but Midi specs say it has to stay at 0.
But this is not the reason for failing to read, just a hint![/quote]

From what I’ve understood: you’re saying JUCE is setting the file to MIDI Type 1, when you’re expecting a MIDI Type 0 file because you’re giving it only one track.

You’re misinterpreting the data: JUCE saves MIDI files as MIDI Type 1, which this type of MIDI file has the potential of containing only one track (and more!). [see http://www.indiana.edu/~emusic/etext/MIDI/chapter3_MIDI10.shtml]

I strongly recommend reading this following link, and demonstrate how the hex sequence should be afterwards, as you see it: http://www.omega-art.com/midi/mfiles.html

[quote]2) The following 2 bytes of the status byte “F0” are variable length bytes to tell the reader the length of the data block.
These 2 bytes are missing if I use juce to write such a Midi file.
As we still have the same problem while reading such Midi files, (viewtopic.php?f=2&t=6049&hilit=Joerg)[/quote]

If you’re using the method MidiMessage::createSysExMessage (const uint8* sysexData, const int dataSize), you should not have an issue… :? I truly believe your version of this method (ie: MidiMessage msgToPutInFile(*appManager->createSysExMessage(*MemoryBlock blablabla));) is the culprit… You should share your code for this method if you can - or at least compare its contents to that of MidiMessage::createSysExMessage…

As said, it’s not the reason for the described issue and has nothing to do with it. I wrote my own Midi file reader and writer using C years ago
and have had some issues with that. Finally I decided to only write Type 0 messages for Sysex to get around it.

Again, *appManager->createSysExMessage(*MemoryBlock blablabla) is supposed to do other thing than just adding top and tail byte like the built-in MidiMessage::creatSysexMessage() methode of juce does! Appart from the actual Sysex data (packed in the MemoryBlock blablabla), the SysEx needs to be
completed with header data like manufactorer id, model id and others like device id, checksum etc. and this is what I do within this methode.
No doubt it does it the right way because the synthesizer is reacting to Sysex messages created with this methode.
Also, the status byte in a Sysex message must be followed by the manufacter id and not with variable length bytes!

No systhesizer can read such length bytes, at least at position 1 after “F0” in the message!
In fact, the Midi file writer must inject these length bytes and the reader must remove it!
At the moment juce doesn’t inject while writing and doesn’t expect to find such lenght bytes to remove while reading,
fairly okay as long as only juce based application have to deal with these Midi files.

Before I start updating to latest git it might be a good chance to finally give my two cents
to the issue above.

The below code snipped enables me to write the variable length bytes into a SysEx Midi file:

void MidiFile::writeTrack (OutputStream& mainOut, const int trackNum)
{
    MemoryOutputStream out;
    const MidiMessageSequence& ms = *tracks.getUnchecked (trackNum);

    int lastTick = 0;
    uint8 lastStatusByte = 0;

    for (int i = 0; i < ms.getNumEvents(); ++i)
    {
        const MidiMessage& mm = ms.getEventPointer(i)->message;

        if (! mm.isEndOfTrackMetaEvent())
        {
            const int tick = roundToInt (mm.getTimeStamp());
            const int delta = jmax (0, tick - lastTick);
            MidiFileHelpers::writeVariableLengthInt (out, delta);
            lastTick = tick;

            const uint8* data = mm.getRawData();
            int dataSize = mm.getRawDataSize();

            const uint8 statusByte = data[0];

            if (statusByte == lastStatusByte
                 && (statusByte & 0xf0) != 0xf0
                 && dataSize > 1
                 && i > 0)
            {
                ++data;
                --dataSize;
            }
            // write variable length bytes ===================================================
            else if ((statusByte & 0xf0) == 0xf0)
            {
                out.write (data, 1); //Status byte
                
                MidiFileHelpers::writeVariableLengthInt(out, dataSize - 1);                
                
                out.write (data + 1, dataSize - 1); //SysEx message w/o status byte
                lastStatusByte = statusByte;
                
                continue;
            }
            // ===================================================================

            out.write (data, dataSize);
            lastStatusByte = statusByte;
        }
    }

    {
        out.writeByte (0); // (tick delta)
        const MidiMessage m (MidiMessage::endOfTrack());
        out.write (m.getRawData(), m.getRawDataSize());
    }

    mainOut.writeIntBigEndian ((int) ByteOrder::bigEndianInt ("MTrk"));
    mainOut.writeIntBigEndian ((int) out.getDataSize());
    mainOut << out;
}

…and the following lines of code should go into the Midimessage class to properly read such length bytes:

MidiMessage::MidiMessage (const void* src_, int sz, int& numBytesUsed, const uint8 lastStatusByte, double t)
    : timeStamp (t),
      data (static_cast<uint8*> (preallocatedData.asBytes))
{
    const uint8* src = static_cast <const uint8*> (src_);
	unsigned int byte = (unsigned int) *src;
	int lengthOfDataToAdd = 0;
    
	if (byte < 0x80)
	{
		byte = (unsigned int) (uint8) lastStatusByte;
		numBytesUsed = -1;
	}
	else
	{
		numBytesUsed = 0;
		--sz;
		++src;
	}
    
	if (byte >= 0x80)
    {
        if (byte == 0xf0)
        {
            const uint8* d = src;
            bool haveReadAllLengthBytes = false;
            int numLengthBytesToRemove = 0;
            
            while (d < src + sz)
            {
                if (*d >= 0x80)
                {
                    if (*d == 0xf7)
                    {
                        ++d;  // include the trailing 0xf7 when we hit it
                        break;
                    }
                    
                    if (haveReadAllLengthBytes) // if we see a 0x80 bit set after the initial data length
                        break;                             // bytes, assume it's the end of the sysex
                    
                    ++numLengthBytesToRemove;
                    ++d;
                    continue;
                }
                
                if (! haveReadAllLengthBytes)
                {
                    ++numLengthBytesToRemove;
                    haveReadAllLengthBytes = true;
                }
                
                ++d;
            }
            
            size = 1 + (int) (d - src);
            
            data = new uint8 [size];
            *data = (uint8) byte;
            memcpy (data + 1, src + numLengthBytesToRemove, size - 1);
        }
		else if (byte == 0xff)
        {
            int n;
            const int bytesLeft = readVariableLengthVal (src + 1, n);
            size = jmin (sz + 1, n + 2 + bytesLeft);
            
            data = new uint8 [size];
            *data = (uint8) byte;
            memcpy (data + 1, src, size - 1);
        }
        else
        {
            preallocatedData.asInt32 = 0;
            size = getMessageLengthFromFirstByte ((uint8) byte);
            data[0] = (uint8) byte;
            
            if (size > 1)
            {
                data[1] = src[0];
                
                if (size > 2)
                    data[2] = src[1];
            }
        }
        
		numBytesUsed += size + lengthOfDataToAdd;
	}
	else
	{
		preallocatedData.asInt32 = 0;
		size = 0;
	}
}

Actually it’s already in there except the line:

numBytesUsed += size + lengthOfDataToAdd;

Jules, feel free to do your code changes but please check it in afterwards.

Many thanks,
Joerg

Thanks Joerg, I’ll have a look at this today…

Erm… Are you sure that works? The code where you add numLengthBytesToRemove to the src pointer looks pretty dodgy to me - you’re adding an offset to the pointer, but not subtracting the same offset from the number of bytes you’re copying, so surely it’ll be copying memory from beyond the valid range?

Puh…

in fact it works but haven’t checked for memory issues. To be perfectly honest,
that idea came from you:

http://www.rawmaterialsoftware.com/viewtopic.php?f=2&t=6049&hilit=Jörg

Just scroll down to your last post there.

Thanks,
Joerg

How about this?

[code] if (byte >= 0x80)
{
if (byte == 0xf0)
{
const uint8* d = src;
bool haveReadAllLengthBytes = false;
int numVariableLengthSysexBytes = 0;

        while (d < src + sz)
        {
            if (*d >= 0x80)
            {
                if (*d == 0xf7)
                {
                    ++d;  // include the trailing 0xf7 when we hit it
                    break;
                }

                if (haveReadAllLengthBytes) // if we see a 0x80 bit set after the initial data length
                    break;                  // bytes, assume it's the end of the sysex

                ++numVariableLengthSysexBytes;
                ++d;
                continue;
            }

            if (! haveReadAllLengthBytes)
            {
                haveReadAllLengthBytes = true;
                ++numVariableLengthSysexBytes;
            }

            ++d;
        }

        size = 1 + (int) (d - src);

        data = new uint8 [size - numVariableLengthSysexBytes];
        *data = (uint8) byte;
        memcpy (data + 1, src + numVariableLengthSysexBytes, size - numVariableLengthSysexBytes - 1);
    }

[/code]