Adding large binary data to XML via MemoryBlock not working?

Audio Plugins
1

I am trying to save some session data as binary to speed up loading and saving the data. But when I save the session, none of the binary data appears in the XML, and I don’t see why not. Here is the code:

			//
		pRegionXml->setAttribute( "FileIndex", (String)fileIndex );
		pRegionXml->setAttribute( "FileOffset", (String)frameOffsetInFile );
		pRegionXml->setAttribute( "StartFrame", (String)firstFrameInTimeline );
		int64 count = this->size();
		pRegionXml->setAttribute( "NumFrames", (String)count );
		//
		if (pInputCyclePeriods != nullptr)
		{
			//
			MemoryBlock cyclePeriodData( pInputCyclePeriods, (count*sizeof(double)) );
			pRegionXml->setAttribute( "CyclePeriods", cyclePeriodData.toBase64Encoding() );
		}
		//
		if (pOutputData != nullptr)
		{
			//
			MemoryBlock outputData( pOutputData, (count*sizeof(OutputData)) );
			pRegionXml->setAttribute( "OutputData", outputData.toBase64Encoding() );
		}

The resulting XML in the session includes this for that node:

<RegionData FileIndex="0" FileOffset="0" StartFrame="-402" NumFrames="15247872"/>

So, it looks like we’re trying to add about 15 million pieces of data there, each of which is 4 floats, or 16 bytes, for a total of 243Mb. Is that simply too large to save as a String?

Should I bypass the XML and simply memcpy() the data into the chunk itself?

Or, do I really need to write the data to an external file instead, and deal with the issues of users taking their sessions to other machines, and not knowing they have to copy the data file(s) as well?

1 Like
2

Do you have an idea where the code fails?

Generally speaking, for this kind of sizes I would go for reading and writing binary anyway. JUCE has wonderful ways of doing this using methods of the OutputStream & InputStream classes. You then could write the XML before or after the binary data as string into these streams and store it in a binary file.

For more sophisticated serialising (with backwards compatibility for example) you could look into ProtocolBuffers.

3

Ah, crap. It’s not writing because those pointers are both NULL. They’re not getting set for some reason now, even though the were getting set prior to my adding the save/load code. Sigh.

4

Ok, it works now. I still might want to store the data externally. That was only 6 minutes of recording audio information. If a user records significantly more data than that, we might still have a problem.

1 Like