Read multidimensional arrays from binary resources

General JUCE discussion
1

My audio app is based multidimensional arrays.
These arrays are stored in binary files (about 10 files of 50 Mo each).

In C++, I found a way to read the file like this:

#include <fstream>
float array[dim1][dim2][dim3];
std::ifstream myfile;

// Read array File
myfile.open(dataFilePath, std::ios::binary);
if (myfile.is_open())
{
    myfile.read(reinterpret_cast<char *>(array), sizeof array); 
    myfile.close();
}

Now, I would like to embed these arrays as binary resources in the app, with the Projucer. But I don’t know how to read them. (I try to use BinaryData:: … )
I tried to use the File and the FileInputStream classes, but I didn’t find how to do as they are based on absolute paths.

Do I have to get a File or a FileInputStream from the binary resources ?
Or maybe il is not a good approach (as they are big files), and it is better to read the files from an absolute path (that I have to find in MacOs).

Thanks a lot for your answer!

2

Do you mean “of 50MB each”? If so, embedding them as BinaryData is no good idea. What the Projucer does when encoding your files into binary data is reading it byte by byte and creating a huge char array from that containing each byte as integer constant. Just have a look at the BinaryData.cpp file from a current project where a few SVG files are embedded as binary data:

image
image1824Ă—1176 670 KB

You can probably imagine how huge your BinaryData file would become if you would try to embed 500MB of data this way and how slow compilation would become. For that reason, the Projucer even limits the max size of each file to be encoded into BinaryData to 20MB. This should definitively be read from file. If I got you wrong and you didn’t mean 50MB then you basically have two options:

First would be to reinterpret_cast your BinaryData to a float pointer that can be accessed like a one-dimensional array. You would then have to compute the offsets in that one dimensional array depending on the three dimensions. Maybe with some helper struct like

template <int dim1, int dim2, int dim3>
struct ThreeDimBinaryData
{
    ThreeDimBinaryData (const char* binaryData) : values (reinterpret_cast<const float*> (binaryData) {};

    const float* values;

    int getIdxAt (int x, int y, int z)
    {
        return x + y * dim1 + z * dim1 * dim2;
    }

    float getValueAt (int x, int y, int z)
    {
        return values[getIdxAt (x, y, z)];
    }
};

(Untested, better check the index calculation, might be wrong, but I guess you get what I mean). Usage would be like

ThreeDimBinaryData<dim1, dim2, dim3> data (BinaryData::something_extension);

float foo = data.getValueAt (1, 2, 3);

However, if you want to stick to the 3D array, you can of course do the same as reading from file through a stupid std::memcpy

float array[dim1][dim2][dim3];
std::memcpy (array, BinaryData::something_extension, sizeof (array));

However that could be quite inefficient, as you are duplicating content that already lives in memory.

If you want to go the file route, you should also check the JUCE File and FileInputStream classes which give you a lot better file handling options for real-world applications.

1 Like
3

In case of such huge files which need to be accessed rather fast, it’s often recommended to use memory mapped files (MemoryMappedFile class). Many DAWs or other software operating on huge data volumes work this way.

1 Like
4

Thanks. They are indeed big files (600 MB in total).