Hey there, I’m working on a sampled instrument and my client would really like the samples to be self contained within the plugin if possible (so far it’s just loading the samples from the hard disk when needed). The problem is, all the samples together are potentially over a gigabyte large. I don’t know if this is at all practical however. I’m having trouble even building the project when this much data is included as a binary resource. Obviously, I wouldn’t want customers loading this much data into ram every time they load the plugin either, however according to this this the operating system doesn’t necessarily load the entire binary into ram, and only loads resources when the program accesses it - is this the case for plugins as well?
I presume at least on mac I can pack all of the data inside the “component”, but on windows, are there any options to make the plugin appear more self contained assuming it’s not feasible to add the data as a binary resource directly?
Well, you can try it… it may work, although the compiler will probably explode at some point! Typically anyone working on that kind of scale will have built a more serious sample management system, to minimise disk streaming and keep startup time low, but who knows, maybe the OS’s virtual memory system would do a reasonable job of it!
Well so far I’ve first tried just loaded all the samples in directly. Building in visual studio would take forever, and eventually error when trying to open the huge library it produces. After searching online I read that I need to build projects like that with the special 64 bit tools using vcvarsall.bat, which I did - and it did succeed in building the plugin, but the resulting plugin was small, I guess the compiler/linker knew not to include the resources as I wasn’t explicitly referencing them.
So, since I didn’t want to write a line of code referencing every single sample (of which there are hundreds), I packed them up into zips and added them as resources instead, and referenced each zip via a switch statement. Some of these zips are very large. When compiling, even with vcvarsall, my computer grinds to a complete halt, but eventually I get a bunch of “compiler is out of heap space” errors, and the project fails to properly build. So I’m at a loss now really.
On Windows at least (not sure about other platforms) you can append data to an executable without issues. So, in your post build step, take you samples .zip file and do something like this: ‘copy /b plugin.dll + samples.zip plugin.dll’
Since the header of a zip file is at the end, you can load your executable as a .zip file and it should just work.
I wrote a self extracting archive creator a while back and this is how it worked. Not 100% sure it will work with a .dll, I only ever tried with .exe. Also not sure if juce .zip file class will ignore the header. It’s worth a try.
Not sure how it will work since you have many zip files, maybe a zip of zips.
Wow you may be onto something. I just did that with a 600mb zip and was able to load the plugin dll in my DAW and didn’t see a jump in memory usage. Do you know if it’s possible to do this on macos (or something similar) as well? That would be very important.
What does the data at buffer + 46 look like? Does fileNameLen look valid? If you use non ascii characters in your filenames, some windows apps won’t encode them as utf8
For macOS you don’t need to worry about all this. Just copy zip into bundle.
None of the files in the zip have names like this of course.
edit: I’m guessing it’s trying to read some chunk of the dll wholly unconnected to the zip I appended and getting garbage like that, incidentally I’m unable to open the dll as a zip with e.g. 7zip.
If it’s not a must, and to be honest it sounds like it shouldn’t be. I’d create some encapsulated format that appends the audio file. A simple archive format would be fine or any arbitrary format.
If it’s purely security concerns this data file can even be encrypted.
So you’ll end up with app and data files. You can provide a simple tool for modifying your big resource file.
