VisualStudio: Compilation freezes with very big array initialization

So I’m storing all my wavetabledata in a single float array, which is of size

float m_wavetables[70][40][512];
//70*40*521 = 1,433,600

Now I’m trying to set these values via the initializer list, where I include a file at the right location containing all the data in initializer list format.

On linux, everything works fine. On Windows this takes a huge toll on the compiler. First I had to upgrade to the 64Bit toolset, since the compiler gave me an error it ran out of heap space.
After upgrading, the compilation just gets stuck at some point (before that my entire system is laggy as hell and unusable). It still uses around 13 of 16Gb available RAM. VS doesn’t freeze, but there’s no more output or progress.

What could be the problem here? I really don’t wanna break up this construct, or else I have to redesign my oscillators.

Regardless of the struggle you have with the compiler…

This seems to me like you are creating this thing on the stack… Just doing some math, you have 70 * 40 * 521 * 8 Byte = 11670400 Byte = 11.6704 MB.

According to this document, the default stack size for a thread in Windows is 1 MB, so even if it will ever compile, this will immediately lead to a stack overflow when executing your code. I think a heap-allocation-based approach where the data is loaded from a file at runtime could be the better approach here.

1 Like

But I can easily compile and run the programm by setting the entire thing to zero:

float m_wavetables[70][40][512] = {0};

and then not include the initializer list… Or is there some magic behind the scenes happening here?

EDIT: definetly not a problem! I’m initializing it to zero and filling it manually via fourier tables later. works like a charm :slight_smile:

Okay, maybe the compiler is smart enough to move the whole thing to the heap? To be honest I don’t know, however I was really facing a stack overflow on Windows a few weeks ago due to a std::array that accidentally had become much too big and in that case the compiler was not smart enough to find a workaround.

However, with around 11MB of data I’d still suggest loading it from a file at runtime instead of putting all values into an initializer list of that size (which you probably created automatically with some script in Matlab, Python or something like that?) to keep compilation time and binary size down.

If the array is declared as a global or static variable, it will be allocated automatically by the compiler in some suitable memory space. Stack size is only an issue with (automatic) local variables within functions/methods.

1 Like

Also if the array is a member variable of an object on the heap it is not a problem.

But anyway, in what scope is that variable defined? If this data is known at compile time and doesn’t change at run time then I would try to declare this as static const float wavetables[70][40][512] = {.....}; at global scope. Or if the memory usage doesn’t go away you can try a float[1433600].

Which version of Visual Studio are you using? The compiler improved quite a lot between 2015 and 2019.

MSVC is sometimes not good at dealing with this kind of table initialization, but it should be possible to make it simpler for the compiler.

float m_wavetables[70][40][512]; looks like a member variable in a class. It is neither const, nor static, and I guess it is defined in a header file. This forces the compiler to do a lot of work, since m_wavetables is not “simple data”. It’s not a huge table of constants that the program can easily load into read-only memory, it’s a huge table of potentially changing variables that might need to be constructed several times.

To reduce compile time (and also execution time), you should make this table static, constant, and global, in a single source file.

You can read up more here: https://aras-p.info/blog/2017/10/24/Slow-to-Compile-Table-Initializers/.

1 Like

Thanks for the tip, I updated 17->19, same problem only that VS now actually freezes :smiley:

Ok could you shed some light on how to do this? I mean if it is in a source file how do I make my class see the data?

Thanks for all your input guys!

Declare a getter function like float*** getMyTableData() in a header file…?

1 Like

Have you considered adding the wavetable data in your binary-resources and simply use that? We have more than 1.4 MB of data in ours and it compiles really fast. It’s pretty much exactly what you’re doing, but it works without a hitch.

Ok, but the data is now const float*[][][]. How would I in my class which actually uses/distributes the data import it? Since it is const I can’t simply load it in the constructor or something?

EDIT: nevermind… the floats are const not the pointers I mixed that up :smiley:

That seems like a good final solution. But right now I’m still in the process of creating the wavetables and they change all the time