LTO Step Taking 15 Minutes

cry for help

Recently seemingly inexplicably my release builds have started consuming a huge and unreasonable amount of time performing LTO.

I’ve looked online and haven’t found any resources on potential ways to speed up LTO. It’s possible I could slim includes or remove unused sources – does anyone else have recommendation?

I’ve set up unity builds and all sources get compiled in under a couple minutes – but it still hangs for ages during linking – it makes optimizing release builds almost impossible.

If anyone has a tip on this I will most definitely buy you multiple beers!

Get more RAM. If your machine starts using swap file, times go up like crazy. But if you are already using unity builds, there isn’t that much point to LTO.

1 Like

I have 64gb :smiling_face_with_tear:

Looks like LTO will have to go off for now if I can’t solve it.

It’s a catch 22 cause I think unity build will slow down incremental builds — it’s kinda amazing how little insight we get into the LTO process — it just looks like it freezes.

I was looking at thin LTO as well but what’s the point when we want to ship the real deal

Swap files are news to me maybe there’s a way to check that

We also experienced great build time increases with LTO, so we only enable it when building actual release candidates. The performance gain is rather little compared to the build time increase. If you develop and optimise your code without LTO, it can only become faster after enabling it.

From a more technical point of view, link time optimisation optimises calls between different TUs. If you write your performance critical dsp code in a way that all of it is compiled in the same TU (e.g. by putting all your processing into a juce::dsp::ProcessorChain template and writing the dsp code header only), you can move the optimisation to the compile stage which is not only a lot faster but also allows for even better optimisation to my knowledge.

I guess you mean catch unit tests? How do you integrate them? We build them as a separate target which makes it easy to build them with other options than the plugin target

LTO times are a bit misleading, which you will easily see if you turn it off and build a release build:
When LTO happens the compile time are really fast, because it’s actually the linker that triggers a bunch of compiler steps for optimization.
Without LTO, the optimizer knows to run those multiple steps during the compilation period.

So even though LTO does add quite a bit of time, I think if you’ll measure the difference you’ll see that not all of those 15 minutes were actually due to LTO itself.

Anyway I definitely think that you probably have a fundamental problem with how your builds are structured if they take that long. My plugins that bring in a ton of libraries, binary data, etc are about 2 minutes for a clean release build.

Turning off some compiler optimizations like LTO might reduce it, but also harm the performance of the product, so I wouldn’t go there…

Also, regarding unity builds: if you correctly structure your code, incremental builds are still really fast. I have about 30 custom JUCE-style modules and incremental builds that involve only 1-2 modules changing are about 3-5 seconds on a regular M1 (not max or ultra) computer in debug builds.

It is important to generally keep as much as you can out of header files that are included elsewhere for faster incremental builds, and that’s true for both unity and non-unity builds.

stuff like keeping external libs or even big JUCE modules as Pimpl/forward declarations so they’re only included in a cpp file really helps.

For example there’s no need for your entire code to include by proxy juce_opengl or juce_cryptography, etc.

Phew sorry got knocked out sick there –

Yeah I’m not sure actually I’ve got to look into it more – you were right @eyalamir – unity builds don’t really have any big impact on incremental build times. I’m doing unity builds through CMake with batch sizes of 50 – the file takes something like 6-8 seconds to compile so that pretty much my incremental build time.

The LTO thing is still wildly different than you’re describing – For now during development I’ve got LTO off – using LTO off + Unity is pretty close in perf to non unity + LTO – so my current strategy is to only turn LTO on for actual release candidates like @PluginPenguin mentioned.

Although @eyalamir – we’re starting to experiment with Ninja + CLion which I know is your fav so maybe that will somehow improve it?

I think when I finish I’ll make a final school report on how we resolved the builds cause it’s a massive project and we’ve just scaled out of a lot of tooling along the way.

“Catch-22” is a novel that gave its name to a concept. See Catch-22 (logic) - Wikipedia

2 Likes

Ah thanks @mcmartin – I was completely unaware of this Term :smile:

Ninja definitely makes things way faster on Windows for me, the difference on Mac is very small, not sure if you’ll see improvements there.

Anyway I think the main point is that ‘faster builds’ is a process of really looking at your code and minimizing dependencies: the less code you include in each translation unit, less include directories, more implementation in .cpp vs .h, all will gradually make your code compile faster, and will be way more effective than any build flag magic.