Code organization vs. build granularity

CMake
1

Hi everyone! This is not specifically JUCE related, but I’d like to hear how you dealt with code organization vs. build granularity.

I currently use a single repository for all my plugin projects. With CMake and bash scripts I can build any plugin by specifying its name, format, configuration, and so on.

Different plugins share some common DSP classes that are currently built as a monolithic static library. The size of the DSP library is now grown to a point where I’d like to be more selective of what’s built when building a plugin.

I thought of a couple solutions to achieve this, but I can’t lend on any that I’d prefer yet:

  1. Split the monolithic DSP library in smaller “thematic” submodules (eg. core, filters, delays, modulations). This seems to be a common solution to the problem, and it might work for core or base type libraries, but not for more complex groups. I’ll rarely need to build 5 different delays for a single plugin. What other “themes” might be a better approach?

  2. Each plugin builds its own DSP static library. Only the needed files are built at the cost of no modular encapsulation, no flexibility and the consumer needing to know everything about the library structure. Ok for a small-size operation like mine, but still annoying to deal with at times.

  3. Keep the monolithic static library and find something to do while building. Also thinking about IP protection: this isn’t optimal, is it? The linker should only include object files from static libraries that resolve symbols my plugin actually uses, but maybe some sneaky compiler optimization setting might inline something and unused symbols might end up in the final build? This way each plugin binary will include all my DSP. Not that there is anything worth spending time reverse engineering, but still.

How did you deal with this?

Thanks!

2

Stuff that is used in more than one product goes in a common library, and each plugin also has a library that defines how the building blocks are stuck together. Occasionally the shared algorithms may need to be tweaked a bit for a product as well, in which case the tweaked version should go in that product’s library, not the common library.

I’ll rarely need to build 5 different delays for a single plugin

Right, you probably shouldn’t. Why do you have 5 different delays in the first place? Are they all even being used? If some of them are used only in a single plugin, move it into that plugin’s product-specific DSP library.

Only the needed files are built at the cost of no modular encapsulation, no flexibility and the consumer needing to know everything about the library structure

Definitely don’t do this. Organize your code into logical library targets using CMake, this makes sharing things between projects and maintaining the build system much more sane than the alternatives.

Keep the monolithic static library and find something to do while building

There are many strategies to try and reduce build times. You’re right that breaking up a library into several smaller ones could be one way to allow greater build parallelism, here are some other ideas:

  • Make sure you’re using CMake’s Ninja generator, and set CMAKE_BUILD_PARALLEL_LEVEL to something like 8 or 12 depending on your machine. This can be set from a CMake preset or from a direnv file.
  • Move non-templated function definitions out of header files and into source files, this way other TUs including the headers have less to parse
  • Break up large TUs into smaller ones
  • Review your code to make sure that no unnecessary headers are included. You definitely don’t want to be including headers you don’t need to! IWYU is the tool for this, and it has built-in CMake support similar to clang-tidy.
  • Introduce precompiled headers. Can be a bit complex but can sometimes work.
  • Use a faster linker like mold or gold

CMake and Clang also have tooling support for profiling build time. IIRC there’s a Clang flag that’s something like -ftime-trace or something. You can see which TUs take the most time and consider breaking those up.

IP protection

I think these concerns are mostly unfounded. For your release binaries, you should be stripping them, which means that only symbols that are actually used will be present in the final binary.

A minor thing, but

With CMake and bash scripts

If you’re using CMake, why do you need bash scripts? I definitely recommend CMake presets, and if you want a kind of “task” runner then I recommend Just.

3

Stuff that is used in more than one product goes in a common library, and each plugin also has a library that defines how the building blocks are stuck together.

Yes my case is similar. My issue is with keeping the common library as a single monolith vs. splitting it in a way that makes sense. I’m trying to find the best tradeoff between not going too crazy with a super granular approach but also not building everything when most of the library might not be used in the plugin.

Why do you have 5 different delays in the first place?

Yeah maybe delays are not a good example, but anything that might get too messy templating or with too many parameters. My reasoning was mostly on this thematic approach that I often see. It’s tidy to have the library divided per effect type (filters, delays, modulations and such) but in practice those are rarely needed at the same time.

If some of them are used only in a single plugin, move it into that plugin’s product-specific DSP library.

Yes I’m only speaking of shared code here. The product-specific DSP is not included in the common DSP library.

Definitely don’t do this. Organize your code into logical library targets using CMake

That’s the culprit. I’m interested in hearing what other developers consider logical targets.

here are some other ideas:

Thanks! I’m already doing most of these, but a couple are new to me. I’ll check them out!

For your release binaries, you should be stripping them, which means that only symbols that are actually used will be present in the final binary.

I’m stripping my binaries. Not sure about what I mentioned on inlining and aggressive optimization. I can see the compiler inlining something sneakily that ends up in the final binary. I know it’s a non-issue really, but still.

If you’re using CMake, why do you need bash scripts?

Not a real reason tbh. I set up the script to do all the building, packaging, signing and notarization and it works. Since I only have little time to work on this I’d rather only refactor my build system when needed :nerd_face:

1 Like
4

Dead code stripping (but also lto) could also help to some extent in term of final binary sizes when including bigger libraries if that is one of your concerns.

1 Like
5

It is a tricky balance to find. Sometimes you just need to try doing some reorganizing and see if it works for the project.

To be clear though, building a plugin target won’t necessarily recompile all your common libraries – only the ones that have changed since the last time they were built. So if you’ve got a library of 5 delays that never change, you only need to build it once, and then every subsequent plugin build doesn’t pay any build-time cost for that library, they just link to it.

With this in mind, one approach might be to separate “volatile” code from “stable” code. If you’ve got some modules/classes that rarely/never change, put them in a different library than code that’s frequently evolving so that the stable code can just be compiled once and reused.

6

building a plugin target won’t necessarily recompile all your common libraries

True. Since I was having the monolithic approach it was easy to change something and needing to rebuild the whole thing. Just splitting it should help for this reason.

With this in mind, one approach might be to separate “volatile” code from “stable” code.

That seems to be a good logical split. Most of the common code only changes if I need to solve bugs, otherwise it stays as it is. Making it slightly more granular should help to begin with, so it might just be easier to start splitting it and see what makes sense.

Thank you very much for the hints!

1 Like