Infinite loop calling `juce_add_module` from a CMakeLists.txt in the module's own directory

[sudara]

I keep running into this problem with modules when I set them up as their own CMake projects (for example to run their tests in CI).

Setting up a CMakeLists.txt in the module root and calling something like juce_add_module("${CMAKE_CURRENT_SOURCE_DIR}") or juce_add_module(".") or some other variety of that configures fine, but causes an infinite build loop (macOS CLion with the default Ninja generator)

[0/2] Re-checking globbed directories...
-- GLOB mismatch!
[1/2] Re-running CMake...
-- Cloning JUCE...
-- Configuring juceaide
-- Building juceaide
-- Exporting juceaide
-- Configuring done (1.5s)
-- Generating done (0.0s)
-- Build files have been written to: modules/melatonin_blur/cmake-build-debug
[0/2] Re-checking globbed directories..
(this repeats forever)

I had this problem on the inspector, and the only “solution” we could come up with was to move the actual .h file with the module header into a subfolder.

We did this on the perfetto module and but I prefer / thought it’s best practice when published modules have their module header in the root (vs. buried in subfolders to be dug out) — but perhaps that’s the pragmatic solution.

Anyway, this behavior only shows up on CLion — just posting it here in case it’s something obvious on the JUCE side or someone else has a work around.

Maybe I’ll go report it to JetBrains, because if I switch to the Unix Makefiles generator or just run cmake -B Builds && cmake --build Builds in Terminal, I don’t get the infinite loop.

[sudara]

Just as another data point, it looks like the Ninja generator on Windows also goes into the infinite loop — but MSVC is fine. For reproduction, the blur module exhibits this behavior.

[eyalamir]

I’m not sure exactly what the problem is, but here’s how I set up JUCE modules from different repos:

Module repo:

Modules
    my_module
        my_module.h
        my_module.cpp
    CMakeLists.txt

CMakeLists.txt

Repo root CMakeLists.txt:

add_subdirectory(Modules)

Modules subfolder CMakeLists.txt:

juce_add_modules(my_module)

Some client CMakeLists.txt:

add_subdirectory(Modules)
target_link_libraries(MyPlugin PRIVATE my_module)

So there’s never any relative directory mentioning and the ‘client’ doesn’t worry about setting up the module.

[sudara]

Thanks Eyal!

I think the issue is juce_add_module takes a path — so if you pass it a directory name relative to the current CMakeLists.txt, it’s happy.

Passing it . or CMAKE_CURRENT_SOURCE_DIR causes this infinite loop problem with Ninja (on build, not configure).

Calling it juce_add_module(my_module) from within the my_module directory (like in your example) doesn’t work for me — CMake errors with Could not locate module header for module because it’s looking for my_module/my_module/my_module.h — but this would be the route I’d prefer!

[eyalamir]

@sudara

CMake errors with Could not locate module header for module because it’s looking for my_module/my_module/my_module.h

Why do you need that extra indirection?
JUCE modules have specific format, they need to look like:

ModulesFolder
    my_module
        OtherCode
            module_internalcode.h
            module_internalcode.cpp
        my_module.h
        my_module.cpp
    CMakeLists.txt

Then in code the include would look like

#include <my_module/my_module.h>

You don’t need an additional subfolder named ‘my_module’ and have yet another 'my module` subdirectory there.

You can look at the format of the built in JUCE modules and how they’re structured as a good reference.

[sudara]

@eyalamir

You don’t need an additional subfolder named ‘my_module’ and have yet another 'my module` subdirectory there.

I think we misunderstood each other! We agree on how modules can and should work. I definitely do not want another subdirectory, that’s what I’m looking to avoid

The problem is calling juce_add_module from its own CMakeLists.txt. Not from a directory above it like modules or Modules. I misread your example in my last post, sorry about that.

In the case where the module header is in the same directory as the module’s CMakeLists.txt, juce_add_module needs to be called with . or CMAKE_CURRENT_SOURCE_DIR (since it takes a directory). Doing so causes an infinite loop (only with Ninja).

Why would I want this? My main use case is a module repo running its own tests in CI, or working on the module in CLion. Basically it’s an issue only when the module’s CMakeLists.txt is the top level CMakeLists.txt.

(Full context is I chatted with @reuk at ADC about the issue, was just providing more info).

[eyalamir]

@sudara

Why would I want this? My main use case is a module repo running its own tests in CI, or working on the module in CLion. Basically it’s an issue only when the module’s CMakeLists.txt is the top level CMakeLists.txt.

Right, so don’t do that.

Move the module ‘down’ one sub folder like I initially suggested. So the root of the repo will contain a bunch of flags for CI that can be enabled on demand for devs of the submodule, but turned off by default.

The simplest way to do this is:

Repo
    my_module1
    my_module2
CMakeLists.txt

And the root CMakeLists calls

option (StandaloneModuleBuild "" OFF)

if (StandaloneModuleBuild)
    #this may setup juce if its missing, etc
    setupCI()
endif()

#In non-standalone mode, we're counting on whoever adds 'us' to setup juce by this point:
juce_add_modules(my_module1 my_module2)

Client code:

add_subdirectory(Modules)
target_link_libraries (MyPlugin PRIVATE my_module1 my_module2)

We have many modules deployed like that in different repos and used by many clients, and of course each repo may have its own unit tests and specific scripts.

[sudara]

Fair enough!

The workflow I prefer is 1 module per repo. I prefer to consume modules that way — usually only interested in 1 chunk of code when people do publish their multi-module libraries. But I also have a bunch of single module repos in use by quite a few people.

I started out with “nest the module folder inside the repo” as you describe. It’s not quite as nice / clear as keeping things in the root (for code browsing, Projucer folks), but is fine!

Module-in-the-root is also totally fine as a module consumer. There’s just a CLion+Ninja bug when at top level.

[eyalamir]

@sudara

The workflow I prefer is 1 module per repo. I prefer to consume modules that way — usually only interested in 1 chunk of code when people do publish their multi-module libraries. But I also have a bunch of single module repos in use by quite a few people.

Sure, I think my suggestion isn’t related to how many modules you want to include per repo, just the structure of how to publish them from the module repo’s point of view.

@sudara

Module-in-the-root is also totally fine as a module consumer. There’s just a CLion+Ninja bug when at top level.

The consumer doesn’t care where the module is, as long as your repo code 'publishes it`.

For example I might move around modules, from Repo/ea_audio_utils to Repo/Modules/Audio/ea_audio_utils.

The consumer only links against ea_audio_utils and they don’t care where it came from in terms of the subdirectory, I can move it somewhere else and everyone’s code will compile.

@sudara

I started out with “nest the module folder inside the repo” as you describe. It’s not quite as nice / clear as keeping things in the root (for code browsing, Projucer folks), but is fine!

I think that part depends on the IDE.
My directory structure looks like:

Workspace
   JUCE
   Repo1
       module1
       module2
    Repo2
        module3
        module4

I have 0 problems of browsing those, not sure why having a flat list of modules would be cleaner in that case, I also like to have my modules in subdirectories with categories and things that help navigate them.

Those become important when you have ~50 modules.

[sudara]

Yeah, for sure everyone has their own way of organizing this stuff! Thanks for sharing.

I’m interested in this as a topic, so I appreciate the back and forth. I have a “unofficial guide to modules” blog post that I’ve been holding back, in part because people seem to consume and produce them quite differently — I don’t want to speak for anyone else.

But I’m also motivated because I provide support for my open source modules on an almost daily basis, so I regularly see what trips people up. That’s why I’m incentivized to keep things as simple as possible.

For example, some people will use add_subdirectory (which as you say, gives you control to then move the source around etc), but others will be on Projucer or will add the module from their CMakeLists.txt with juce_add_module(‘modules/my_module’). Others will nest the module many folders deep. So I’ve just found everything is kept a bit more simple when the module is the root (which means less DMs for me!) and all the naming is consistent.

This is how I organize: Every folder in modules is a git repo. For me, it’s nicer when the code is right in those module folders vs. my_module/my_module/my_module.h (or slightly_different_repo_name/my_module/my_module.h)

[eyalamir]

@sudara

but others will be on Projucer or will add the module from their CMakeLists.txt with juce_add_module(‘modules/my_module’)

I haven’t used the Projucer in a couple of years, so can’t comment on that flow, but the second flow you mention with the user adding the module themselves should be a flow you’re not supporting - similar to JUCE not recommending doing juce_add_module(JUCE/modules/juce_core).

The way to ‘not support this’ or ‘reduce DMs’ is to have a CMakeLists that publishes the modules and nothing else. If your ‘root’ CMakeLists will add a ton of noise to the clients like unit tests, they will not want to use your CMakeLists and will want to bypass it instead.

You want to suggest a flow where the user either uses add_subdirectory(Module) or CPMAddPackage("gh:module/main") or fetch_content or any other flow that will auto-publish everything your module exports, without letting the user poke in there.

I’m sure a simple script could be written for Projucer as well, kinda like ‘collect modules’ that adds them to the Projucer ‘user’ folders - but again, it’s been a while since I’ve used that so no idea.

[sudara]

If your ‘root’ CMakeLists will add a ton of noise to the clients like unit tests

This isn’t a problem, as you can ask CMake if my_module_IS_TOP_LEVEL.

Yeah, I’ve gone back and forth on juce_add_module for the reasons you describe — the problem is that people will attempt to use it anyway and then wonder why it doesn’t work. add_subdirectory is also a less “known” path, especially by people starting out (and there are many published modules with no CMakeLists.txt at all, including some first ones I made) But it might still be worth it (also easy to document, this stuff isn’t rocket science!).

[sudara]

In general a lot of module related things would be easier if Projucer support wouldn’t be needed. It’s currently hard to have a published module “do much” with CMake (additional dependencies, assets, etc) without making it tough to install/use for people on Projucer (and if it’s Projucer friendly, then it’s juce_add_module friendly too)

[reuk]

@sudara

-- Build files have been written to: modules/melatonin_blur/cmake-build-debug

This was raised on Discord, but I’ll mention it here too - generating build files into the module folder might cause problems.

CMake has to glob the contents of the module folder due to the way JUCE modules are specified (this spec predates CMake support). We also use CONFIGURE_DEPENDS to force a reconfiguration if any module files are added or removed. This is intended to automatically update the build if a new .cpp is added at the top level of a module. When build files are written into the module directory, this will force a reconfiguration at the beginning of every build, which could be slow. I’m not sure whether the infinite-looping behaviour you’re seeing is related, but I wouldn’t be surprised.

If you’re still seeing this problem, please could you try relocating your build directory outside of the module?

[sudara]

Thanks @ruek,

Yes, this is the same issue as Andross ran into on Discord.

Building out-of-source resolves, as does avoiding Ninja (the CONFIGURE_DEPENDS does trigger, but it’s quick and doesn’t go into the endless loop).

It’s funny, I’ve always marched around downplaying the warnings around GLOB_RECURSE, but now I can say I’ve finally been bitten by it!

I thought it might work to replace the GLOB_RECURSE in JUCEModuleSupport with a GLOB of just the top level directories in the module, then exclude common build directories (like cmake-build*|Builds|builds|xcode|\\.vs|\\.idea|\\.git before doing a GLOB_RECURSE on the remaining module subfolders, but I didn’t quite get it happy.

[reuk]

I think we’d need to keep the glob_recurse anyway; we use it to collect up all the other files in the module so that we can tag them with HEADER_FILE_ONLY when source groups are enabled. Building out-of-source seems like the easiest solution for now.

[sudara]

Gotcha! Yeah, I was thinking GLOB_RECURSE would still run, just not on explicitly excluded common build directories.

For now, I’ll just let people know if they want to work with the modules in CLion, etc, they’ll need to do it out-of-source or with a non-Ninja generator.

Thanks!