Organizing 3rd Party Modules

General JUCE discussion
1

I can totally understand that. This is why I separate my demo application from the libraries. Note that VFLib doesn’t come with JUCE:

It does follow the same directory structure as JUCE. You add the root directory to your include paths, and then you can do:

#include "modules/vf_core/vf_core.h"

If JUCE is also in your path then yes you will have two directories called “modules”. One in JUCE and one in VFLib. This is fine, since the child directory names don’t conflict (unless someone does something silly like call their module juce_core). Prefixing your modules would of course help: “modules/drow_core”, “modules/drow_audio”, etc…

When I create a demonstration application, I put it into a separate repository and use git-subtree to add the library repository directly to the demo repo. I also git-subtree the JUCEAmalgam sources into the demo repo. AppletJUCE has both JUCEAmalgam and VFLib as subtrees:

https://github.com/vinniefalco/AppletJUCE

A git-subtree is independent of the original repo. The files are added to the destination repository and become part of it. This is different from submodules, and easier to maintain. Since they are regular files, the repository can be cloned without hassle.

LuaBridgeDemo brings in JUCEAmalgam (in Demo/) and LuaBridge as subtrees:

When VFLib changes, bringing in the commits to the AppletJUCE project is as easy as:

git subtree pull -P VFLib vflib master --squash

You could even git-subtree the JUCE repository into your repo like this:

git remote add juce git://github.com/julianstorer/JUCE.git
git subtree add -P JUCE juce master

There is the objection that it now becomes necessary to update JUCE frequently in all the repositories that use it, instead of keeping just one copy of JUCE on the local developer’s machine and sharing the code. I will argue that this is not a problem, in fact it is superior. JUCE usage falls into two cases:

  1. Demo application code
  2. Production code

For #1, it is not important that JUCE always be kept up to date. As long as the version of JUCE used is one that supports all the features of the demo (and without bugs), a repository can happily stay at an older version of JUCE. Once in a while, JUCE will make a breaking change and the underlying library (drowAudio in this case) might need to change. Then it would be necessary to also pull the latest JUCE tip into the demo project. As these are infrequent events, the benefit of having a self contained and independent repository outweighs the cost. Note that DSP Filters uses a JUCE that is months old. And yet the demo still compiles, works, and does everything it needs to do.

For #2, it will be necessary to make more frequent updates to JUCE in order to pick up bug fixes, or important features in the critical path of development. Even in this case, adding JUCE as a subtree to your existing repository (which is presumably private since it is commercial software) makes more sense. One obvious benefit of subtree-ing in the JUCE repo is that you can make changes, branch it, tag it, and have the development timeline synchronized to your application’s source tree.

For example, if you create a beta release candidate you can tag the version, and the exact snapshot of the JUCE sources used to build the executable will become part of the tag. This is not possible when using the traditional IntroJucer workflow (where the JUCE sources are not under revision control, and downloaded automatically by IntroJucer).

NO professional software company could ever put out reliable products if they do not freeze the JUCE library code during the testing stage, or else product that ships would be different from the one that was tested! What happens if JULES makes a ton of unrelated changes to JUCE and you need to cherry pick just one or two commits? It is not possible when the sources are shared by multiple projects on the same machine. And then if you need to make changes to JUCE they aren’t revision controlled, etc…

Adding JUCE as a subtree, amalgamated or otherwise, has a lot of benefits and little disadvantage.

2

All good points and a very elegant solution to bundling JUCE/other libs within git source trees.

One thing I do definitely want to stick with however is the JUCE module system. I noticed that none of the vflib modules contain a juce_module_info file. This has a number of implications:

  1. The user will have to manually add your AppConfigTemplate settings into their AppConfig.h file, they have no control of these options from the Introjucer. This also means that the modules can’t be copied or browsed as per JUCE modules. What if you update your module? The copied config flags wont reflect these new changes.
  2. When building applications using vflib and the Introjucer users have to manually add the .cpp/.c/.mm files of the modules to the source tree
  3. Users also have to add the modules folders to the “Header Search Paths” setting so the modules can be found and vflib modules can find JUCE
  4. You have to manually create a StandardIncludes.h file for including your module header files
  5. What do you do if your module requires other system libraries or frameworks? The user would have to add yet more flags to the Introjucer through hidden (i.e. not visible through the Introjucer) directions.

Now, none of these things are necessarily wrong, indeed some are good practices (e.g. having a StandardIncludes.h file) but it does sort of split and obfuscate the module paradigm. What this approach does do extremely well is make it easy fo users to pull the repo and build existing projects.

If someone wanted to build an application from scratch using your modules they would have to jump through far more hoops and really understand all the intricacies of the Introjucer and where everything gets included and what the search paths are etc.

Now take the stk, juced or dRowAudio modules. Because they are designed exactly like JUCE modules they work in exactly the same way. Simply clone the module, drop it into your juce/modules directory, create a new Introjucer project, tick the modules, set any configs if necessary and away you go. No additional steps required[1].

Now what would be nice is something in the middle. I would like a few options added to the Introjucer and the module_info files:

  1. An Introjucer option to specify additional module search locations
  2. Have the Introjucer automatically include the module directories in the search path. Then we can legally do something like: #include “juce_core/native/juce_BasicNativeHeaders.h” no matter where our modules are relative to the main JUCE modules directory.
  3. Let users (or even better module_info files) specify additional namespaces to add to the JuceHeader file.

What I’m trying to achieve here is complete transparency between official JUCE modules and 3rd party ones. In theory they should all be able to be used in the same way without additional hacks to the Introjucer or the files they generate. If you can create a blank Introjucer project, tick a few modules, build and go why should the procedure be any different if you are using a 3rd party module?

I think this is where we ultimately differ in opinion Vinn, we see modules as different entities used in different ways. Does anyone else have any ideas on the subject, I know aiit and masshacker have both created modules.

[1] Yes, modules are usually in their own namespace so you would have to allow for this somehow (e.g. add a using directive in StandardIncludes header). Also some modules could rely on 3rd party libraries that are not included as part of the module’s source e.g. the dreaded cURL. While it would be nice to avoid these linkage problems some libraries don’t make this easy so it is necessary to build and link them manually. In this instance clear instructions should be included in the module and these dependancies should be optional (a’la module config flags).

3

I agree with you 100% but there is no JUCE module “system”. At least, not one that seamlessly supports third party modules. That’s why I haven’t bothered creating the juce_module_info file. When Jules upgrades IntroJucer to support multiple paths for locating modules, and some kind of central URI for pointing IntroJucer to new places to find modules, then I’ll add the necessary juce_module_info files. No sense doing this until there is official support in IntroJucer.

Yep this is true, and hopefully Jules will do the sensible thing and make this automatic for third party modules located in places other than the JUCE modules directory.

And that’s exactly what I was aiming for. But also, remember that IntroJucer is the arch-enemy of structured build environments. It imposes its organization on the developer and forces you to work in certain ways. First of all there is the requirement that every IntroJucer project must share a JUCE source tree. This is totally unworkable, if you have two products and one is frozen for testing then you can’t update JUCE in the other one? At the very least, the modules location should be part of the .jucer file and not part of the per-user operating system settings. This way each .jucer project specifies its own location of the modules.

Then we have the problem that IntroJucer only gives you access to a subset of the functionality in your build system. I have made numerous posts about these problems, including the inability to generate a symbol file for a release build (mandatory for testing and profiling), and no way to turn off Link Time Code generation. Similar problems exist with Xcode. For example, using IntroJucer means entire IDE features become unavailable. Useful features like Visual Studio property sheets or Xcode xcconfig shared build settings.

The biggest problem of IntroJucer is that it holds the entire spectrum of JUCE developers hostage to the whims of a single developer (Jules). If he is too busy, or unwilling, to let Link Time Code Generation get turned off well then you’re out of luck unless you want to start maintaining your own version of IntroJucer (impractical). We can already see this problem with the issue of keeping the static library project up to date. Since my commercial application doesn’t use IntroJucer (and can never use IntroJucer due to the intricate, IDE-specific build steps required) the only way I can browse JUCE sources on Windows is to add the JUCE static library project from “extras” to my solution. Otherwise I would need to use Windows Explorer to open the file (yuck!). Without the static library project it is not convenient to do a solution-wide “Find in Files” or set a breakpoint. And Jules said he plans to drop support for the static library. This is a glimpse into what the future holds for people dependent on IntroJucer.

Yeah you’re going to need to do a little extra work to incorporate VFLib but this is only temporary until Jules adds the necessary 3rd party module support into IntroJucer. Keep in mind that the audience for VFLib is not a casual programmer who wants to display an audio thumbnail. Besides miscellaneous utilities, the “meat” of VFLib is the concurrency model and the classes it provides for implementing that model. Compared to the complexity of building a concurrent system using these powerful tools, the effort required to use VFLib without better IntroJucer support (or use it without IntroJucer at all) is negligible. The demonstration applications serve as an example for using the library (SimpleDJ in particular) so it was more important to provide something that builds and runs out of the box, versus better IntroJucer integration.

I don’t consider copying sources out of a local git repository into JUCE/modules/* to be a viable solution at all, this only creates a maintenance nightmare. How would you make local changes? Those foreign modules wouldn’t even be under revision control. No structured build environment would accept such a workflow. That is why I did not bother with the IntroJucer integration.

[quote]I would like a few options added to the Introjucer and the module_info files:

  1. An Introjucer option to specify additional module search locations
  2. Have the Introjucer automatically include the module directories in the search path. Then we can legally do something like: #include “juce_core/native/juce_BasicNativeHeaders.h” no matter where our modules are relative to the main JUCE modules directory.
    [/quote]

These are both certainly needed. As I mentioned, I would add:

  • Public URI that Introjucer can consult for discovering new 3rd party module locations
  • Module locations part of the .jucer file instead of the user’s system settings

My opinion is the same as yours; of course, it should be possible for third party modules to be integrated into IntroJucer just like the regular JUCE modules. And I believe that was Jules’ intention all along, he just hasn’t gotten around to it. He’s only one person after all (a separate problem, better discussed in a new post). When IntroJucer makes it practical to have a workflow that includes 3rd party modules I will make the necessary changes in my library.

BUT…what I do not want to see is the requirement that IntroJucer is used, for JUCE, my modules, or any modules! It should always be possible to support a workflow that does not use IntroJucer at all. That means keeping the JUCE static library project up to date (and my own VFLib static library project, which as you can see I keep up to date). Ideally, IntroJucer could pick up features that synergize rather than conflict with these external workflows (for example, importing a Visual Studio 2010 project to IntroJucer). At a minimum, practical development should be possible without IntroJucer.

4

One of the things I used to love about JUCE was that no extra build tools or programs were required. This is why I didn’t go with Qt. It needed its own build environment and that ridiculous source code preprocessor for handling their “signals and slots” implementation (which JUCE manages without such extra tools).

Unfortunately JUCE is going the way of Qt with its increasing exclusive reliance on IntroJucer - right down to the integrated CodeEditorComponent (Qt Creator anyone?)