Juce amalgamation woes


#1

I’m in the middle of updating to the modules tip, now that it is on the main branch. Looking at the amalgamation files, I observe the following:

  • The juce_* files for the amalgamation are not true amalgamations, they #include other sources instead of being large self contained files

  • The Amalgamator is no longer part of extras

  • An externally provided “AppConfig.h” is required to compile modules. This is different from the previous approach, where the JuceLibraryCode#.cpp file included AppConfig.h before including the amalgamation cpp file.

  • Using my own modified version of the Juce Amalgamator (which will soon be able to amalgamate Freetype), I run into this problem:

juce_events.cpp

#if JUCE_CATCH_UNHANDLED_EXCEPTIONS && JUCE_MODULE_AVAILABLE_juce_gui_basics
 #include "../juce_gui_basics/juce_gui_basics.h"
#endif

The problem here is that a proper amalgamation cannot have #include lines for juce headers which are conditionally compiled, because the values of the macros are not known at the time the amalgamation is produced.


#2

Same problem here

juce_gui_basics.cpp

#if JUCE_MAC || JUCE_IOS
...
 #if JUCE_IOS
  #include "native/juce_MultiTouchMapper.h"
...
 #endif
...
#elif JUCE_WINDOWS
...
 #include "native/juce_MultiTouchMapper.h"
...
#endif

juce_MultiTouchMapper.h is conditionally included. So in the amalgamation, the file only gets inlined in the path where JUCE_IOS is defined to 1, thus malfunctioning on Windows.


#3

My first thought in fixing this is to simply #include “juce_MultiTouchMapper.h” at the beginning of juce_win32_Windowing.cpp but upon further analysis, it would seem that there is an unwritten rule when it comes to the .cpp files which are not directly compiled but instead included in other .cpp files: they are not allowed to use #include directives

Say what? :shock: … :cry:

They walk and talk like .cpp files but are second class citizens… this is looking like a mess!

Right now it looks like we’ve lost the functionality of having an amalgamation and bringing it back would be quite the challenge.


#4

Well, like the comments say inside them, the amalgamated files are just there to help people transition to the new module layout, and I’ll be getting rid of them eventually.

If you need some flags setting before including them, the thing to do is to just create a cpp file that includes your header and then includes the amalgamated.cpp file. That’s how the code was structured in the past anyway, I never designed the amalgamated.cpp file to just be added directly to a project without the appropriate flags.

[quote]Right now it looks like we’ve lost the functionality of having an amalgamation and bringing it back would be quite the challenge.
[/quote]

I’m failing to think of what lost functionality you might mean…? As far as I can tell nothing of value has been lost!

And something I want to explore in future will be to make the introjucer capable of amalgamating modules on the fly - so that when you tell it to use a module in your project, it will have 3 modes of doing so: Using the module files from their source folder; copying the module files into your project folder; or creating an amalgamation of the module in your source folder.

Nonsense. There’s now less boilerplate code, each file is much cleaner.

Besides, it has been many years since the last version of juce whose cpp files could all be compiled independently. The “native” folders have always had their headers separated from the cpp files that use them, so that all 3rd-party headers could be included first, avoiding conflicts. So really all I’ve done is carried that process to its logical conclusion, with the result that compiling a module-based build takes a fraction of the time it used to take to compile all the files one-by-one. It’s a vastly better system now IMHO.


#5

I don’t see how to produce an amalgamation…


#6

I would like to know how you plan on solving the problems mentioned in the first and second post to make this happen. I’m assuming that by amalgmation you mean a multi-megabyte source file that has no external dependencies other than system headers?


#7

Good! I don’t really want anyone to do that any more! You’re still welcome to use the old amalgamator tool, but I just don’t see any advantages in using it on the library now that it’s been modularised, so won’t be supporting that any more.

Yes, but I haven’t worked out any details yet, and don’t plan on thinking about it for a while.


#8

Good! I don’t really want anyone to do that any more! You’re still welcome to use the old amalgamator tool, but I just don’t see any advantages in using it on the library now that it’s been modularised, so won’t be supporting that any more.[/quote]

If I wanted to update the DSP Filters source tree to use the latest (modularized) tip, then how would I do it? Add 1,099 Juce source and header files to the SVN repo?

People use the DSP Filters demo application as a starting point for writing their own audio-enabled apps. Including the full Juce source tree is more cumbersome than simply adding a couple of large .cpp files.

Compare:

Amalgamation:
[attachment=1]amalgam1.png[/attachment]

Full Juce source tree:
[attachment=0]modules1.png[/attachment]

At this point one might expect cries of “Use Introjucer” but is that now mandatory? Is it a requirement now to have the full 1000+ file Juce source tree?

What if every good sized open source project had its own “IntroJucer” type app for downloading modules and maintaining the source tree can you imagine how annoying that would be? Me personally I am using five external third party libraries it would be quite frustrating to have to deal with five new added build tools.

The amalgamation was a great way to produce an open source project distribution that has no external dependencies. It is a shame that we must lose this functionality in order to have modules.

Surely there has to be a way?


#9

FWIW yanking support for an amalgamation really stinks!


#10

TBH, since I’ve used GIT rather than SVN, adding 1000 files doesn’t sound like a big deal, and yes, that’s how I would manage my own projects. You can check-in and update those files using a wildcard, so it doesn’t really matter how many there are, does it?

But like I said, to reduce the number of files, I would eventually like to get the amalgamator to help out with this. But I don’t see it as vital.

No, but if you have some kind of custom build system and something I change breaks it, that’s not my fault! I can’t promise that all my changes will play nicely with any possible weird custom set-up that people might be using!


#11

The amalgamator is not the problem, I have a modified version of it that works really well. The problem is that the new system of includes makes it impossible to produce a correct amalgamation without changes to the juce sources - this is a departure from the way things worked before modules.


#12

Well, that’s not surprising, since I’m no longer testing my changes to see if they break amalgamation. But if you think of any minor tweaks that would make the code work better for you without affecting anything else, I’d obviously be willing to add that kind of thing.


#13

Well in order for the amalgamation to work it would be necessary to be more strict when organizing the source files. This is a general guideline:

  • A juce header should not appear as an include line in more than one place (whether in the same file or different files).

The problem with juce_gui_basics.cpp illustrates this perfectly:

Original code:

#if JUCE_MAC || JUCE_IOS ... #if JUCE_IOS #include "native/juce_MultiTouchMapper.h" ... #endif ... #elif JUCE_WINDOWS ... #include "native/juce_MultiTouchMapper.h" ... #endif

To fix this we need to pull the #include line out and up into its own conditional include area which covers all the cases:

Fixed code:

#if JUCE_IOS || JUCE_WINDOWS
#include "native/juce_MultiTouchMapper.h"
#endif

#if JUCE_MAC || JUCE_IOS
...
#elif JUCE_WINDOWS
...

Another work-around is to let the amalgamator allow the inlining of headers to happen more than once if the header appears in a conditional compilation directive but this might be ambiguous since every header includes a multiple-inclusion supression conditional compilation directive at the very beginning. And the resulting amalgamation would be larger than it needs to be (although this might be an acceptable compromise).


#14

Ok, I’ve no objections to making that change.


#15

Cool, but this would need to be done for every Juce file that is included more than once in the juce source tree. Finding all of them is non trivial. Plus, you would need to remember to do this when adding new #include lines (if the resulting file would be included in more than one place).


#16

Are you sure there are many instances like that? I hit the same problem when I originally did the amalgamation, but there were only a couple of places where I had to fix the code.


#17

Not “many”. Less than ten I think. It seems to happen mostly when detecting platform.