Future directions for JUCE library structure

General JUCE discussion
1

Hello Juce-people!

I just wanted to post a quick description of some big structural changes that I’m currently planning, and hopefully get some feedback from you good folks about it.

Having spent the last couple of years happily letting the introjucer do all the dirty-work involved in knitting the juce library into my application projects, I’ve become more and more convinced that the best way to use libraries in C++ is just to let a tool automatically stuff the library’s cpp files into your projects, and to avoid all the messing-about with static libs. The introjucer does this really well by having a lot of custom code that understands how the juce library works, but I’ve been figuring out a way to extend the same technique to a wider range of libraries.

So… Phase 1 of my cunning world-domination plan will involve the following changes:

  • I’ll break the juce library up into about a dozen smaller modules. Some of these will be dependent on other ones (e.g. there’ll be a “core” unit that all the others depend on), but you’ll be able to choose which modules you use in your project. So if you don’t need a GUI, you don’t need to include that module; if you don’t need audio, you won’t need to waste time including or bloating your app with those classes.
  • I’ll design a format for a library descriptor file (user-editable, in JSON or XML), which describes a c++ library module. This will contain its name, version, dependencies, header files, cpp files that need to be included in the app that uses it, etc etc. Each module will be uniquely identified by a URL, from which it can be downloaded (in a special kind of zip file).
  • One of these library module zip files contains just the raw source code, and the library descriptor file. There will be no makefiles or anything platform-specific in there, as the library will never be compiled except as part of another project.
  • When creating an introjucer project, your app’s project will specify a set of library modules that it requires, and the introjucer will be able to go off and automatically fetch each of the library modules by downloading its zip file.
  • The libraries that your app uses will be unzipped into a sub-folder (hidden away in the introjucer’s auto-generated files area). The library’s header files will be automatically added to your main app header, and all of its cpp files will be added to the project (in the same way that the juce amalgamated files are added now).

So… having extended the introjucer to be able to deliver any library, the next logical step will be to allow all of you guys to use it for your own libraries. The introjucer needs to have a list of all available libraries, so it’ll connect to juce.com for that information - so, as well as listing the standard set of juce modules, I’ll also be able to include 3rd party modules in there. That means that if you’re on the list, somebody will be able to download your library and incorporate it into their project by simply ticking a check-box in the introjucer.

There are lots of little details that I need to flesh-out, (e.g. versioning), but hopefully this will give you the gist! None of it’s particularly difficult to implement, but could create a great platform for easy installation and management of c++ libraries. Comments and opinions welcome!

2

Could you outline the modules that will exist. Will it be like “audio/midi”, “network”, “graphics” and so on? or will it be a different split ?

3

I’m just tinkering with that at the moment. It’ll mainly depend on which bits can be separated without any dependency problems.

4

I like the single monolithic library approach. I use my own Visual Studio project file for juce. I do not use Introjucer. I fail to see how these changes will benefit me, and I see potential they will slow down my workflow.

Why are we doing this again? The linker strips out code that doesn’t get used, so what’s the big deal?

5

This part sounds pretty neat actually.

6

I have nothing against modules (for additional third-party stuff), but the current juce stuff should be monolithic like it is. Why? There is no real benefit, instead it makes things a little bit complexer. And i would like to use JUCE without Introjucer, there are often settings you can also set in the native IDE, and if they will be overwritten every time you update your JUCE-version, that will be horror!

7

Sounds great, as a former JAVA programmer, it’s like having the modularity of Spring with the sharing capabilities of Maven. That’s a big upgrade.

8

[quote=“TheVinn”]I like the single monolithic library approach. I use my own Visual Studio project file for juce. I do not use Introjucer. I fail to see how these changes will benefit me, and I see potential they will slow down my workflow.

Why are we doing this again? The linker strips out code that doesn’t get used, so what’s the big deal?[/quote]

It’s getting just too big as a monolithic library, and breaking it up slightly will not only make it easier for people to pick and choose the bits they want, but will also improve build times. It won’t be any more difficult to use than it is now - hopefully easier! And sure, the linker does a great job of throwing away dead code, but there are many cases where it simply can’t do so, and it’s by no means perfect. All those OS callbacks and native message switch statements can’t be optimised, and nor can many of the static objects in there, all of which have complicated dependency chains.

But the main reason is that there’s also a lot of functionality that’d be great to do, but which I simply don’t have time to write myself, and some of you have been writing cool libraries that complement juce - I want to make it easier for you to get your code into the hands of the users. I see it as being similar to the linux package-manager systems, but for cross-platform code.

It also makes sense to separate things like the plugin framework into separate modules. At the moment the introjucer is packed with hand-coded functionality to create the appropriate project settings for an audio plugin, but I’d like to be able to abstract all of that stuff out of the introjucer and put it into the library module itself. So then if someone wanted to write their own module to provide some other kind of plugin (Photoshop plugin, anyone?), they could do that without needing my help.

Another goal is to use these modules to wrap libraries that are commonly-used but a PITA to deal with. E.g. anyone who’s ever tried using ffmpeg, truetype, mpglib, etc in both a win32 and unix build won’t have enjoyed the experience. So rather than having to face the horrors of cygwin and to juggle a bunch of platform-specific libs and custom build settings, what I want to be able to offer are modules that contain wrapped versions of these libraries, so that you can simply open the introjucer, tick the box that says “ffmpeg”, and your project will be instantly updated to include all the code and headers, ready to build on any platform.

9

Sounds awesome, Jules, +1 from me.

JUCE is great as a library for audio features and core stuff, like strings and files, even when one does not need using graphics, gui and the message manager.

This sounds like a good occation to decouple audio and graphics parts.

10

This won’t make any difference to your ability to do it that way if you want to - the only difference will be that instead of putting juce_amalgamated or JuceLibraryCode1.cpp, JuceLibraryCode2.cpp, etc into your project, you’d include juce_core.cpp, juce_events.cpp, etc., which is actually a lot more intuitive.

11

I really like the convenience of the amalgamated, which simplifies a lot the project building (just one reference to add). However, as you said, it’s getting too big, too long to compile (the full amalgamated takes at least 20 minutes to compile on my Macbook :expressionless: )…

The modular library thing is a great idea! :smiley: If I understand this right, it would be several parts of the library amalgamated in independent .cpp/.h files, but still not as big as the whole amalgamation, and not as individual as the source?

12

Yep, that’s the idea.

(But not actually amalgamated, each module would be included via a single cpp file. Like including juce_amalgamated_template.cpp rather than juce_amalgamated.cpp - you get the same result, but original the files are referenced directly)

You need more memory! My macbook does it in 20 secs, but it uses every bit of the 8GB ram.

13

So, if I understand this right, the single files will still be available but only the way they are included changes, right?
I depend on the single .h and .cpp files for easier debugging and for subclassing purposes. These things would be much more complicated if it was all wrapped up into some big module files.

I sometimes faced the situation, where I had to modify some small bits of the juce code here and there, so I hope the single files will still be the base for the whole code. I also have setup my own personal juce git repository and merge this with the official tip from time to time.

And the last one: I like the Introjucer, but in most cases I need the freedom to set up my projects and IDE settings on my own. For this reason I hope that the Introjucer won’t be the only way to get the juce files and to setup a project - what I mean is, will the files still be available using just git like it is right now?

14

You’ll still only have to include one header and one cpp (per module). But those won’t be huge mega-files that contain all the other code, they’ll just be normal files that include all the separate code files directly.

Then this is probably better for you, because each of your projects will get its own personal copy of the juce code tree, which you can edit and check into your repository. The introjucer will be able to keep these copies up-to-date (probably supporting the ability to merge new versions with your existing changes).

Yes, I’ll always need a git repo containing the code, so it’ll always be there to download directly if you want to. The introjucer will just be a more automated way of getting hold of it.

15

I disagree with Vincent, the linker doesn’t throw anything out unless you are not using ALL the functions of a compilation unit (file.cpp). This is because the COMPILER emits relative jump between function whenever it can to speed up the software, and the linker can not “remove” a function without re-compiling the unit to fix the relative offset (which it doesn’t obviously).
The only solution, at least on gcc, is to build with -ffunctions-in-section (or something like that) which put each function in a different section (bloats your elf anyway), and then the linker is able to garbage collect the sections, but you’re loosing the speed avantage of functions being in cache while being called. That’s why even the basic HelloWorld project in Juce takes 5MB!

That said, I think there is also a more important issue here with deprecation.
As some may already know, I’ve a lot of components cooking, but I can hardly distribute them, since the Juce code base is always ahead of my branch (and I don’t have time to backport/maintain the new stuff from Jules in my branch)
Whenever I get 1 hour in front of me, I’m trying to port the code to the HEAD, but it’s usually very frustating to see that I would always lag behind since each time I think I’m done with porting, Jules already moved behind.

So, to sum up this discussion a bit:

Jules said “I’m the maintainer / dev of the ever growing Juce library, and it’s a pain for me to maintain so many lines of code, because it takes ages to compile and the dependencies are too large”
TheVinn said “Since I don’t modify Juce code, I’m using a static lib that was compiled once for all, and it’s quite fast for my competition’s CPU”
Some users using amalgamation said “Yes, please do”
I say “The linker is not magical, it will not rewrite a compiled code. So modularizing the library is a good idea anyway. The very very best would be the ability to build “static library” project/makefile for the downloaded content via the Introjucer too. So whatever the solution (amalgamation I don’t like, static lib), you will still be able to use the code. Also, please find a way to solve the dependency/deprecation/obsolescence issue. Believe me, the versioning issue is probably the biggest one.”

16

As long as I’m not dependent on using Introjucer in order to retrieve Juce and compile my projects, I think I’m ok.

17

Versioning is one of the main reasons why I want to do this.

With this system, whatever libraries you use in a project will get stored inside that project, and I’d expect people to check them into their project’s repository. That way, your project is 100% self-contained, so it can be rolled backwards and forwards to any point in its history without worrying about external dependencies. If you have multiple projects (or branches) that were built against different version of juce, there’s no problem, each one will always work.

I should also have mentioned that I’m planning on providing command-line tools for this, rather like GIT, so you’d be able to manage your project’s libraries in the same style that you’d manage a repo, e.g here’s how it might look when you want to update your version of juce to the latest one:

[code]$ cd myproject
$ juce status
Project “myproject”
Libraries used:

  • juce_core, version 1.2.3 (up to date)
  • juce_audio, version 1.2.3 (up to date)
  • juce_gui, version 1.2.3 (New version available! 1.2.4)

$ juce update --all
Updating juce_gui… Downloading 1.2.4… Merging (No conflicts detected)… Updating projects… Done!

$ juce library --add juce_cryptography
Adding library: juce_cryptography… Downloading… Updating projects… Done!
[/code]

Re: static libraries, it’ll still be possible to build a static library for juce by creating a wrapper project which itself simply includes all the juce modules, and compiles to a static library, exposing the symbols. Easy enough to do, and this may suit some people, but it’s not going to be an idea that I actively encourage.

18

I’m using a very big classpath (around 1M line of code), and I can’t really deal with importing all the files in the projects using it (I guess you know the issue, since it’s quite similar to Juce).
Using static library has the enormous advantage of being built once so bootstrapping a new project or simply creating a new branch out of a commit I’m trying to merge, is fast, as I don’t need to recompile the whole stuff.

If I had to rebuild the whole library each time (takes me around 15mn on my computer), wouldn’t be pratical.
That said, I really love the idea of the command line application (but, please don’t call it “juce”, don’t reproduce the “Jucer” “jucer” issue again. Give it a understandable name).
How will you deal with other’s people “package” ?
For example, when someone release a VideoComponent using this system, how do you deal with the “dependency” on Juce’s commit version ? I mean, my component worked on Juce’s version of 06/27/11, but it doesn’t mean it’ll work on Juce’s version 06/28 ?
Because unless it doesn’t deal with Juce whatsoever, you’ll always end up with broken version for the libraries.

19

I think that’s the perfect name! After all, there’s no other app that’s actually just called “juce”! Anything longer would be a pain to type.

Exactly the same way it deals with my modules. They’ll all be treated equally.

A module’s definition file specifies which other modules it depends on, including their version numbers. So it can go and fetch all the correct versions and install them for you.

(But obviously if you try to combine two modules which depend on conflicting versions of the same library then you’ll just get an error message!)

20

Ok, then you should handle version “range” in the definition => compatible with version > 2.4.3 and <= 2.6.9. Else, it’ll really become a nightmare.
Ideally, (I know it’s very difficult), the versionning scheme should follow the class interface for all classes in the module (that is, each time you/I change a class interface, even slightly, the version should bump). It’s quite hard to do on a class-by-class basis, but that’s the only way it could made sure it’ll work.