New JUCE module format proposal!


#1

Hi folks - I've been working recently on an update to the way that modules are defined.

Apart from just wanting to simplify things a bit, I wanted to make it less dependent on the Introjucer's project generation, and to make it easier for non-introjucer-users to add juce modules to their projects by simply throwing a few files and include paths into their project.

Below is my first draft of the new spec for how a module folder should look - comments welcome!

The TL;DR is that the JSON manifest file has been replaced by a more human-readable comment block in the module's .h file, and a lot of the things that were explicit in the manifest have become implicit behaviours based on file names, etc.

 

                                The JUCE Module Format

                                ======================

 

A JUCE module is a collection of header and source files which can be added to a project

to provide a set of classes or related functionality.

 

Their structure is designed to make it as simple as possible for modules to be added to

user projects on many platforms, either via automated tools, or by manual inclusion.

 

Each module may have dependencies on other modules, but should be otherwise self-contained.

 

                                    File structure

                                    ==============

 

Each module lives inside a folder whose name is the same as the name of the module. The

JUCE convention for naming modules is lower-case with underscores, e.g.

 

juce_core

juce_events

juce_graphics

 

But any name that is a valid C++ identifer is OK.

 

Inside the root of this folder, there must be a set of public header and source files which

the user's' project will include. The module may have as many other internal source files as

it needs, but these must all be inside sub-folders!

 

 

Master header file

------------------

 

In this root folder there must be ONE master header file, which includes all the necessary

header files for the module. This header must have the same name as the module, with

a .h/.hpp/.hxx suffix. E.g.

 

juce_core/juce_core.h

 

IMPORTANT! All code within a module that includes other files from within its own subfolders

must do so using RELATIVE paths!

A module must be entirely relocatable on disk, and it must not rely on the user's project

having any kind of include path set up correctly for it to work. Even if the user has no

include paths whatsoever and includes the module's master header via an absolute path,

it must still correctly find all of its internally included sub-files.

 

This master header file must also contain a comment with a BEGIN_JUCE_MODULE_DECLARATION

block which defines the module's requirements - the syntax for this is described later on..

 

 

Module CPP files

----------------

 

A module consists of a single header file and zero or more .cpp files. Fewer is better!

 

Ideally, a module could be header-only module, so that a project can use it by simply

including the master header file.

 

For various reasons it's usually necessary or preferable to have a simpler header and

some .cpp files that the user's project should compile as stand-alone compile units.

In this case you should ideally provide just a single cpp file in the module's root

folder, and this should internally include all your other cpps from their sub-folders,

so that only a single cpp needs to be added to the user's project in order to completely

compile the module.

 

In some cases (e.g. if your module internally relies on 3rd-party code which can't be

easily combined into a single compile-unit) then you may have more than one source file

here, but avoid this if possible, as it will add a burden for users who are manually

adding these files to their projects.

 

The names of these source files must begin with the name of the module, but they can have

a number or other suffix if there is more than one.

 

In order to specify that a source file should only be compiled on a specific platform,

then the filename can be suffixed with one of the following strings:

 

_OSX

_Windows

_Linux

_Android

_iOS

 

e.g.

juce_mymodule/juce_mymodule_1.cpp         <- compiled on all platforms

juce_mymodule/juce_mymodule_2.cpp         <- compiled on all platforms

juce_mymodule/juce_mymodule_OSX.cpp       <- compiled only on OSX

juce_mymodule/juce_mymodule_Windows.cpp   <- compiled only on Windows

 

Often this isn't necessary, as in most cases you can easily add checks inside the files

to do different things depending on the platform, but this may be handy just to avoid

clutter in user projects where files aren't needed.

 

To simplify the use of obj-C++ there's also a special-case rule: If the folder contains

both a .mm and a .cpp file whose names are otherwise identical, then on OSX/iOS the .mm

will be used and the cpp ignored. (And vice-versa for other platforms, of course).

 

 

 

                        The BEGIN_JUCE_MODULE_DECLARATION block

                        =======================================

 

This block of text needs to go inside the module's main header file. It should be commented-out

and perhaps inside an #if 0 block too, but the Introjucer will just scan the whole file for the

string BEGIN_JUCE_MODULE_DECLARATION, and doesn't care about its context in terms of C++ syntax.

 

The block needs a corresponding END_JUCE_MODULE_DECLARATION to finish the block.

These should both be on a line of their own.

 

Inside the block, the parser will expect to find a list of value definitions, one-per-line, with

the very simple syntax

 

 value_name:   value

 

The value_name must be one of the items listed below, and is case-sensitive. Whitespace on the

line is ignored. Some values are compulsory and must be supplied, but others are optional.

The order in which they're declared doesn't matter.

 

Possible values:

 

    ID:             (Compulsory) This ID must match the name of the file and folder, e.g. juce_core.

                    The main reason for also including it here is as a sanity-check

    vendor:         (Compulsory) A unique ID for the vendor, e.g. "juce". This should be short

                    and shouldn't contain any spaces

    version:        (Compulsory) A version number for the module

    name:           (Compulsory) A short description of the module

    description:    (Compulsory) A longer description (but still only one line of text, please!)

 

    dependencies:   (Optional) A list (space or comma-separated) of other modules that are required by

                    this one. The Introjucer can use this to auto-resolve dependencies.

    website:        (Optional) A URL linking to useful info about the module]

    license:        (Optional) A description of the type of software license that applies

    searchpaths:    (Optional) A space-separated list of internal include paths, relative to the module's

                    parent folder, which need to be added to a project's header search path

    OSXFrameworks:  (Optional) A list (space or comma-separated) of OSX frameworks that are needed

                    by this module

    iOSFrameworks:  (Optional) Like OSXFrameworks, but for iOS targets

    linuxLibs:      (Optional) A list (space or comma-separated) of static libs that should be linked in a

                    linux build (these are passed to the linker via the -l flag)

    mingwLibs:      (Optional) A list (space or comma-separated) of static libs that should be linked in a

                    win32 mingw build (these are passed to the linker via the -l flag)

 

Here's an example block:

 

     BEGIN_JUCE_MODULE_DECLARATION

 

      ID:               juce_audio_devices

      vendor:           juce

      version:          4.1.0

      name:             JUCE audio and MIDI I/O device classes

      description:      Classes to play and record from audio and MIDI I/O devices

      website:          http://www.juce.com/juce

      license:          GPL/Commercial

 

      dependencies:     juce_audio_basics, juce_audio_formats, juce_events

      OSXFrameworks:    CoreAudio CoreMIDI DiscRecording

      iOSFrameworks:    CoreAudio CoreMIDI AudioToolbox AVFoundation

      linuxLibs:        asound

      mingwLibs:        winmm

 

     END_JUCE_MODULE_DECLARATION

 


Creating user modules tutorial?
#2

...also.. 

In the course of implementing this I'd really like to remove the local-copy-of-juce-module-code option from the introjucer. In my experience it's not a very good way of managing your code - much better to have juce in a GIT submodule or just a sibling GIT repo rather than to rely on the introjucer to copy the code whenever it changes.

Are people using the local copy thing? If so, is it a vital part of a well-thought-through workflow for you, or are you just using it out of habit? Feedback wanted!


#3

Excited to see this. Removing the JSON file in favor of parsing the header file is a great idea!

EDIT: Is the "searchpaths" option from what I suggested here? Or is it for something else?

http://www.juce.com/forum/topic/request-adding-additional-header-paths-juce-modules

Also, it would be really nice if in addition to the header search paths we could add precompiled static libraries in the platform-specific files, since that's how a lot of middleware comes packaged - i.e. a way to add .lib/.a files to the projects.

Additionally, I for one think removing local copying is a great idea. I never really understood why you would do that when you can just use whatever directory you already have JUCE cloned to. The only time I've seen people mentioning using it on this forum is to avoid all the "is not a namespace" Intellisense problems under Visual Studio. By the way, the fix for that (since I haven't seen anyone else mention it) is to have the "namespace juce { ... }" around the code in every .h/.cpp file rather than around a block of includes in a higher-level .h/.cpp file (namely the main .h/.cpp file at the top directory of the module). It's a few more lines to add to each file, but working Intellisense is invaluable to me and as a result this is how I code all my JUCE modules.


#4

My main question is why get rid of the JSON manifest file? Is it not simpler to have it in a seperate file rather than parse it out of a header file?
My arguments for keeping it are mainly based around my contributions to the Juce package manager that bazrush has started - being able to import the juce_module_info file directly into a document database is very straightforward.

Having gotten used to the package.json of Node's npm, and having used Ruby's Gemfile, PHP Composer's composer.json etc.. I don't see why it needs to be included as a comment in a header file. Why not keep it in a seperate file that can be easily read into or written from a data structure?

On the other hand I can see the value in reducing the number of files. And parsing a bunch of colon seperated lines is only a couple lines of code..


#5

Jon, could you post that solution to http://www.juce.com/forum/topic/vs2013-editor-failing-resolve-symbols ?

That seems to be the official "broken intellisense" thread.

ŌÄ

PS Bazrush's fix worked for me!


#6

So here's a scenario under which local copy seems to be beneficial:

Developer has three projects A, B, and C which are at different stages of their life.

A is a final, released product for which the code is effectively "locked"; only critical bugs will justify modifying the code.  It is compiled with an "old" JUCE commit from when it was originally created and tested.

B is about to enter beta testing, so while the code isn't locked down there is also a desire to avoid unnecessary instability due to riding the JUCE tip.  So the code is based on the latest stable JUCE commit.

C is a new product at the earliest stages of development and uses all the latest features; because it's a prototype it can be unstable and riding the JUCE tip is just fine.

Local copy seems to be beneficial in that the local module copy can be held at an arbitrary commit indefinitely; A B and C each have their corresponding snapshots of the JUCE modules and they are only ever updated through an intentional act.  One will never accidentally compile A with B's version of the modules.  That is entirely possible if you rely on checking out the appropriate JUCE commit every time you switch from working on A to B to C.  And what happens if you want to have A and C projects open at the same time?  You'd have to remember to switch JUCE checkouts before every compile. 

Now imagine instead of just A B C you have [A .. Z] and it becomes even more critical to be able to hold each with it's corresponding version of the JUCE modules.

One solution might be "just use a separate JUCE repo for each project then".  Add in local customizations to the JUCE repo... customizations that need to be shared across the multitude of projects.  How does one manage it then?

It is entrely possible that there is an alternate workflow that would allow this - I'm no git expert.  But please make sure there IS a solution to this scenario before removing local copy. 


#7

Nicely put!

It seems that one would never start out with a 'local copy'. Only when the product is ripe for beta testing -- this is the first moment a developer might wish to create a local copy. So maybe ProJucer could have a button that makes a local copy of the current tip and reconfigures the project files accordingly? And maybe another one that goes the other way? That would be enough to cater for any workflow I can think of.

Is it correct to suppose that Team JUCE hold back breaking changes for major version releases?

I generally wouldn't use local copy in my own work. It would (surely?) be extremely rare for a commit to break code. And what if it does? We live in an age of live updates. The team could fix within hours if the problem is their end. And if it is an intentional breaking change I would prefer to update all my affected projects and continue to ride the tip.

I don't want multiple projects spanning multiple JUCE versions. That sounds like a headache! I would like the JUCE source code on my hard drive in exactly ONE location, and CURRENT. And all my GitHub JUCE projects to not contain JUCE source (nice and lean).

I plan to move ahead using /Dev/JUCE/JUCE which I regularly sync with the JUCE GitHub repo using GitHub Desktop (would be nice to automate), and /Dev/JUCE/Projects which will contain my own projects.

If I were designing firmware for hospital equipment monitoring heart rate (which is something I have done in the past) I would take care to work with a local copy at some point, but I would expect to take on the extra effort of setting that up.

ŌÄ


#8

The tip contains some known bugs sometimes, so you really want to be aware of them when you build/release (fogbugz please).
(but the latest tagged version often got the same ones anyway).

I used to release my builds with the latest tip. But lately there were some bugs introduced (mainly concerning plugins), and it's really annoying to read all the forum posts to see what could have possibly break, wait for a fix, re-build and release etc.
So I think I will stop riding the tip for all my projects, and will copy locally for the released ones. so that I stop being scared of introducing juce bugs everytime I fix/add a small functionnality in a project.

But I think you're right that a git solution (GIT submodule or just a sibling GIT repo) would be better. Not sure at all if that is feasible or not, but it would be great if the introjucer could offer and manage that actually :)


#9

We have exactly this situation and like you mention we "just use a separate JUCE repo for each project then".

We manage it using git submodules. This is the best workflow known to me to keep track of which JUCE versions exactly you used..


#10

Afaik there have been regressions in the AU wrapper since JUCE 4.1 was released (first there was the feedback bug when stopping playback on midi-controlled effects, and then it changed to mishandling of mono clips and silence on stereo tracks).

You can try SR's branch https://github.com/soundradix/JUCE where the AU wrapper is fixed and sidechain support is also available in RTAS. Hopefully the fixes will come to main JUCE too soon.

Cheers, Yair


#11

My main motivation was to have fewer, more human-readable files, and to simplify the usage of the modules. This means that the most trivial module would simply be a folder with a single header in it (or a header + one cpp file), and including it in a project without using the introjucer would just be a case of adding those files to your project. At the moment, that's not possible without reading the JSON and figuring out which cpps it wants to be compiled.

And it's very easy to parse - just a few lines of code, which will be there in the introjucer if anyone wants to use it.


#12

Yes, this is exactly the use-case for GIT submodules. For example, we do this at ROLI with Equator - it lives in a repo that has JUCE as a submodule. It provides a lot more flexibilty than local-copying and doesn't require the introjucer when you want to update it.


#13

Are people using the local copy thing? If so, is it a vital part of a well-thought-through workflow for you, or are you just using it out of habit? Feedback wanted!

The local copy thing is useful for projects that require a little alteration on juce classes. If you have a single source to read from, then these alterations will be taken with other projects based on juce too. This may not be the case you want. Now of course you could go through the trouble of making different branches, but this does complicate the updating proces. Now the complication is still real as it is, since updating Juce using Introjucer doesn't make it easy either, when making changes to the juce source.

My feedback? The local copy thing is useful for quickness but the git alternative would be better to prevent mistakes from happening, like accidentally overwriting your modified juce source copy. Using git this can be prevented. From a professional-workflow point of view, I'd say it is also better since changes to the project and juce source copy are required to be aligned with possible updates. If not, updating to git would fail and you are made aware that these changes are conflicting. Also, in the case where an update is available through git and the updates do not interfere with your modified source, then there is still no problem and decreases the time of the upgrade process.

In short: I do use local copy thing, but using git would be much better due version/source control. Removing this option in introjucer would force a better workflow ultimately.


#14

Thanks - that's my feeling too.

Especially in cases where you've got some local alterations, GIT is the smart way to do it.


#15

sounds good.
And so to work with thirdpart/custom modules, we would also create git submodules?
It would be great if we could have basic guidelines/recommended practice to organise projects code with the juce code, third/custom modules, the git repos and the introjucer.


#16

My guidelines would be easy to sum up:

Put your project in GIT. Use GIT submodules inside it for JUCE and any other 3rd-party code.


#17

That is probably the worst typesetting I've seen on the web so far this year. I'm not sure but it doesn't look like it's supposed to render that way.

The proposal looks good. We are not using local copies, we have our modified JUCE code in our own GIT repo.

--
Roeland


#18

(Yeah, all-new replacement forum coming in a couple of weeks!)


#19

I am also concerned in the possibility of adding header search path to static libraries for Juce Modules (Win/OSX/Linux).


#20

Adding to my prior post double checking the functionality of adding header search paths and requesting adding per-platform static library files, I'd like to also request adding a way to add binary resources to a module such that they'll be picked up by The Introjucer for projects in which the module is included.

This would be really useful for including common binary resources across products like company logos, test sound files, etc. Right now I'm using a hacky workaround for this where I'm using a BinaryBuilder Python script to build a JUCE-compatible binary blob for my module and including the .h/.cpp in my module code. It would be nice if I could just list out binary files in my module information and have Introjucer do this automatically with its nice features like breaking up the blob into multiple .cpp files, etc.