Breaking change by build number revision: AudioProcessor cannot be but in some static library any longer

[DrTT]

Hello,

first of all thanks for all your effort going into the development of JUCE. I use that framework for about six years now for open-source multi-platform audio plugins and have been quite satisfied with the results.

As I am a more traditional developer and IT architect, I have built my own CMake based build system without actively incorporating Projucer. After significant release changes in JUCE I just use Projucer on the example files to find out what include paths, configuration variables and source code include files are needed now and then update my CMake files accordingly.

This has worked out well enough in the last years, but now I am struggling. In the transition from 8.0.10 to 8.0.11 there have been several “breaking” changes for me, which are still there in 8.0.14.

• I have a static library with my own specialized classes for AudioProcessor, AudioProcessorEditor and AudioProcessorEditorWidget (as a Component within an editor) factoring out the commonalities for all my plugins into some parent classes. The technical wrapper (VST, AU) is then later given for each plugin by adding the corresponding juce_audio_client_XXX.cpp or juce_audio_client_XXX.mm to its list of files for the build. This works fine for VST3 on Windows, Linux and MacOS and AU on MacOS.

  Beginning with 8.0.11 this does not seem possible any longer. The juce::AudioProcessor class compiled for the static library has no idea for what wrapper kind it will be responsible later (since e.g. the configuration variable JucePlugin_Build_VST3 is not yet set). It then seems to reference code for AAX, LV2, VST3 etc. But you cannot just include those additional files into the static library, because then you have to provide the JucePlugin_PluginCode. Unfortunately that code is only available for each plugin at a later build stage and it is counterintuitive to produce a static library intended to have common code for all plugins and then have a required individual setting to compile the code fragments.

• Required audio processor includes like pluginterfaces/base/fplatform.h seem to have migrated in 8.0.11 from juce_audio_processors to juce_audio_processors_headless.

  I can see the reason for the change from an architecture perspective, but it breaks several include paths. And even if those are corrected accordingly, the first problem above will be triggered: so both changes are obviously interconnected and that makes an incremental transition of client code impossible.

We are talking about a version change on the build number level, which is - in common understanding - much less than a minor or even a major version change and should be either cosmetic or bug-fixing. Granted, I have great sympathy for an architectural development of a system, but those kinds of changes are severe in my opinion and are not just tiny revisions, so this is at least a minor version change. And the first one does not even occur in the BREAKING_CHANGES.md or in the CHANGE_LIST.txt file.

On can argue that the use of Projucer is mandatory for JUCE and that I should just annotate my files accordingly and use it. But I do not want to do it, because I think that even a framework should be accessible in a more technical way from scripts outside of that ecosystem similarly as you would do for a library.

Can anybody help here?

Best regards,
DrTT

[reuk]

Just to make sure I understand correctly: Are you using JUCE’s built-in CMake support (e.g. add_subdirectory(JUCE), juce_add_plugin(...)) or have you written your own build from scratch? I’d strongly recommend to use the built-in support if if you’re not already, since this is tested on each release and we make efforts not to break it.

@DrTT

• The juce::AudioProcessor class compiled for the static library has no idea for what wrapper kind it will be responsible later

In a ‘standard’ JUCE build, the static library (SharedCode) will be shared amongst all plugin formats, so it must be capable of supporting all the enabled formats simultaneously. However, the static library is only intended to be used for a single plugin, so the JucePlugin_* macros (PluginName, VersionCode, IsSynth, etc.) will be set on the static library.

@DrTT

It then seems to reference code for AAX, LV2, VST3 etc. But you cannot just include those additional files into the static library, because then you have to provide the JucePlugin_PluginCode.

I’m not sure about this. The SharedCode library shouldn’t depend on any plugin-format-specific code unless plugin hosting is enabled. All the format-specific stuff happens in the juce_audio_plugin_client module. Currently, you must build this module separately for each plugin format, passing preprocessor flags appropriate for each particular plugin. Note that juce_audio_plugin_client will pull in included files from juce_audio_processors_headless. The best way to handle this is to set up the include paths so that juce_audio_plugin_client can locate the files each time it is built; it is a bad idea to try to move all the plugin-format-specific stuff into the shared static library. JUCE is not written to support this use-case so it’s unlikely to work.

@DrTT

those kinds of changes are severe in my opinion and are not just tiny revisions, so this is at least a minor version change. And the first one does not even occur in the BREAKING_CHANGES.md or in the CHANGE_LIST.txt file.

The vast majority of users use the built-in CMake support or Projucer. For those users, this was not a breaking change. Again, I’d encourage you to check out the CMake support that JUCE provides, as this is much less likely to break than a custom-rolled solution.

[DrTT]

Hello @reuk,

thanks for your quick and detailed answer!

You wrote:

Just to make sure I understand correctly: Are you using JUCE’s built-in CMake support (e.g. add_subdirectory(JUCE), juce_add_plugin(…)) or have you written your own build from scratch?

I have written my own CMake build chain from scratch a significant time before JUCE adopted CMake and had no problems so far, because I am experienced with CMake programming.

As far as I can see you are the main author of the JUCE CMake support, so I understand that there is a lot of effort going into that tool chain and I thank you for that.

But I have looked at it some time ago and it does not fit my needs. For example, the following points were minor or major annoyances:

• The definition of the plugin parameters is done within the CMake file, while in my case each plugin has an include file for that (as formerly in JUCE). Both approaches have their merits, but I prefer mine, because it keeps configuration information with the source code. One could parse the include files and use the functional approach, but this is a bit overkill, when the original approach works.

• Settings for the compiler & linker are automatically defined and then also apply to my own modules.

• My impression is that the JUCE CMake approach assumes that CMakeList.txt files are embedded in the source tree or at least that the CMake file structure somehow mirrors the source tree. I do not like either solution: my source tree does not at all contain any CMake file and my CMake modules are in a common cmake-buildsetup directory with a different module structure.

I’d strongly recommend to use the built-in support if if you’re not already, since this is tested on each release and we make efforts not to break it.

I appreciate your advice, but it means that I either have to use ProJucer or your CMake macros and that JUCE obviously cannot be used as a simple C++ framework without either of those.

The juce::AudioProcessor class compiled for the static library has no idea for what wrapper kind it will be responsible later

In a standard JUCE build, the static library (SharedCode) will be shared amongst all plugin formats, so it must be capable of supporting all the enabled formats simultaneously. However, the static library is only intended to be used for a single plugin, so the JucePlugin_* macros (PluginName, VersionCode, IsSynth, etc.) will be set on the static library.

Presumably we are not talking about the same thing. I am building my own static library consisting of my own classes AudioProcessor, AudioProcessorEditor and AudioProcessorEditorWidget for later use by my plugins and using the following list of JUCE classes:

• AudioBuffer, AudioChannelSet, AudioParameterChoice, AudioParameterChoiceAttributes, AudioParameterFloat, AudioPlayHead, AudioProcessor, AudioProcessorEditor, AudioProcessorParameter, Colour, ComboBox, ComboBox::Listener, Component, Graphics, Label, LookAndFeel, MemoryBlock, MessageManagerLock, MidiBuffer, NormalisableRange, NotificationType, ParameterID, Rectangle, Slider, String, StringArray, TextEditor

Except for AudioProcessor with possible wrapper kind variants, nothing looks like it could pose any specific problems when being precompiled into a static library. And up to and including v8.0.10 there never have been any. And what I did up to then is link that static library to the plugin code and also provide a wrapper file of the correct kind configured with plugin-specific JUCE defines. And that had satisfied any open external references.

Now in 8.0.11 the library built from the above classes suddenly has open external references to e.g. getAAXClientExtensions or AU_EffectParameterListener which cannot be satisfied by later including the wrapper client file like e.g. juce_audio_plugin_client_VST3.cpp.

So in my opinion this does not have anything to do with my CMake build chain, but with collecting those classes into a static library and a changed interpretation of some of the JUCE configuration variables. Obviously now some code fragments are activated that have been passive before.

The vast majority of users use the built-in CMake support or Projucer. For those users, this was not a breaking change.

Fair enough, but this means that you consider JUCE and its build systems a unit: then JUCE is not primarily a C++ framework with optional support tools, but a suite with more or less mandatory generators.

Best regards,
DrTT

[reuk]

@DrTT

• The definition of the plugin parameters is done within the CMake file

That’s not the case, parameters are still defined in regular C++ code.

@DrTT

• Settings for the compiler & linker are automatically defined and then also apply to my own modules.

I think this is the case for JUCE’s CMake support too. If you set target_compile_options on the shared code target, those options will apply to every module built into that target, including user modules.

@DrTT

• My impression is that the JUCE CMake approach assumes that CMakeList.txt files are embedded in the source tree or at least that the CMake file structure somehow mirrors the source tree.

That’s how we arrange things in the repository, but I don’t think user code would need to do that. You could equally have a CMakeLists in your cmake-buildsetup folder that uses target_sources to add individual source files from another folder into your build. If you’re using JUCE modules, you can call juce_add_module from another directory too.

All of that to say, maybe it’s worth taking another look. I expect the built-in support should be able to handle your use-case.

@DrTT

I either have to use ProJucer or your CMake macros and that JUCE obviously cannot be used as a simple C++ framework without either of those

That’s accurate, but I don’t think that using JUCE’s CMake functions is necessarily a bad thing, supposing they’re flexible enough to build your project.

@DrTT

Now in 8.0.11 the library built from the above classes suddenly has open external references to e.g. getAAXClientExtensions or AU_EffectParameterListener which cannot be satisfied by later including the wrapper client file like e.g. juce_audio_plugin_client_VST3.cpp.

It’s difficult to say for sure what’s going on here, it would be helpful to see the actual linker errors. getAAXClientExtensions() is defined inline, so I can’t imagine why the linker would complain that the definition is missing. I don’t see AU_EffectParameterListener referenced anywhere in the codebase.

@DrTT

this means that you consider JUCE and its build systems a unit: then JUCE is not primarily a C++ framework with optional support tools, but a suite with more or less mandatory generators.

Yes, I think it’s standard practice for libraries that ship with a CMake build to expect clients to consume that build instead of rolling their own. Of course, clients may choose not to, but in doing so the client is accepting responsibility for adapting their custom build to any upstream changes. This isn’t a hypothetical: JUCE vendors some third-party libraries internally and replaces their build-systems. When we update our dependencies, we also need to spend time ensuring that our builds of those dependencies still work.

[DrTT]

Hello @reuk,

thank you for your kind description and explanation.

You wrote:

All of that to say, maybe it’s worth taking another look. I expect the built-in support should be able to handle your use-case.

Okay, I’ll give it a try. Even for the three platforms Windows/MacOS/Linux it is tedious to adapt my own build pipeline to their changes. And it is fine, when you at JUCE do the heavy lifting

To be honest: I do not think that the problem with the static library is one of the build pipeline. In my opinion there has been a subtle code change between 8.0.10 and 8.0.11 in the AudioProcessor or related classes that break this build strategy with combining those parent classes into some library before deciding what kind of wrapper a plugin needs.

Anyway, I try using the “official CMake includes” and call you back when it still does not work.

Promised.

Best regards,
DrTT

[peteatjuce]

FWIW, I also don’t use ProJucer or the JUCE CMake approach.

The reason is that my code base is very old, and TBH I’ve had to apply a lot of patches to JUCE for e.g. tvOS support and what have you over the years.

I recently moved to using my own CMake to “auto-build” my Windows projects, but these don’t use the JUCE Cmake scripts; I’ve found it easier not to. Android has always used CMake at is core, of course.

My Apple projects are all maintained “by hand”. I can’t use CMake, as I’ve been unable to figure-out a way to have CMake automatically generate unit tests for multi-target cross-platform projects (my Apple projects are for macOS, iOS, tvOS; and support WatchOS and CarPlay).

Anyhow, there you go. Just an alternative perspective!

Best wishes, Pete

[DrTT]

Hello @peteatjuce ,

thanks for your encouraging message!

It seems that the history of your pipeline is similar to mine: we had already set up a sort of batch automation for the build even before official CMake support was available.

I am fortunate, because my CMake pipeline can handle the - for me - relevant platforms Windows, Linux and MacOS (all with x86_64 architecture) as well as the VST3 and AU plugin formats. So it covers already a lot of ground.

The official JUCE CMake pipeline is on the one hand impressive, because it allows to provide results for so much more platforms and so much more plugin formats without great effort on your side. You do not have to fiddle around with include paths, configuration variables, libraries etc.: it is all taken care of by the JUCE pipeline with just a few CMake function calls and C++ code annotations.

But on the other hand it seems to be restrictive. I had a look at it in the last days and several things were - in my opinion - not optimal. Maybe I have not yet understood it to the necessary level, but those were my first observations:

• Documentation is a bit thin. I have found only one HTML page with hints on how to use that pipeline, containing the description of only a few CMake helper functions and a few variables (plus the description on how to use CMake, which I already knew ). The main message is: look at how we did it for the examples. Hmm.
• Obviously the idea is that one first builds the complete JUCE project (but you can also skip the examples or extras, if you want to, which IMHO should be the default).
  I have not yet seen such an approach in a framework. Normally you just include some framework header files into your project and those themselves then include other code files such that all the required framework code is by magic incorporated on demand into your project. Of course, this can lead to other problems, but those can be solved…
  And no: the framework is not a subproject of my project, it is just a code fragment supplier and my project is its client.
• Within that official pipeline there does not seem to be any concept of an intermediate library target with JUCE classes. I would like to have a static library containing some overall relevant JUCE classes without any plugin-specific configuration so far, but I only found the three functions juce_add_gui_app, juce_add_console_app and juce_add_plugin and a static library is none of those in my opinion: those describe end product targets.
• As I am trying to take given advice seriously, I had a look at the CMake pipeline itself, to see whether I can reuse some of it to craft my own juce_add_intermediate_target function. Unfortunately, however, the CMake code is inconsistently commented: whilst some sections contain an acceptable number of comments, very often it is only sparsely commented or contains no relevant comments at all.

All in all, this doesn’t strike me as particularly promising. Of course, this is all rather nit-picky, and perhaps these are just sorehead remarks. And possibly the support tools are simply there and don’t actually need to be documented or commented deeply, contrary to what the standards for a code-base would require.

Nevertheless I am going to try to adapt my pipeline to those “official tools”, but so far I am not convinced.

Best regards,
DrTT

[anthony-nicholls]

@DrTT

• Documentation is a bit thin. I have found only one HTML page

You’re correct that the CMake API page is the primary piece of CMake documentation. If the included examples aren’t enough for your needs, you can also find a few third party templates users are using here Browse Products | JUCE Marketplace. Pamplejuce is one of the most popular templates.

@DrTT

• Obviously the idea is that one first builds the complete JUCE project (but you can also skip the examples or extras, if you want to, which IMHO should be the default).

I’m not sure I follow. You don’t need to build the complete JUCE project. The examples and extras are disabled by default.

You do, however, need to add JUCE to your project (using something like add_subdirectory() or find_package()) like any other CMake dependency (this is covered in the getting started section).

@DrTT

• Within that official pipeline there does not seem to be any concept of an intermediate library target with JUCE classes. I would like to have a static library containing some overall relevant JUCE classes without any plugin-specific configuration so far, but I only found the three functions juce_add_gui_app, juce_add_console_app and juce_add_plugin and a static library is none of those in my opinion: those describe end product targets.

JUCE isn’t really designed around sharing JUCE code via an intermediate static library that is then linked into multiple plugin or application targets (internally there is a static library shared between the plugin formats, but as JUCE handles this for you we can ensure the configuration is correctly managed). Sharing a user-defined static library between independently configured targets is much harder to make safe, as the JUCE configuration needs to remain consistent across all consumers.

If you do have code that relies on JUCE that you would like to share between multiple projects the most obvious way to handle this is to turn your common code into a JUCE module. You can read more about the JUCE module format here JUCE/docs/JUCE Module Format.md at b6d739c22b2d29792c3deae8c887769bf64340db · juce-framework/JUCE · GitHub. Once you have your module you need to call juce_add_module from CMake which will create a library (but not a static library) you can link each of your targets to.

Hope that helps.

[peteatjuce]

Hi @DrTT

In case of interest…

A long (long!) time ago, I generated my original project files using ProJucer. Ever since, I’ve had to maintain these files manually, especially as I have had so many things to accommodate including multi-target projects, Swift, WatchOS, tvOS etc. etc.

These days… here is “where I’m at”…

For Apple platforms, I build a private (platform specific) Framework with most of my code, and JUCE code, contained therein. This is used by my various Apple App variants, and by the Apple AUv3 variants. The App and AUv3 projects are quite thin wrappers around this core private framework. I maintain all my Apple projects by hand, as noted, because I’ve not been able to figure out to persuade CMake to build multi-target frameworks which reference unit tests (I use these extensively, and really rely on them). I was able to get everything else generating as I wanted them, within reason, using my own CMake files … with the exception of unit tests; so for me, that option remains on hold and perhaps never to be re-visited.

Windows: I have two separate static library projects; one building for DLL, one for non-DLL. The appropriate variant of these is linked-in to my VST3 (DLL) or App (non-DLL) app projects, which are very thin wrappers on top. As of a couple of months back, I started generating my Windows projects using CMake, just to try to make things easier to maintain; before, they were built-up of various sub-projects, as I kept hitting various linker size limit issues, as my codebase is so large.

My Windows CMake files are as simple as I could make them, but don’t use the JUCE files. Honestly, the reason for this is that I found it too hard to figure out how to use the JUCE ones!

@anthony-nicholls I can only give my honest opinion: the JUCE CMake documentation assumes more CMake expertise than I have (I’m OK at writing C++, but have never been a make system expert). Therefore, I had to find a solution that worked for me.

Android: I build my app projects a separate CMake projects, as per normal Android native apps

As a side note, I now build my Windows projects using Clang, as that allows me to use #embed, which I find very useful! And, this means that for all platforms, I use Clang C++ consistently.

Best wishes, Pete

[peteatjuce]

Reading what I’ve written above, and @DrTT 's comments, perhaps this approach of building JUCE and/or core app/plug-in code into a shared (or static) library, and then linking with App / Plugin “wrapper” is more common than is realised.

Anyhow, just my 2 cents.

Best wishes, Pete

[DrTT]

Hello,

I managed to get my build pipeline up and running again. The trick, once again, was to use Projucer for a demo project and check which files needed to be included (and this has changed between 8.0.10 and 8.0.11).

The missing files were e.g. the include_audio_processor_headless.xxx and the new include_juce_gui_basics_X.xxx files. Once they had been included in the projects, the intermediate static libraries could also be built as expected.

Thank you very much for all the tips and the discussion; I’m back on board .

Best regards,
DrTT