Support for 3rd party modules in Projucer

projucer
#1

Something I’ve been wanting to see for years now. Support for discovering and installing 3rd party JUCE modules. This might involve the use of libgit, along with a server-side database of published user modules (which could be listed on the JUCE website).

At the minimum, without any of the database stuff, a field to enter the address of a git repository and branch/tag and automatic/manual download of the module.

0 Likes

Components for Viewing Graphs/Plots/Charts/Diagrams
#2

Yes!! I’d be willing to help convert a few existing projects to get this up. The incentive for 3rd parties to modularise right now is just too low…

So, it sounds like the concept of a package manager? I’ll check out libgit soon which will answer my question probably.

0 Likes

#3

Yes, anything past just having a Git address starts to become a package manager. @jimc started “JPM” a couple of years back, I jumped in to help but went down a bit of a rabbit hole trying to copy how npm (Node Package Manager) does things (I’d been doing some NodeJS stuff at the time). I think we both got really busy, and then Jules changed the JUCE module format. Anyhow, important/useful features and things to consider in a JUCE package/module manager as I see it:

  • Initialise a new module based on given information (either CLI or GUI-based)
  • “Publish” a module so it is discoverable
  • Add a given module to a JUCE project

Personally I think this kind of thing is better done in a scripting language such as Python or Ruby (or NodeJS!), at least if it’s going to be a CLI tool. I imagine, there could be a CLI tool (maintained by the community) for creating and managing JUCE modules, as well as a simple field in Projucer with the Git location or unique name of the module, or it could all be done in Projucer…

0 Likes

#4

Given that some people choose not to use the Projucer I guess it would wise to have a simple CLI tool for the management of the modules. What are the key commands you think it would need to have - my experience with npm is extremely light - I guess I can imagine it being something like homebrew but on a per project basis.

For the updates required for the Projucer we can do that and make a pull request later (assuming it gets accepted).

0 Likes

#5

Come to think of it, particularly for those that don’t want/need the Projucer, we could potentially have the package/module manager pull and manage the JUCE modules too, this might save some people having to pull the entire JUCE repo containing the Projucer / tutorials / examples etc. as a submodule.

3 Likes

#6

Support for discovering and installing third-party JUCE modules in Projucer sounds like a nice idea. But writing a generic CLI tool that can handle all JUCE modules (including the ones from https://github.com/WeAreROLI/JUCE) sounds like a waste of time.

There are already several package managers for C/C++ (Buckaroo, Conda, Conan, Hunter, pacm, Spack, vcpkg, …). We don’t need another JUCE-flavored one. What we need is the JUCE (including third-party) modules to be usable as any C/C++ library, by most of these package managers.

This is why I’m working on adding CMake support to JUCE and third-party modules.

10 Likes

#7

Makes sense seeing that cmake is so widespread.

I was just thinking about Hunter in JUCE terms the other day as it’s something I use for the cross-platform framework I work on at my non-audio job. As much as I love open-source and find it works fine for tiny contexts, there are issues with configuring it that I really don’t like - lots of complex workarounds for legacy systems (particularly set-top boxes and consoles…). This is especially true for Curl and OpenSSL.

Things like Hunter make me a slight bit apprehensive with supporting JUCE-based modules, but then again my perspective is coloured by these wacky and over-complicated non-audio, non-mobile/desktop projects…

Suffice to say that I’m looking forward to seeing where this goes, and staying optimistic that a tidier solution will come up.

0 Likes

#8

Just playing devil’s advocate here: there’s a good chance of a large burden of maintenance that comes along with supporting all sorts of package managers.

Unless I’m missing something, there are two strategies to pick from:

  • selecting the most popular package manager and hit the ground running from there
  • find a way to unify interacting with the top package managers, somehow
1 Like

#9

I personally think JUCE could do with it’s own package manager, specifically for JUCE modules. I’m aware that there are other C++ package managers, and it would be great to have support for those too, but that is a different problem, which IMO should be tackled separately.

To boil down the problems that I think this feature request should attempt to solve:

  • Currently the only way to discover JUCE modules is by searching the web and/or Github.
  • There is a culture of expecting the library to provide everything, and if something doesn’t exist we write it ourselves.
  • There aren’t any tools available that help us manage JUCE modules.
  • All the above points make for little motivation for developers to provide and maintain public JUCE modules.

There are different approaches to package managers that I know of:

  • Composer for PHP, where developers submit packages for inclusion via a web interface
  • npm, RubyGems - fully featured CLI tool for publishing and managing dependencies (packages are stored in a database)
  • CocoaPods - combination of CLI tools for installing and Git for publishing (package locations are stored in a central Git repo)
1 Like

#10

If we think there is some milage in this idea I’m more than happy to help out - not so long ago I wrote a JUCE module validator in python, it could actually be used to read and retrieve and validate just about anything from a JUCE module and it was pretty easy to be honest.

I certainly want to agree with you, however my experience with package managers is minimal so I’m not completely sure if I’m just ignoring existing C++ package managers because I’m not familiar with them?

If we go ahead with the idea, I would vote that we adopt a similar system to this based on the three options you’ve proposed above.

I wonder if it might be possible that whatever system we went with, if it could be used to automatically generate packages for other package managers in the future? That way publishing a JUCE module via this mechanism could potentially allow the module to be published to multiple package managers? This would likely make it even more appealing for developers to make third party JUCE modules. However as I say my experience with package managers is minimal so it might just be wishful thinking?

1 Like

#11

Sounds good - I’ve not got much experience with Python but it seems like it would be a good choice.

I think using a Git repo as a module database is a bit clunky. If ROLI are up for hosting it, then a proper database to hold the module details would be preferable.

1 Like

#12

Pythons great, especially for the argument parsing. However for Windows users they would have to install python if they haven’t already, or we could look into generating executables using py2exe or pyinstaller, unfortunately I’ve not got experience with them and they seem a little clunky.

Alternatively we could use JUCE? I think JUCE is a little lacking on the argument parsing right now, but on several occasions I’ve gone to start a decent one written in JUCE and share it via a 3rd party module :wink:

I guess we could prototype with python and if required move over to something like JUCE at a later stage?

Fair enough, anything web based is not my strong point TBH. Couldn’t the database be stored in a git repo for now? I guess there’s risk around what changes getting pushed.

I was imagining that the database would hold the name of the module, the link to the Git repo, and an optional dir for which the module in placed in (such as modules for the JUCE repo). Would it need anything else?

I’m imagining that the package manager would use the info in the module declaration to pull in dependencies? One of the things that’s been crossing my mind as potentially one of the more difficult things to implement is how a 3rd party module declares which version(s) of a dependency it requires/supports. One thing we could consider if we wanted to add more information to these module declarations instead of placing information inside the BEGIN/END_JUCE_MODULE_DECLARATION section we could encourage 3rd party modules to also have an optional BEGIN/END_<PACKAGE_MANAGER_NAME>_MODULE_DECLARATION section. I’m still not sure exactly what you would put in there though, optional minimum / maximum versions referred to by tag/branches/refs?

Do you see any other sticking points that I might be missing? or did you imagine anything differently?

0 Likes

#13

I think it should also store author/vendor.

Yes, that’s what I was imagining too - leave the dependency management to Projucer. Dependency management is the hard problem in a package manager!

Unfortunately, it looks like the current JUCE module format does not support dependency version numbers, which is pretty important once you start dealing with third party modules. It could be adapted using e.g. dependency_name@git_tag/branch e.g.

     BEGIN_JUCE_MODULE_DECLARATION

      ID:               my_cool_module
      vendor:           acme
      version:          1.0.0
      name:             My Cool Module
      description:      Some cool stuff
      website:          http://www.acme.com
      license:          GPL

      dependencies:     juce_audio_basics, some_other_module@v1.0

     END_JUCE_MODULE_DECLARATION

Where if there is no branch/tag defined it defaults to master.

1 Like

#14

I really like the idea of improving 3rd party module support but rather than going down the full package manager route, wouldn’t a logical first step simply be to provide a place for people to view modules?

Actually finding modules is the real hurdle here. Adding a Git submodule is relatively straightforward and I imagine most modules at the moment don’t have complex dependencies.

As long as you can see what a module depends on (like in the PJ), and have a place to go and get that module, I think at least for now users can physical get the module themselves.

This may be of interest to watch: https://www.youtube.com/watch?v=sBP17HQAQjk
The main take away point I had from this was to make sure your libraries only work with the latest. Trying to support multiple versions of modules is basically an impossible task, especially more the individuals and small companies in this ecosystem.

Just my 2p…

7 Likes

#15

Don’t bother, use this one:

If there are things missing, let me know, I still have access to maintain it…

2 Likes

#16

Breaking news is that the C++ committee just approved the module proposal for inclusion in C++20…

We weren’t really expecting this to happen yet, but it means that by next year there’ll be a standard format for modularising C++ code, with wide compiler support. So in the light of that, we’ll probably need to think about migrating our juce modules over to that new format. Quite an exciting prospect as it’s likely that it’ll open up the door to all kinds of useful tools, package managers, etc.

http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1103r2.pdf

8 Likes

#17

I think it would be preferable if we could avoid having to use Git submodules. They’re not difficult to work with in principle, but they have to be updated manually when the submodule repo gets updated. It would be so much nicer, for example, if from a single .jucer project file, all the dependencies of a project could be pulled down in one go, fetching the latest versions if necessary.

An MVP might be a combination of a list of modules on a web page, along with basic Projucer support for installing modules from a Git address.

0 Likes

#18

Yes, very exciting proposal, and great that you are already considering how it would integrate with juce modules.

0 Likes

#19

Yeah, but isn’t that the whole point? If a project is building then you know it will continue to build?

Otherwise, you’re in a situation where you have a “global” module that dozens of projects are referring to, that module makes an API breaking change, you update it and suddenly all of your other projects are broken and now unshippable.

That’s great for mature projects with stable APIs or if you’re Google and can write a tool to update your billion lines of code that rely on that module but for most people it’s just an inconvenience. In reality, you only need to update your modules if there’s a change/fix/feature you need.

Please please please think about all of this in terms of CI where you might have a new VM every time you build your project or even if you have multiple projects sharing the same machine, possibly building at the same time. What if you have two different projects building concurrently with each referring to a different version of the “global module” you’re using?

There’s so much to this module business I’d rather just have an online directory I can browse. If each module has clear instructions on how to get it and add it to a project this should be straight forward.

Git handles so much in terms of conflicts and changes etc. that I can’t think of a better tool for simply downloading modules stored online. Having to manually update snapshots of repos that are copied in to your own code feels like a big step backwards.

2 Likes

#20

As cool as this is, keep in mind that adopting this feature or using it as the foundation would put a hard-stop on the platforms you can support, limiting you to what you have going on now.

1 Like