Flutter with JUCE - my solution approaching mobile accessibility

Dear JUCE community,

If you are interested in pairing JUCE with Flutter, I’d like to share with you my stub project (including some documentation on how it was done).

I’d be glad to come to exchange with you about it. Or about any Flutter, React Native, Xamarin, … approaches that you are familiar with connecting to JUCE. Or about JUCE and accessibility.

As much as I love JUCE and its GUI abilities, as much I learned to customize components and look and feel, unfortunately JUCE still has no approach to accessibility, and might never have, without providing native widgets of all platforms or spending immeasurable effort into various accessibility APIs.

Since I’m currently engaged in a project developing an accessible mobile app, I looked for existing solutions and I believe Flutter is an incredible and promising cross-platform UI framework.

Any feedback or suggestions for improvement is welcome,
Thank you and stay jucy

7 Likes

This looks exciting!

Would you mind trying to wrap most of the sources/configurations in a JUCE module? That would make it much easier to integrate into an existing project, and you could customize all the platform specific stuff internally from within your module instead of reconfiguring it for each project.

1 Like

I will consider this, good idea…

I wouldn’t be so sure of that…

1 Like

Cool - That would be so awesome! :raised_hands:

I’m a little out of my depth here, but what do you guys think about the oncoming WASM hype train?

Wouldn’t that solve all our cross compatibility woes?

I have no experience nor knowledge on WebAudio or WebAssembly, but I‘ve been really impressed by the SOUL demos. That looks very promising indeed! With ARIA web UIs can be well accessible. So far I‘m still struggling if audio web apps can already compete against native apps and would be very curious to know what you think or experienced so far:

  • Is a WebAudio software as stable and performant (cpu/battery) as native apps dsp/audio processing?
  • Could it continue playing in the background as a player when switching off the phones display?
  • can a WebApp or hybrid app link 3rd party precompiled libraries (static or dynamic)?

I think you’re missing the point a little bit.

It seems the cliche of bad naming things in Computer Science has struck again. WebAssembly is not really Assembly, and it doesn’t have to be Web!

What makes WASM special is that its an intermediate level byte code (like LLVM) written to an open spec that for the first time many of the big players all agree on. Web Audio is just a browser spec. Definitely Web-y and not necessarily related at all. WebAssembly has taken on this trend lately of prefixing everything with Web, but this imo is a mistake.

I don’t know where WASM is headed, but all that would be needed is something that can interpret the WASM code and interface with the OS directly. Now you’ll get all the speed benefits advertised by the browsers of WASM, but without the weight of a browser itself.

2 Likes

Cool to have another juce as backend example!

Another wonder about such approaches,
So far I’ve yet to see any full cross-platform framework that really makes it possible to write for both desktop and mobile.

Flutter does seem to promise that (https://flutter.dev/desktop)
So I wonder if you even tried.

Since I really avoided React Native or similar approaches since they make it only useful for mobile.

To have a full cross-platform UI framework you currently got:

Qt and JUCE.

All other alternatives seems to keep you in a specific scope.

2 Likes

Thanx for clearing things up - I was completely unaware of what a game changer that could be! Sounds like you are already did some work with WebAssembly? A very interesting field indeed. Still, as for today, some boilerplate work will remain if you have to deal with interfacing closed-source platform-dependent-apis or 3rd party libs if the manufacturer have not precompiled them for WASM machine code. But it could indeed be a big hit for cross-platform as well as web solutions, especially since it’s doing native performance.

Thank you. JUCE still is my favorite cross-platform framework for reaching all major platforms - and I really would love if they could make it screen-reader-able for mobile+desktop usage. I did all my GUI work with JUCE in the last 5 years and the JUCE Team did a very good job in the meantime, e.g. in supporting Android.

I am now at a point where I am forced (which is a good thing!) to finally stop ignoring screenreader users. I even happen to know some non-vision-users and would always have loved to support them with JUCE based software, just did not have the time or resources to solve that.
At the moment I need an accessible GUI solution for Android and iOS (desktops are currently out of my scope, maybe again in future projects). I’m also looking for mobile app developers joining my team, so I have to think about what skills I can expect. Most people around I talked to (business mentors and developer teams) go with ReactNative, so I startet looking if that could be the best solution. I compared some frameworks and was really impressed by Flutter and its huge potential - especially since it’s addressing mobile+web+desktop - even if the latter are still in beta/alpha stages. It’s growing very fast and has many supporters. Since I found no projects online trying to run JUCE with Flutter I wanted to try it out on my own, if Flutter could be a working solution at all to take into account.

Yes, QT might be another very well fitting solution, do you have some experience in mobile UI with V-Play/Felgo?

This seems super interesting, I know nothing about Flutter so I tried out the install demo on flutter.dev and was impressed with the hot-reload working on both iOS or Android device hooked to Android studio. As I’m working solo on a mobile app, this seems like a promising path!

I have a few questions, forgive me if they sound somewhat flutter-noob :

Trying out your stub project, I managed to compile and run on Android, from Android Studio, but couldn’t run on iOS device from this context (as in the getting started demo). Not sure what I’m missing?

On the iOS side, I generated the xcode project with the pod install, opened the .xcworkspace but the compilation fails while linking with the following:
ld: library not found for -lPods-Jucy Fluttering - App

sounds familiar at all?

Also, I’m curious, have you tried outputting audio with this architecture? Obviously that’s the point, but any example is welcome or war story is good to take. :wink:

Thank you

1 Like

Hi Bulien,

some days ago I updated to Projucer 6.0.7 and got exactly the same ld error (library not found for -lPods-Jucy Fluttering - App).
The problem is, that Projucer now changed the output of the iOS exporter config Binary Location field. Projucer now interprets the path to be relative by default when writing the XCode Project. Simply adding a / character at the beginning helps. I now pushed that change to the git repository.

For your question about audio: a simple way would be to initializing a juce::AudioDeviceManager in the main application class (JucyFlutteringJuceApplication). You then could init a juce::AudioProcessorPlayer and register it as audio callback to your audio device manager via addAudioCallback(...), to play back some synth or even audio processor graph…

1 Like

Thanks, it worked!
Yes I’ll try adding audio, I was just curious if you’d hit any issues.

1 Like

You’re welcome. Up to now, I did not do much with it, just implemented a very simple MP3 asset file playback. Worked like a charm. I believe / hope the issues that might come up would mostly be the same as with any JUCE iOS or Android project. The Flutter part should be running completely isolated from the JUCE “audio engine” which should not depend on what Flutter is doing. But I would be very interested if you report your experience here… Good luck! :grinning:

1 Like

Just wanted to drop in here to say I see a lot of potential with flutter as a good cross platform frontend audio projects in general. I’ve helped a with with IPlug2 last year looking for a way to get my tuner application working across all major platforms, but in the end switched to flutter for the front end (from Oli’s suggestion). My application has C++ backend (with RtAudio/web audio as the transport right now). I’m using FFI and some borrowed StandardMethodCodec code to communicate back and forth with very low overhead. Very impressed with the performance (skia is amazing) and over time it is going to be hard to beat the UI development support that Flutter offers vs JUCE/Qt (I have hot reload working for all the frontend dev–that was a big shift for me as someone developing C++ frontend code for 20 years)

On the backend, I’m also able to debug/setup through the C++ code in visual studio by creating a dll and debugging with the flutter application as the startup app. The only gripe I have is the two language dichotomy (C++/dart) makes life harder – but for audio applications the frontend/backend should be very well defined (especially if you are completely doing a lock free message passing design)

I haven’t looked at what it would take to get flutter embedded as a audio plugin in a standard DAW, but it looks feasible. Flutter has an embedding layer which I am starting to use on my raspberry pi.

My 2 cents – Joe

1 Like

Thank you @broms. Based on your experience, do you think it’s possible to use Flutter for only rendering specific components of a Juce app? I have an existing app where I’m perfectly happy to use Juce components for certain screens (where I need tight audio/UI interaction), but I wouldn’t mind outsourcing the development to Flutter developers for the other screens (assets browser, navigation, login forms etc.).

Hi @audiooffler very interesting work, it has helped me at the very least to call functions in juce from flutter. I was wondering if you, by any chance, know why it’s throwing “Unhandled Exception: Failed to initialize Dart API”
I’m trying to figure out how to make the async messages work.
Many thanks.

Of course I find the issue after asking on in the internet :wink:
Had to increase the DART_API_DL_MAJOR_VERSION to 2 in dart_version.h

Edit: I guess I will take the chance to ask if anyone has had luck using JUCE as a library and letting flutter make all the boilerplate code for cross platform purposes?

@audiooffler great work on combining Flutter UI with JUCE backend! It’s amazing how you made it work so simply.

This is great as a proof of concept for mobile. Has anyone tried doing something similar for macOS and windows? Just curious if it’s worth pursuing that path as I’m planning to implement a vst but wondering if it’s even feasible to do this on a desktop? If not then I’d love some tips or suggestions on where to start exploring implementing this integration to reach a potential proof of concept for a desktop vst with flutter ui and juce as backend