Introducing Blueprint: Build native JUCE interfaces with React.js


Check it out! Do not forget this had been done quick and dirty with excitement and only a proof of concept :slight_smile:
There is also a version of simple gain plugin there which I used naked-paper.js for a couple of things as a proof of concept.
When I have more time I will try to understand the source code of paper.js deeper so maybe I can adapt it much better.

6 Likes

Diving down this exact same rabbit hole at the moment myself. Have tried two.js to see if it works without the dom but no dice.

Perhaps time to investigate your naked-paper.js approach @alisomay! I’m looking for a js library I can incorporate to draw simple shapes and then convert said shapes to an SVG path string which could be passed either to Blueprint’s existing “border-path” view property or to a new component type which uses Drawable::parseSVGPath and the like in it’s paint routine to draw a collection of Path objects passed.

I’m wondering if this will be cheaper and more flexible the the current ImageView approach which parses a full svg doc and tears down the associated Drawable each time.

@ncthom has there been anymore investigation around this area recently?

I did find https://github.com/andreaferretti/paths-js which I was able to pull in but the drawing support is pretty rudimentary.

I haven’t done much more on this myself, but it is definitely something I’m still interested in. Similarly, I never found a suitably powerful drawing library that operated without the DOM, which is a bummer.

It’s definitely a feasible proposal: a drawing library that represents “draw commands” in its own internal format and then has multiple export formats, such as SVG string. That same library could take a Canvas 2D Context-like object and call its draw commands, which would provide rendering to a web canvas and also we could imagine writing a canvas-like object that just calls through to juce’s graphics routines, and pass that object into the same API. That way you would skip the step of encoding to SVG string and then decoding from SVG string.

Anyway, I think that Canvas-like object with calls through to graphics routines is a good first step, because then you could just draw using that either in web or in blueprint. For example:

class MyCoolKnob extends React.Component {
  draw(ctx) {
    // The DOM Canvas 2D Context API
    ctx.moveTo(50, 50);
    ctx.lineTo(100, 100);
    ctx.stroke();
  }

  render() {
    return (
      <Canvas onDraw={this.draw} />
    );
  }
}

You could imagine writing such a <Canvas> object for the browser that internally renders a <canvas>, captures its drawing context, and then calls this onDraw method in every requestAnimationFrame callback and passes the drawing context to the onDraw callback. Then imagine writing a custom Blueprint Canvas component that, in every paint(Graphics& g) creates a native object in the js engine with a bunch of native methods that just call back to that g, using API that mimics the Context 2D API. Portable rendering :slight_smile:

I think this is probably the more straightforward path to introducing a really valuable draw hook into blueprint.

1 Like

Thanks for the awesome advice @ncthom !

Yeah this certainly sounds like the way to go.

So I can wrap my head around how this might work with ReactApplicationRoot::registerNativeMethod, i.e. I could stash the reference to the Graphics object passed to paint() and bind equivalents for the various CanvasRenderingContext2D API calls.

i.e. ctx.lineTo maps to juce::Path::lineTo which is rendered by the Graphics object held in the stash.

To trigger the Canvas components onDraw callback I guess we would essentially call EcmaScriptEngine::invoke(“drawCanvas”) from paint() after registering the native object/method bindings for the graphics object? i.e. to actually “call-back” to the Graphics instance and render.

I’m not even close to fluent with duktape yet. I’m assuming I can register an actual object with methods rather than calling registerNativeMethod for every Graphics call. I think I can head down the right track with EcmascriptEngine::registerNativeProperty/duktape_push_object and workout how to create an object with the necessary methods via juce::Var and co.

Failing that I think I can see how calling on a “Graphics” target would work based on looking over the approach used for the BlueprintNative calls.

@ncthom if there’s any chance you could give a quick example of registering an object with methods via the EcmaScriptEngine and calling said object in js that would be ruddy marvellous. I can sort of see how some of this works based on the pushVarToDukStack helper in EcmascriptEngine.cpp.

( I realise I’ve promised various pull requests to land on github soon (They are coming!) but I’d be happy to have a crack at a basic implementation of this, not totally sure what the performance implications of registering native methods/objects inside every call to paint() will be like … ).

Yup, you’re totally on point, and the callback bit is a little rough around the edges right now: https://github.com/nick-thompson/blueprint/issues/32 :smiley:

I’ll carve out some time this week to take a quick stab at that issue and follow it up with a little Canvas example! I’ll try to remember to write back here when I do

2 Likes

Awesome sounds great Nick. Let me know if I can assist with anything.

Cheers

@ncthom This gives me an idea.

I’m currently writing a React based slider. It’s a decent amount of work for me to write logic to handle gestures and convert said gestures to values for various slider types (rotary, two-value, linear-vertical, linear-horizontal etc.). Not to mention skew logic, ranges etc.

I’m wondering if another paradigm could emerge where you could override LookAndFeel methods in a class derived from a stock juce component. Users could then have the js context do the drawing whilst keeping the tried and tested juce components in charge of functionality?

i.e. for a slider I might have something like the following:

// A Look and feel class which dispatches look and feel overrides to js
class ReactBasedSliderLookAndFeel : public juce::LookAndFeel_V4
{
public:
    //TODO: Obviously we wouldn't want to provide the full EcmascriptEngine
    //      ref here. We want some intermediary class to delegate registering native functions
    ReactBasedSliderLookAndFeel(blueprint::EcmascriptEngine &engine)
        : engine(engine)
    {
        
    }
    
    void drawRotarySlider (  juce::Graphics &g
                           , int x
                           , int y
                           , int width
                           , int height
                           , float sliderPosProportional
                           , float rotaryStartAngle
                           , float rotaryEndAngle
                           , juce::Slider &slider) override
    {
        // This block of code is entirely wrong and exists just to demonstrate a point.
        // We would require a way via the EcmaScriptEngine class to register a DynamicObject
        // so that functions can be called upon it. This will involve stashing pointer to the DynamicObject
        // etc. etc.
        juce::DynamicObject jsGraphicsContext;
        
        jsGraphicsContext.setMethod("drawRect", [&] (int xPos, int yPos, int rectWidth, int rectHeight) mutable
        {
            g.drawRect(xPos, yPos, rectWidth, rectHeight);
        });
        
        
        engine.registerNativeProperty("GraphicsContext", juce::var(&jsGraphicsContext));
        
        // Somehow invoke drawRotary slider in js which will use the previously
        // registered GraphicsContext object?
        engine.invoke( "drawRotarySlider"
                      , x
                      , y
                      , width
                      , height
                      , sliderPosProportional
                      , rotaryStartAngle
                      , rotaryEndAngle
                      , &slider);
        
        //TODO: Deregister the GraphicsContext property here?
    }
private:
    blueprint::EcmascriptEngine &engine;
};

// Where CONTROLLER here is a SliderListener (Probably the Component class owning the ReactApplictionRoot instance) which uses the ID of the slider
// by doing something akin to the following in it's listener:
//
//   ReactBasedSlider *slider = dyanmic_cast<ReactBasedSlider>
//   if (slider.id == "GainSlider")
       // Do gain stuff
template<typename CONTROLLER>
class ReactBasedSlider : public blueprint::View
{
public:
    ReactBasedSlider()
            : blueprint::View()
    {
        addAndMakeVisible(slider);
    }

    virtual ~ReactBasedSlider() = default;

    void parentHierarchyChanged() override
    {
        if (blueprint::ReactApplicationRoot *root = findParentComponentOfClass<blueprint::ReactApplicationRoot>())
        {
            lookAndFeel = std::make_unique<ReactBasedSliderLookAndFeel>(root->engine);
            slider.setLookAndFeel(lookAndFeel.get());
        }

        if (CONTROLLER *rootController = findParentComponentOfClass<CONTROLLER>())
        {
            slider.addListener(rootController);
        }
    }

    void paint (juce::Graphics& g) override
    {
        blueprint::View::paint(g);
    }

    void resized() override
    {
        blueprint::View::resized();
        slider.setBounds(getLocalBounds());
    }

private:
    juce::Slider slider;

    //TODO: Better way to do this than having separate instance for each slider?
    std::unique_ptr<ReactBasedSliderLookAndFeel> lookAndFeel;
};

This way I don’t need to rewrite slider logic that already exists in the juce codebase but I still get the benefits of hot reloads, react based drawing/styling and flexbox layout. This seems like it could be a powerful way to add a set of reusable components to blueprint for people to get going with quickly.

Things like juce::Slider’s range and skew factor should be settable via props from js. blueprint::View::setProperty is virtual so looks like we could override this in ReactBasedSlider (or BlueprintSlider if the name suits better :-)) and dispatch property changes to the relevant juce::Slider methods. i.e. ‘skew’ prop changes triggers juce::Slider::setSkewFactor().

This does open up interesting discussions around how you would then bind callbacks to a ReactBasedSlider. i.e. add a SliderListener in some way either using a template or constructor arg in the ViewFactory. This listener could use a dynamic cast to ReactBasedSlider and read the slider’s ID which has been set via a js prop.

So I could register my ReactBasedSlider component using your excellent ShadowViewMagic:

appRoot.RegisterViewType("ReactBasedSlider", [] () {
   // Slider with listener typed on owning class of appRoot
   using ReactBasedSliderType = ReactBasedSlider<OwningControllerClass>;
   
   auto v   = make_unique<ReactBasedSliderType>();
   auto sv = make_unique<blueprint::ShadowView>(v.get()); 

   return ViewPair(move(v), move (sv));
});

Then users simply have some js function for overriding drawRotarySlider and friends pulled in at the AppRoot level which uses the Graphics like object registered by the C++ code to do it’s magic.

I’ve no idea if this would all work. I might have a crack at an approach like this tomorrow just to see how it shapes up. It may be that this is simply a design pattern that works better for me and my current project and doesn’t at all align with your plans for the mighty Blueprint!

I think in theory this approach works orthogonally to the current examples in which slider components are written from scratch handling mouse events etc. etc. In my case it could mean less js function binding to handle UI actions/calbacks as I’m working outside of a generic plugin project and am not using a ValueTree backed data model etc.

If this approach works for me I’d happily drop you some contributions for components using this model. I needs sliders, buttons and comboboxes etc and probably don’t have time two write them all from scratch in React.

Again feel free to shout if I can help with the DynamicObject binding stuff or anything.

Well the basics appears to work. I can register my ReactBasedSlider and my SliderListenerCallbacks are all working.

I’ll attempt a little hacky version of the look and feel override just using registerNativeFunction and stashing the Graphics reference in context and let you know how that goes.

I think these two component approaches might be able to live side by side …

@jmarler exactly! You’re spot on, this approach would surely work and indeed it’ll save you some time not having to reimplement the slider internals in js.

The drawing bit in your example is very close, and right now it’s a little odd to pass these kind of temporary callbacks from native into js. I made some strides on the Canvas idea from above yesterday, and I’m going to pick that back up tonight and hopefully finish up a kind of “MVP” that will show you how to do that drawing callback binding stuff.

2 Likes

You the man @ncthom.

The framework really is super powerful when you realise you can take this approach to reuse existing components in your code base. Just augment them with the additional props you need so you can hot reload color changes and the like.

Took a moment to workout how to actually register a pre-existing component. I missed the call to React.CreateElement in the relevant js file. Would be happy to contribute an example doc/README for this if it would be of use.

1 Like

I agree :smiley: And yes please! That should definitely be a page on the wiki (like GitHub - JoshMarler/react-juce: Write cross-platform native apps with React.js and JUCE) but I clearly never got around to it heh.

In other news, I got the first steps for a <Canvas> view up! It took some nudging with the EcmascriptEngine/duktape interface to freely pass functions around, but I think it ended up really neat actually. Edited the GainPlugin example with a silly little example: https://gfycat.com/indolentfrankcockatiel

Basically a little animated line flickering underneath the gain meters. The JavaScript:

function animatedDraw(ctx) {
 let now = (Date.now() / 10);
 let width = now % 100;

 ctx.fillRect(0, 0, width, 2);
}

<Canvas {...styles.canvas} animate={true} onDraw={animatedDraw} />

And the CanvasView is basically:

    void paint (juce::Graphics& g) override
    {
        View::paint(g);

        if (props.contains("onDraw") && props["onDraw"].isMethod())
        {
            auto* jsContext = new juce::DynamicObject();
            juce::Path jsPath;

            jsContext->setProperty("fillRect", juce::var::NativeFunction {
                [&g, &jsPath](const juce::var::NativeFunctionArgs& args) -> juce::var {
                    jassert(args.numArguments == 4);

                    const int x = args.arguments[0];
                    const int y = args.arguments[1];
                    const int width = args.arguments[2];
                    const int height = args.arguments[3];

                    g.setColour(juce::Colours::blue);
                    g.fillRect(x, y, width, height);

                    return juce::var();
                }
            });

            std::vector<juce::var> jsArgs {{jsContext}};
            juce::var::NativeFunctionArgs nfargs (juce::var(), jsArgs.data(), jsArgs.size());
            std::invoke(props["onDraw"].getNativeFunction(), nfargs);
        }
    }

Full version here: https://github.com/nick-thompson/blueprint/blob/master/blueprint/core/blueprint_CanvasView.h. You can see that obviously I’ve only implemented one method so far, but the idea would be to enumerate much of the API given here CanvasRenderingContext2D - Web APIs | MDN with implementations written in JUCE. The closer we can keep to parity with that API, the easier it would be to render a Blueprint app in a browser :wink:

Of course you can use that same idea in tandem with your code above, just basically copy/paste the construction of that jsContext with the paint routines into your LookAndFeel and kick it off. Let me know what you think!

2 Likes

Ahh man this is awesome.

I’ve got the work refactoring a HotReloader, ReactApplicationRoot and a ViewManager coming over to you shortly.

I’d be happy to make your above example more generic and and implement a decent chunk more of the Canvas API.

Would be nice to have some simple builder for that js/canvas context object which registers all the methods. Users can just call a single factory function for buildCanvasContext or whatever.

It will be interesting to profile this and see what kind of overhead is introduced by registering a full set of canvas API methods at the start of the paint() call. I shall run some tests!

It might even be nice to provide some LookAndFeel derivations which implement the thing I’m after as an example. This is all just boiler plate so I’ll crack on for the next couple of days and then drop you some pull requests over the weekend.

This is all getting very exciting!

1 Like

I’ve been following this and love the way it’s going but I have a couple of questions about future cross-platform support. It’s my understanding that if you implement the CanvasRenderingContext2D API calling in to JUCE Graphics methods that means you can draw arbitrary code in the JS side of things. This gets you a declarative model and hot-reloading which is cool.

However, if you are relying on juce::Slider for the mouse interaction and value scaling etc, if this ever runs on a different platform (e.g. a web browser) that slider implementation won’t work as it won’t have all the juce::Slider methods and mouse handling. This seems like a bit of a step backwards.

Is the idea that when running on the web you’d need to provide your own Slider that does all the mouse handling and value scaling etc? This would make this method just a quick way to use juce sliders before re-implementing all that logic in the long run anyway? (Which is of course a very valid approach).

1 Like

It’s a fair comment. In my case that approach served to illustrate that Blueprint can be multi-paradigm and classes like ReactBasedSlider allow me to avoid writing a substantial amount of code whilst I’m iterating a design or on a tight deadline etc. So yeah exactly that @dave96, it’s a quick way to get up and running or add React/Blueprint based components to an already existing codebase.

As I see it there are 3 main ways to use blueprint components:

  1. A pure Javascript based implementation. i.e. a Slider.js file that handles all of painting, gesture handling, value change callbacks and more.

  2. A ShadowView based approach where you make one of your existing components available in React and painting is handled in C++. React is still in control of layout and you could augment your existing paint code by reading simple properties declared on the react component declaration. i.e.

    <MyPreExistingComponentDrawnInCpp background-color='#ff0000' />

  3. The above ReactBasedSlider example in which React only deals with painting and layout and C++ code deals with the “Controller” element of the component. I.e. adds listeners, sets ranges etc.

I think these three approaches would make valuable examples in the repo docs. I wouldn’t necessarily suggest that they become baked into Blueprint.

Whether Blueprint should include a library of stock components is something @ncthom could probably tell us. It seems valuable. Would there be benefit in including both approaches? i.e. a NativeSlider.js and NativeSlider.h (approach 3) and a Slider.js (approach 1). I doubt all users will want to (or can) sit and write full slider, button, combo-box (and so on) implementations in JS for their project.

Of course if Blueprint or some complimentary component library implemented JS equivalents for all the existing juce components, allowing users to specialise/style them, approach 3 wouldn’t be necessary (though there may be performance benefits to the “native” wrapper approach in resource constrained environments with less js function binding/interpreting etc). I suspect a full JS component lib is a way off at the moment though.

Maybe examples in the Blueprint docs are better with Blueprint just left as the underlying framework that makes this all possible, without being opinionated on how components are constructed. It would be very easy for community driven component libraries to emerge. Perhaps some projects (possibly the majority?) won’t care that their UI can’t be rendered in a browser, they just want a fast UI development workflow, quick and easy layout and maybe a chance to farm out some UI work to designers/JS gurus.

1 Like

Yes, agreed. I understand a full web-based view is a bit out of scope for what blueprint was originally intended for. I just want to make sure fully understand where the lines are so I can decide how to structure things for our existing and future use cases.

2 Likes

Yep makes sense. Exciting stuff!

I’d love that! Thanks :smiley: And yea the single factory function makes 100% sense. I just pushed one more thing to the current implementation to support writable properties (only fillStyle is implemented so far), which will hopefully make extending the API easier: Add fillstyle canvas support · JoshMarler/react-juce@44d4e6d · GitHub

Oooo yeah that’s neat. That’s a much nicer style.

I’ll finish off the HotReload and friends refactor and then crack on with the Canvas API implementation.

About to have a busy weekend …

1 Like

@dave96 @jmarler I appreciate the conversation here, and it’s something I’ve thought about a lot. I think it can get tricky because I imagine different people will want blueprint for different things– some people will want a fast dev approach for their native apps, some will want maximum portability from native to web, etc.

So far, the only native views currently provided in Blueprint are the ones for which I can see a straightforward path to implement a web-based shim, which is to say that if you write an app using only the standard native views, you should (in the future) be able to just pop the same bundle into a web app and run it (well… almost :slight_smile: you’d have to write a couple lines of code). I’m careful to say “in the future” because this is definitely not implemented yet, but it has informed my decision making about what standard components to offer.

At the same time, I totally take @jmarler’s point that wrapping a core set of JUCE components like the juce::Slider will get people up and running super fast, which is sort of its own value proposition and which I think would help get people using the lib in the first place.

I think we can have both, we just have to be careful about it. For example, the blueprint core module could carry only the standard native views that port easily to web, and we could ship a second “blueprint extras” module which wraps some juce components. For those people who want that, just add the module to your project and render a <WrappedJuceSlider> . And for the people who don’t want it, it’s not intrusive in any way. I think this is also generally supported by just how flexible React itself is. For example, let’s say that I’ve added the “blueprint extras” module and I want to use the wrapped juce slider, but I also want portability to web. I could do this:

function MyPortableSlider (props) => {
  if (runningInWeb) {
    return (<MyWebSlider {...props} />);
  }

  return (<WrappedJuceSlider {...props} />);
}

Of course in this example it’s up to me, the user, to decide exactly how MyWebSlider imitates the behavior of the wrapped juce slider (or doesn’t), but the point here is that even if you use a wrapped core component you’re not necessarily “locked in”

This.

Yeah that’s exactly how I view it, make these offered components an additional module/repo. I was even going to suggest an if (window) return <Slider /> approach heh. Some condition to switch the render based on environment.

I’ve already got some juce wrappers up and running @ncthom so feel free to drop me a message about an “extras” module or similar and I can contribute some stuff.

1 Like