BR / FR: mouseEnter / mouseExit events on Apple Vision Pro

MacOSX and iOS
1

Now that Apple has released the Apple Vision Pro, and devs can opt in for users to run their (JUCE) apps on it as iPhone/iPad compatible apps, it seems there are a couple of things that are currently missing in JUCE for a good experience:

  • mouseEnter/mouseExit callbacks are not triggered. This makes the gaze-based navigation on Vision Pro quite cumbersome, because there is no way (visual feedback) to let the user know that he/she is looking at the right button/sider/popup menu item etc. Is it possible to get mouseEnter and mouseExit events based on gaze so we can provide visual feedback?

  • Audio works, but it seems impossible to disable the Vision Pro Spatial Audio feature for JUCE apps. Some other apps (but not all) do have the option to disable Spatial Audio, which I think is appropriate for audio apps…

2

For testing purposes you can disable Spacial Audio globally - Settings>Bluetooth>turn off Personalized Spatial Audio, or by long-pressing the volume slider in control center. To do it programmatically, I think there is an Entitlement for modifying this buried somewhere deep in the Vision Pro SDK.

3

Thanks - however ‘Turn off Personalised Spatial Audio’ does not turn off Spatial Audio itself, only the personalization feature of it that you acquire after scanning your head and ears with an iPhone. There’s still Spatial Audio active when that toggle is switched ‘off’, it is just using universal instead of personalised rendering.

4

Ah, roger that (my experience with AVP is limited to a few hours of testing over the past few weeks) - but I’m still pretty sure there is a missing Entitlement in the Info.plist for the project which will disable Spatial Audio for the app (JUCE App?) universally. Just haven’t found it in the SDK docs yet …

5

no worries; I’ll keep digging! Thanks for the plist hint.

6

Hope you’ll share the details when you find it! I’m also evaluating AVP as a target for some of my projects, and am keeping a keen eye on these kinds of hiccups …

7

Sure - will report what I find. The lack of mouseEnter/Exit events is the biggest issue with JUCE apps at the moment I think; the navigation is quite challenging without visual feedback. Did you encounter this as well?

8

Ok, turns out mouseEnter / mouseExit events aren’t triggered on VisionOS due to privacy restrictions - apps don’t have access to eye gaze information.

It seems that one has to support the hoverEffect to allow UI elements to provide visual feedback; AFAIK this is currently not supported in JUCE.

I’m hoping the JUCE dev team (@reuk , @anthony-nicholls) is aware of this and can provide gaze-based hover functionality such that JUCE (8?) iPhone/iPad apps can be used on VisionPro devices with gaze-based visual feedback.

Also a warning for other JUCE devs that are tempted to click on the ‘make available for VisionOS’ button in Appstoreconnect: your user experience may not be great if you allow your iOS/iPadOS apps to run on VisionPro without further changes.

1 Like
9

I haven’t investigated this in any depth, but my feeling is that supporting hover functionality in the way that is required by Vision Pro will require substantial changes to the framework. Therefore, the value of this work (and ongoing maintenance cost) needs to be weighed against all the other features and bugfixes in our backlog. I’m sure we’ll address this if demand for this feature starts to outweigh the need for other work in our backlog, but I don’t expect this to happen in the short term.

1 Like
10

This seems like you could do it in the XCode rig yourself, with relative ease, given a careful review of the structure in place to wrap the NSView in a JUCE standalone app.

After all, isn’t Hover just another kind of Focus event coming from the OS… perhaps extend Component for VisionOS devices, and UIBehaviour, so that hover gets turned into a JUCE Accessibility ‘focus’ event …?

Edit: after a small review, it seems like the only issue would be the use of the OpenGLES.Framework - but otherwise, a ‘naive port’ of a JUCE App to VisionOS seems relatively approachable …

11

After reading a bit more on the topic it seems a bit more complicated.

The highlights of buttons etc on native Apple Vision Pro apps are done by the OS without triggering any mouse events or alike in the app itself, because gaze information is protected due to privacy restrictions. I guess this means that one would need a bottom up approach where the app tells the OS which regions can/should be highlighted based on gaze, but it won’t trigger any callbacks.

12

For the sake of archival and information sharing: you can make the JUCE UI respond to gaze with a couple of lines of objective c code by adding the hoverStyle to the JUCE native UIView:


#if __APPLE__
#include <TargetConditionals.h>
#include <AvailabilityMacros.h>

#import <Foundation/Foundation.h>
#import <UIKit/UIKit.h>
#endif
#if TARGET_OS_IOS || TARGET_OS_VISION

    juce::ComponentPeer* p = c.getPeer();
    if (p != nullptr)
    {
        UIView* h = (UIView*)(p->getNativeHandle());
        if (h != nullptr)
        {
            h.hoverStyle = [UIHoverStyle automaticStyle];
        }
    }

#endif

This doesn’t seem to work for individual components as they don’t have individual UIViews as far as I know, but I suppose one could create a plugin consisting of UIViews, and each UIView has the hoverEffect enabled…

13

If you’re already spending time on that, I’d suggest you’ll review how JUCE Accessibility integration works as it might have some core that you can hack/patch.

1 Like