Android mouse events are missing, touches getting stucked for more than 2 fingers

[yoogle]

Hi Folks,

I am trying to make a multitouch piano application for android in juce framework. The multitouch is working fine on ios devices, also in android it works till maximum 2 touches.

When i am trying to play a chord with 3 fingers, the touches gets stucked, due to system gestures like screenshot.

Is there any way to

1. prevent system gestures in android ?
2. dealing with multitouch event of more than 2 fingers for android ?
3. I tried using this

juce::Desktop::getInstance().getNumDraggingMouseSources()

but still it does not recognises the actual count after the touches gets stucked. Looks like the cancelled touches did not reached to juce.

[aamf]

Hi there. Could you post more information about your system (API level etc.) and if possible some code?

[yoogle]

Here is the code I am using for drawing circles with multitouch events.

private:
    juce::HashMap<int, juce::Point<float>> points;

void mouseDown(const juce::MouseEvent& event) override
{
    points.set(event.source.getIndex(), event.position);
    repaint();
}

void mouseDrag(const juce::MouseEvent& event) override
{
    points.set(event.source.getIndex(), event.position);
    repaint();
}

void mouseUp(const juce::MouseEvent& event) override
{
    points.remove(event.source.getIndex());
    repaint();
}

void paint (juce::Graphics& g) override
{
    g.fillAll (juce::Colours::black);
    g.setColour(juce::Colours::white);

    for (juce::HashMap<int, juce::Point<float>>::Iterator it (points); it.next();) {
        auto point = it.getValue();
        auto rect = juce::Rectangle<float> (0, 0, 100, 100);
        rect.setCentre(point);

        g.drawEllipse(rect, 2.f);
        g.drawText(juce::String(it.getKey()), rect, juce::Justification::centredBottom);
    }
}

From mouse events [0, 1, 2] → [1 and 2] got stucked.

Tried on Android 12, 13, 14, 15 with multiple devices.

[aamf]

I’ve tested on three devices with Android 8, 13 and 14. Everything worked perfectly, got five input sources on the older device, ten on the newer devices. No stuck events whatsoever.

Could it be that your UI thread is overloaded? Have you attached an OpenGLContext to your main window (this is pretty much a must with Android)? How do you have the system gestures set up?

My app also relies on multitouch and Android can be somewhat… “surprising”, so please post more information!

[yoogle]

Yes, I had attached it to opengl context. I had tested without attaching the context also. Both the things are giving same result, touches getting stucked.

Tested on OnePlus 12R.

Adding fingers one by one had no issues, issue arrives when 3 or more fingers touches at the same time, like playing a chord.

[aamf]

On my tests there are no problems with three or more fingers touching simultaneously. You say that a screenshot or other system events are triggered? How do you have that set up?

Perhaps another thing to check would be the AndroidManifest.xml or the theme settings.

Which JUCE version are you on?

EDIT:

Investigated a bit further since you mentioned OnePlus and apparently it’s a problem with some Android devices which have system-level three-finger gestures. The problem: Android doesn’t allow apps to disable these gestures. sigh.

I guess the only option is to show a message to the users to disable these gestures manually. Perhaps as soon as “stuck notes” are detected.

I think I will also check on startup if the user is using a device by one the “risky” manufacturers.

bool checkForThreeFingerGestureRisk()
{
    juce::StringArray riskyManufacturers = { "xiaomi", "oneplus", "realme", "oppo", "vivo" };
    return riskyManufacturers.contains (juce::SystemStats::getDeviceManufacturer().toLowerCase());
}

[yoogle]

I had removed these following 3 finger gestures and its working fine.

1. 3-finger swipe down for screenshot
2. 3-finger touch and hold for screenshot
3. 3-finger swipe up to enter split screen

The problem is now, how to detect these kind of gestures in juce first time when the user opened the app, and tell user to disable these things. Because if they encounter this touches stucked thing, the notes on the piano will keep ringing and stucked.

[aamf]

Hi there, have you figured out a good way to deal with this? I would at least like to have something in place which detects when a gesture has become “stuck”. But I don’t have a device by any of the “problematic” manufacturers. Could you describe exactly what happens? Do you get mouse down events for source numbers 1 and 2 but then not a mouse up event for those sources? Also not mouse drag I suppose? And if these touch events are “stuck”, do they become “unstuck” if you then add fingers one by one? Thanks!

[yoogle]

Yes, Exactly the same issue.

When android touches are getting cancelled, only first event is getting triggered, the other cancelled touches are not reported to juce. There is a bug in “ComponentPeerView.java” file for “ACTION_CANCEL” event. “ACTION_CANCEL” event has to be iterated for all the pointers.

[aamf]

Yes, I also think that’s a bug within “ComponentPeerView.java”. When ACTION_CANCEL is received mouseUp should be called for all pointers.

Have you tried modifying that file? Perhaps moving the line
case MotionEvent.ACTION_CANCEL:
immediately before
case MotionEvent.ACTION_POINTER_UP:
would already work, but I can’t test this myself.

Could you perhaps give it a try? Note that since this is Java you would need to also follow the first three steps in JUCE/modules/juce_core/native/java/README.txt

[yoogle]

Yes, I tried it and its started working. I dont know how to contribute it to the juce github repo.

[reuk]

Don’t worry about contributing to the repo - I’m planning to look at this issue alongside the Android safe-display-area updates. I still can’t promise a release date for this work, but it’s on my backlog.

[reuk]

Thanks for your patience. We’ve now pushed a change to handle ACTION_CANCEL correctly:

Android: Update handling of ACTION_CANCEL to terminate all ongoing pointer events, instead of just the primary pointer

Please let us know if you’re still seeing problems with this change applied.