Win7 MultiTouch

Look very WIN32 like http://msdn.microsoft.com/en-us/library/dd744775(v=VS.85).aspx i wonder if they ever run out of WM_ names (WM_TOUCH is the new one)

Hi!

Reviving an old thread here instead of creating a new one.

I tested the Kivy GUI framework demo (kivy.org) on my multitouch screen, running on Windows 7, and got exactly what I wanted:

I grabbed two faders, one finger each, and I could control each one separately.

I then ran the latest Juce widget demo from the repository, but it did not work the same way, I only got a single fader moving at a time, no multitouch support.

Seeing that last year there was discussion in this thread on Juce supporting multitouch control over its widgets in Win 7, I was wondering whether anyone has had any progress towards this goal.

Has anyone managed to get this kind of functionality with Juce on Windows 7, or is this still only supported on iPads/iPhones?

OR perhaps someone has managed to somehow fuse Juce and Kivy? Sounds unlikely but it is worth asking :slight_smile:

Thanks!

Sorry - it’d probably only take a few lines of code to get it going on Win7, but I don’t have a touchscreen so never got around to looking into it… I suspect it’d just need a couple of handlers to translate the appropriate WM_ messages, as all the multi-touch front-end classes were put in place ages ago for iOS/android.

That’s encouraging to hear, it’s not that far from working then!

I’m no seasoned C++ programmer at all so I don’t know if I could pull this integration into Juce off myself, my re-introduction to C++ programming was through Juce, so arcane Win32 API’s are not my forte… If I read up on it and it seems manageable I might tackle it.

If anyone / Jules wants to give integrating Win32 multitouch into Juce a go, but doesn’t have a multitouch screen, there’s a solution that I’d say is very neat, it allows you to simulate multitouch devices using multiple mice:

http://michaelsync.net/2010/04/06/step-by-step-tutorial-installing-multi-touch-simulator-for-silverlight-phone-7

This is the download you’ll need: http://multitouchvista.codeplex.com/

I just tested it on Windows 7 64 bit with two mice, and managed to get multitouch going just fine, it worked with all multitouch software I threw at it!

Just make sure the “Universal software HID device” driver is installed correctly. It took my enabling the checkbox that windows should always trust the driver’s provider to make it work.

Interesting solution! Unfortunately I run Windows in a VM, so I doubt whether it’ll work for me!

I’d say if you have two mice on your machine and it can use both, it is not impossible they are both visible by the VM, no?

As an alternative solution, it also works as a TUIO receiver I forgot to mention!

If you’re running on a VM, this would be a good option, as the only thing it requires besides installing the device driver on windows, is sending TUIO messages to the correct IP & port.

Then, if one generates TUIO calls in a scripting language for example, this virtual multitouch driver can receive those, and use then to simulate a windows multitouch screen.

Hi!

I’ve progressed quite a bit with this.

From looking at microsoft’s MTScratchpadWMTouch example app, and the changes posted earlier in this thread by symfonysid, I got something promising.

On windows each individual touch has its own up/down/move flags, as opposed to the IOS case where those flags seem to me to be global for the whole bunch. There are thus some differences in my code compared to what I used as a starting point in “juce_ios_UIViewComponentPeer.mm”: UIViewComponentPeer::handleTouches.

I also do not keep the “currentTouches” array, as to my reading of the code it doesn’t seem to be used for anything currently, but I could of course be very wrong on this :slight_smile:

Now, in the Widget panel of the Juce demo, when I grab two sliders and drag them it works!

Trouble is however that when I lift the last finger, sometimes in the Slider::mousedrag(…) function the “jassert (sliderBeingDragged == 2);” assertion fails, sliderBeingDragged having instead a value of -1.

Is this a result of my doing something wrong? Perhaps I should be keeping track of the number of active touches in “currentTouches” after all for some reason? Or is the -1 value above the result of some (unlikely) bug in juce?

Of course this modification required changes in several juce files. I’ve uploaded them as attachments in case anyone is curious, my changes are all commented with “// onar3d was here”.

I’ll press on with this in the meantime.

Doh, it turns out (and I did do a search, promise!), that the thread right before mine was about this very bug in slider :slight_smile: (http://www.rawmaterialsoftware.com/viewtopic.php?f=2&t=7940)

I’ll try and fetch the new code for slider from the modules branch then, and see if this solves it!

I tested my code on the modules branch of juce and it worked fine!

I also tested it on my dell latitude xt2 which has four touch points and it responded to all four.

Strange thing though: on the XT2, the touch point ID’s I get from windows are not on a range of 0-9, but from 24 and up!

For the indexes to function in the fixed MouseInputSource array in the Desktop class, I had to increase its size to 30, but of course that is not a good permanent solution. Perhaps a dynamic hashtable is in order instead? Who knows, on another computer those values may be in the thousands! I’ll have to check the win api documentation to see why these ID’s vary as they do…

http://msdn.microsoft.com/en-us/library/windows/desktop/dd317334(v=vs.85).aspx

The number given by windows for identifying each touch is an ID, not an index, so there seems to be no certainty as to what their value range is, only that they uniquely identify a particular touch point.

So a bit more code is needed to cater for that, and the MT implementation for Win 7 seems to be done…

Sounds cool! Keep me posted, I’ll check out what you’ve done as soon as possible!

So yeah obviously I can see why you kept an array of the touch points now.

The code now is as close as can be to that for ios. From the modules branch this time around.

I’m sure something must be wrong with it, but it works, so… :slight_smile:

EDIT: There’s some silliness left in there still. The updating of the touchpoint array seems to be broken after certain series of touches…
I’ll look at it tomorrow again.

Thanks, I’m tidying this up now (well… “completely rewriting” might be a better phrase!)

One question: you use an Array<TOUCHINPUT*> and fill it with pointers to temporary variables allocated on the stack (!!!)… This doesn’t seem to make any sense - is there a reason why you didn’t use an Array containing the touch IDs?

Anyway, going well! I can’t test it myself, but when I’m done I’ll check it in and you can sanity-check what I’ve done!

Ok, have pushed a new version - might possibly work!

Hi!
I just tested this, and have some good news and some bad news.

Good news first, the Juce Demo sliders work just fine on both my touch-screen systems, tested with up to 4 fingers simultaneously.

Bad news is, I think I’ve found a concurrency bug in button click handling. I’ve been testing with the toggle buttons on the demo’s graphics tab, the first one to appear when starting the demo.

With no breakpoints in visual studio, in the latest “modules” tip from yesterday, clicking a toggle button with the mouse works correctly, but tapping with the finger works only rarely.
When putting a breakpoint in juce_Button’s “void Button::mouseDown (const MouseEvent& e)” method, at the “if (isDown())” line, mouse clicks suddenly have no effect after resuming execution, i.e. the button is not toggled.

In the 1.53 version (without my own modifications of course) clicks and finger taps function correctly when the demo is running, BUT with the breakpoint placed at the same line I get the same behaviour: the toggle is not switched.

It may be that these two issues are not connected – i.e. toggle not switching with touch, and the breakpoint causing the same behaviour also for mouse clicks, but to me it seems like a racing condition that may well be the underlying common cause.

I’m not too knowledgeable about how Juce synchronizes events between various threads internally, so for now I’ll just go out in the sun for a while :slight_smile:

As for the array of pointers, I did recognize it as risky behaviour, but it seemed to be a similar solution to the juce_ios_UIViewComponentPeer.mm “Array <UITouch*> currentTouches;”. Since it didn’t crash horribly, I left it in there for the sake of similarity between the code for different platforms. Note that my code was very much intended as a proof-of-concept sketch, not something I would have been comfortable with checking in and shipping to a client :slight_smile:

For the sake of my learning (most of my experience is with managed code), I imagine your horror was because I risked leaving dangling pointers? Or inefficiency? Or perhaps both? :smiley:

Well, good to know that my changes worked!

No idea what you mean about a concurrency issue - all GUI events happen on a single thread! What other threads are you talking about there??

The only place I can think of where events could get getting lost would be here:

if ((flags & (TOUCHEVENTF_DOWN | TOUCHEVENTF_MOVE | TOUCHEVENTF_UP)) != 0) { if (! handleTouchInput (inputInfo[i], (flags & TOUCHEVENTF_DOWN) != 0, (flags & TOUCHEVENTF_UP) != 0)) return; // abandon method if this window was deleted by the callback }

…if the event had both the down/up flags set simultaneously. I looked at msdn to try to see whether that’s possible, but it didn’t give any details - perhaps add an assertion like this to find out?

[code] if ((flags & (TOUCHEVENTF_DOWN | TOUCHEVENTF_MOVE | TOUCHEVENTF_UP)) != 0)
{
jassert ((flags & (TOUCHEVENTF_DOWN | TOUCHEVENTF_MOVE | TOUCHEVENTF_UP)) == TOUCHEVENTF_DOWN
|| (flags & (TOUCHEVENTF_DOWN | TOUCHEVENTF_MOVE | TOUCHEVENTF_UP)) == TOUCHEVENTF_UP
|| (flags & (TOUCHEVENTF_DOWN | TOUCHEVENTF_MOVE | TOUCHEVENTF_UP)) == TOUCHEVENTF_MOVE);

                if (! handleTouchInput (inputInfo[i], (flags & TOUCHEVENTF_DOWN) != 0,
                                                      (flags & TOUCHEVENTF_UP) != 0))
                    return;  // abandon method if this window was deleted by the callback
            }

[/code]

Hi!

The issue above also appears in unmodified, normal juce 1.53, (just cloned fresh using git), if I place a breakpoint within the button class, and then continue from it. Mouse only, no touch involved.

It appears (to me) as a concurrency issue only because otherwise how could pausing at a breakpoint and then continuing cause different behaviour? Perhaps there are other possible reasons, but still I imagine it is undesired behaviour.

Quick steps to reproduce:

  1. Run the juce demo, and press the toggle “clip to rectangle”. Of course it toggles fine with no breakpoint.

  2. Now put a breakpoint in the “Button” class, in the “void Button::mouseDown (const MouseEvent& e)” method, at the line 420, " if (isDown())" ,
    and press the toggle.

  3. After telling Visual Studio to continue from the breakpoint, the button is not toggled like it should be!

Finally, I tried the assertion you posted and it never was fired despite an intricate four-finger-workout :slight_smile:

I also noted by mistake, that with multitouch toggles do actually toggle when double-tapped. Which isn’t how it works with checkboxes in other non-juce applications, so I never thought to try it before.

I will read the relevant juce code tomorrow to see whether somehow this can be explained.

It’s incredibly difficult to debug the user input events of an application from the same machine as the application. The events may not be firing because once the breakpoint is hit, the focus has changed to the debugger. You may be able to debug this with the remote debugger, but otherwise I consider it very “normal” for events to be lost if attempting to debug input events like keyboard presses and mouse clicks in an application with the debugger running on the same machine collecting its input from the same devices. The same goes for trying to debug focus-related events, since switching to the debugger alters the focus. I think you’ve been hit by Heisenberg here and the act of trying to monitor something has affected it.

pwhittemore: Thank you for a great explanation!

I was aware that events such as lost and gained focus were subject to such problems, but hadn’t thought that there could also be interactions with click events. It makes a great deal of sense though!

I realize now if I put a breakpoint after the mouseDown event, the focus will be on visual studio in time for the mouseUp event, and it will never be received by the program, potentially causing all kinds of weird behaviour!

I’ll just look for other causes then - the fact that a double-tap works is a good lead to follow - and learning to do remote debugging is a good bonus in this process :slight_smile:

Thanks again!

Hi!

I figured out what it was, it had nothing to do with multithreading or other bugs.

Windows sends both mouse and touch messages simultaneously, and expects the application to choose between them.

In the case of Juce, both messages were reacted to, and so a single tap on a button resulted in the button receiving two such messages; and appearing not to respond at all since it was switched on and then immediately off again.

The solution is to check what kind of device the message comes from and react to it accordingly: http://msdn.microsoft.com/en-us/library/windows/desktop/ms703320(v=vs.85).aspx

I cannot think of a reason why this check would be necessary also for other messages besides mouse up & down, but it may well be so. After having tapped about merrily in the juce demo for a few minutes I didn’t come across any more problems though.

EDIT: MouseMove events were also doubled in both touch and mouse equivalents. I updated the example below to reflect the corresponding change. Other messages seem to be affected too, as detailed in the above msdn link, but JUCE doesn’t seem to be using those, for example double-click.

I tested with the following in juce_win32_Windowing.cpp:

LRESULT peerWindowProc (HWND h, UINT message, WPARAM wParam, LPARAM lParam)
{
	LPARAM extrainfo = GetMessageExtraInfo();

	switch (message)
	{   
	(...)
		case WM_RBUTTONDOWN:
		if ((extrainfo & 0xFFFFFF00) == 0xFF515700) // Checking if message is from touchscreen
			return 0;
		doMouseDown (getPointFromLParam (lParam), wParam); 
		return 0;

		case WM_LBUTTONUP:
		case WM_MBUTTONUP:
		case WM_RBUTTONUP:          
			if ((extrainfo & 0xFFFFFF00) == 0xFF515700)  // Checking if message is from touchscreen
				return 0;
			doMouseUp (getPointFromLParam (lParam), wParam); 
			return 0;
		case WM_MOUSEMOVE:        
			if ((extrainfo & 0xFFFFFF00) == 0xFF515700)  // Checking if message is from touchscreen
				return 0;
			doMouseMove (getPointFromLParam (lParam)); 
			return 0;
(...)
	}
}