So, I tested the time between touch events (via Java log) on mouseDrag (ACTION_MOVE) and its averaging around 70-85ms. I'm not sure why its so slow. I'm digging around to see if there's anything that can be done on the Java side.
UPDATE: Testing on an HTC Desire, with OpenGL switched on (via attachTo on topLevelComponent) I get crazy graphical glitches appearing. Same APK runs fine on a Samsung Galaxy Ace 2. I've switched back to software rendering for now, as I think my performance issues stem from elsewhere.