Accelerometer/gyroscope support for iOS and Android

[aamf]

Does JUCE have no support for sensors of mobile devices (accelerometer, gyroscope, proximity sensor…)?

I would think that many people developing musical applications would want to use such sensors to modulate sounds, affect velocity or pitch, etc. And of course it’s fundamental for games. It just seems such basic functionality.

Accessing the sensors is almost trivial on iOS with Swift, for example, but I have not found a way to do so using C++ (or Objective-C++). I have looked at the few examples in this forum, examined the way OpenFrameworks reads the sensors, etc., but was unable to figure it out.

Is there a way to access the sensors which is easy enough for a beginner to understand and implement?

[aamf]

If this is of interest to anyone, this is how I got the motion sensors to work under iOS. It has no dependencies so it should work even if you are not using Juce.

You must include the CoreMotion framework. If you are using Projucer don’t do so within XCode, because Projucer will overwrite your changes. Instead, add the word Coremotion in Projucer under Exporters → Xcode(iOS) → Extra System Frameworks.

Since CoreMotion should be turned on only when needed, do so with the start() and stop() functions. Also, create only one MotionManager object. Within iOSMotionManager.mm, set the update interval to whatever you need. According to Apple, their devices support at least 100 Hz.

I think it’s all quite self-explanatory, so here’s the code:

iOSMotionManager.h

class MotionManager
{
public:

    MotionManager();
    ~MotionManager();

    void start();
    void stop();

    double accelerationX = 0;
    double accelerationY = 0;
    double accelerationZ = 0;
    
    double gravityX = 0;
    double gravityY = 0;
    double gravityZ = 0;

    double rotationX = 0;
    double rotationY = 0;
    double rotationZ = 0;

    double attitudeX = 0;
    double attitudeY = 0;
    double attitudeZ = 0;
        
    void accelerationChanged (double x, double y, double z);
    void gravityChanged      (double x, double y, double z);
    void rotationChanged     (double x, double y, double z);
    void attitudeChanged     (double x, double y, double z);

private:

    void* motionManagerWrapper;
    bool isRunning;
};

iOSMotionManager.mm

#ifdef __APPLE__
#import <TargetConditionals.h>
#ifdef TARGET_OS_IOS
#include "iOSMotionManager.h"
#import <CoreMotion/CoreMotion.h>

@interface MotionManagerWrapper : NSObject
{
    MotionManager* owner;
}

@property (strong, nonatomic) CMMotionManager *motionManager;

@end

@implementation MotionManagerWrapper

- (void) startMotionManager
{
    self.motionManager = [[CMMotionManager alloc] init];
    self.motionManager.deviceMotionUpdateInterval   = 1.0 / 60.0;
    [self.motionManager startDeviceMotionUpdatesToQueue:[NSOperationQueue currentQueue]
                                            withHandler:^(CMDeviceMotion *motion, NSError *error)
     {
        [self outputUserAcceleration    :motion.userAcceleration];
        [self outputGravity             :motion.gravity];
        [self outputRotationRate        :motion.rotationRate];
        [self outputAttitude            :motion.attitude];
    }];
}

- (void) stopMotionManager
{
    [self.motionManager stopGyroUpdates];
    [self.motionManager stopAccelerometerUpdates];
    self.motionManager = nil;
}

- (void) outputUserAcceleration: (CMAcceleration)acceleration
{
    owner->accelerationChanged(acceleration.x, acceleration.y, acceleration.z);
}

- (void) outputGravity: (CMAcceleration)gravity
{
    owner->gravityChanged(gravity.x, gravity.y, gravity.z);
}

- (void) outputRotationRate: (CMRotationRate)rotation
{
    owner->rotationChanged(rotation.x, rotation.y, rotation.z);
}

- (void) outputAttitude: (CMAttitude *)attitude
{
    owner->attitudeChanged(attitude.pitch, attitude.roll, attitude.yaw);
}

- (id) initWithOwner: (MotionManager*) owner_
{
    if ((self = [super init]) != nil) { owner = owner_; };
    return self;
}

- (void) dealloc
{
    [self stopMotionManager];
    [super dealloc];
}

@end

MotionManager::MotionManager()
{
    isRunning = false;
    MotionManagerWrapper* newManager = [[MotionManagerWrapper alloc] initWithOwner: this];
    [newManager retain];
    motionManagerWrapper = newManager;
}

MotionManager::~MotionManager()
{
    [((MotionManagerWrapper*) motionManagerWrapper) release];
}

void MotionManager::start()
{
    if (!isRunning)
    {
        [(MotionManagerWrapper*) motionManagerWrapper startMotionManager];
        isRunning = true;
    }
}

void MotionManager::stop()
{
    [(MotionManagerWrapper*) motionManagerWrapper stopMotionManager];
    isRunning = false;
}

void MotionManager::accelerationChanged (double x, double y, double z)
{
    accelerationX = x;
    accelerationY = y;
    accelerationZ = z;
}

void MotionManager::gravityChanged (double x, double y, double z)
{
    gravityX = x;
    gravityY = y;
    gravityZ = z;
}

void MotionManager::rotationChanged (double x, double y, double z)
{
    rotationX = x;
    rotationY = y;
    rotationZ = z;
}

void MotionManager::attitudeChanged (double x, double y, double z)
{
    attitudeX = x;
    attitudeY = y;
    attitudeZ = z;
}

#endif
#endif

By the way, the code above was adapted from here (thanks to hanley for posting):

iOS MidiKeyboard Touch Velocity

I’ve integrated iOS MotionManager into the MidiKeyboardComponent to pick up touch velocity on the iPad. It’s not bad, but there’s a fair amount of unwanted latency, especially when compared to Garageband’s keyboard velocity, which is one of the better feeling velocity keyboards I’ve come across. I thought the info and code below might be useful for anyone else looking to do this, and I’m hoping for suggestions on how one might improve this. I had originally hoped to add touch sensitive velocity…

Now, does anyone know how to get this working in Android and would like to share? That would be great!

[jrlanglois]

This is a sweet Feature Request (hint to convert your post )! Though I suggest stepping back to look at it from its whole because there are many more sensors available on mobile, both iOS and Android.

That includes magnetometer, pedometer, altitude, and GPS ( Add support for obtaining a user's GPS location, and its accuracy ), among others.

For Android, the list is here: Sensors Overview  |  Android Developers

For iOS, the list is here: Apple Developer Documentation

I don’t believe it’s much more work to involve these extra bits because you/someone will have their hands in that area already.

[aamf]

Yes, by all means! All of those sensors open up many posibilities. I’m sure that if they were easily available in Juce, people would come up with great ideas and completely unexpected ways to use them. They are great fun for sure

[aamf]

Small correction (for some reason I am not able to edit the post anymore):
the two #ifdef in the beginning should instead be just #if

[aamf]

So, here is a very bare-bones way to access the sensors in Android. It’s surely far from perfect (no security checks, does not take the orientation of the display into account, etc.) but hopefully enough for people to get started and not have to waste too much time dealing with boilerplate. I just included a few of the many sensors and very roughly matched the values to the iOS counterparts. One difference though (which is purely the result of my inexperience) is that on my Android implementation you have to constantly call update() for the values to get updated (simply do so from within a timerCallback, for example).

AndroidMotionManager.h

#include <android/sensor.h>
class MotionManager
{
public:
    MotionManager();
    ~MotionManager();

    void start();
    void stop();
    
    void Update();
    
    double accelerationX = 0;
    double accelerationY = 0;
    double accelerationZ = 0;
    double gravityX = 0;
    double gravityY = 0;
    double gravityZ = 0;
    double rotationX = 0;
    double rotationY = 0;
    double rotationZ = 0;
    double attitudeX = 0;
    double attitudeY = 0;
    double attitudeZ = 0;
  
private:
    const int LOOPER_ID_USER = 3;
    int SENSOR_REFRESH_RATE_HZ = 100;
    int SENSOR_REFRESH_PERIOD_US = 1000000 / SENSOR_REFRESH_RATE_HZ;
    
    ASensorManager* sensorManager;
    ASensorEventQueue* motionEventQueue;
    ALooper* looper;

    const ASensor* accelerometer;
    const ASensor* rotation;
    const ASensor* gravity;
    const ASensor* attitude;
};

[aamf]

AndroidMotionManager.cpp

#if __ANDROID__
#include "AndroidMotionManager.h"

MotionManager::MotionManager()
{
    sensorManager       = ASensorManager_getInstance();
    accelerometer       = ASensorManager_getDefaultSensor(sensorManager, ASENSOR_TYPE_LINEAR_ACCELERATION);
    rotation            = ASensorManager_getDefaultSensor(sensorManager, ASENSOR_TYPE_GYROSCOPE);
    gravity             = ASensorManager_getDefaultSensor(sensorManager, ASENSOR_TYPE_GRAVITY);
    attitude            = ASensorManager_getDefaultSensor(sensorManager, ASENSOR_TYPE_GAME_ROTATION_VECTOR);
    looper              = ALooper_prepare(ALOOPER_PREPARE_ALLOW_NON_CALLBACKS);
    motionEventQueue    = ASensorManager_createEventQueue(sensorManager, looper, LOOPER_ID_USER, NULL, NULL);
}

MotionManager::~MotionManager()
{
    ASensorManager_destroyEventQueue(sensorManager, motionEventQueue);
}

void MotionManager::start()
{
    ASensorEventQueue_enableSensor(motionEventQueue, accelerometer);
    ASensorEventQueue_setEventRate(motionEventQueue, accelerometer, SENSOR_REFRESH_PERIOD_US);
    ASensorEventQueue_enableSensor(motionEventQueue, rotation);
    ASensorEventQueue_setEventRate(motionEventQueue, rotation, SENSOR_REFRESH_PERIOD_US);
    ASensorEventQueue_enableSensor(motionEventQueue, gravity);
    ASensorEventQueue_setEventRate(motionEventQueue, gravity, SENSOR_REFRESH_PERIOD_US);
    ASensorEventQueue_enableSensor(motionEventQueue, attitude);
    ASensorEventQueue_setEventRate(motionEventQueue, attitude, SENSOR_REFRESH_PERIOD_US);
}

void MotionManager::stop()
{
    ASensorEventQueue_disableSensor(motionEventQueue, accelerometer);
    ASensorEventQueue_disableSensor(motionEventQueue, rotation);
    ASensorEventQueue_disableSensor(motionEventQueue, gravity);
    ASensorEventQueue_disableSensor(motionEventQueue, attitude);
}
    
void MotionManager::Update()
{
    ALooper_pollAll(0, NULL, NULL, NULL);
    ASensorEvent event;
    
    while (ASensorEventQueue_getEvents(motionEventQueue, &event, 1) > 0)
    {
        if (event.type == ASENSOR_TYPE_LINEAR_ACCELERATION)
        {
            accelerationX = event.acceleration.x * -0.08;
            accelerationY = event.acceleration.y * 0.08;
            accelerationZ = event.acceleration.z * 0.08;
        }
        
        if (event.type == ASENSOR_TYPE_GYROSCOPE)
        {
            rotationX = event.vector.x * 1;
            rotationY = event.vector.y * 1;
            rotationZ = event.vector.z * 1;
        }
        
        if (event.type == ASENSOR_TYPE_GRAVITY)
        {
            gravityX = event.acceleration.x * -0.1;
            gravityY = event.acceleration.y * 0.1;
            gravityZ = event.acceleration.z * 0.1;
        }
        
        if (event.type == ASENSOR_TYPE_GAME_ROTATION_VECTOR)
        {
            attitudeX = event.vector.y * 2;
            attitudeY = event.vector.x * 2;
            attitudeZ = event.vector.z * 2;
        }
    }
}

#endif

[jrlanglois]

Nice work! I wonder if it would be simpler to use juce::Vector3D (which I wish was part of the juce_graphics module instead of the juce_opengl module) for data type to DRY it up.

[aamf]

Thanks! I guess that depends. For what I am working on, having the values separately (not as a 3Dvector) is slightly more convenient. Of course, a proper JUCE class would ideally offer several different mathematical representations of the data, such as euler angles, rotation matrix, quartenion, etc.

[benvining]

Thank you so much for sharing this code!

Here is an attempt to condense this into a Juce module format: Shared-code/modules/bv_motion at main · benthevining/Shared-code · GitHub

[aamf]

Brilliant! I see that you added a Timer to not update the Android sensor elsewhere, as I was doing. Nice!
I had to add: #include “juce_events/juce_events.h” for it work in my configuration, but it all works perfectly.
Thanks for sharing!