Tap pressure strength detection using accelerometer

Yesterday, in the presentation about the new Garageband for iPad 2, Apple demoed an interesting feature: The detection of t开发者_Python百科he tap pressure by using the accelerometer. (See the drums section on the Garageband page.)

I'm wondering how that's supposed to work if the iPad lays flat on the table. No movement, no measurable acceleration, no?

Some good answers. Here's some working code. I implemented this as a subclass of UIGestureRecognizer so that you can just drop it in and attach it to a UIView or UIButton. Once triggered, it will have the "pressure" set to a float between 0.0f and 2.0f. You can optionally set the minimum and maximum pressures required to recognize. Enjoy.

//
//  CPBPressureTouchGestureRecognizer.h
//  PressureSensitiveButton
//
//  Created by Anthony Picciano on 3/21/11.
//  Copyright 2011 Anthony Picciano. All rights reserved.
//
//  Redistribution and use in source and binary forms, with or without
//  modification, are permitted provided that the following conditions
//  are met:
//  1. Redistributions of source code must retain the above copyright
//     notice, this list of conditions and the following disclaimer.
//  2. Redistributions in binary form must reproduce the above copyright
//     notice, this list of conditions and the following disclaimer in the
//     documentation and/or other materials provided with the distribution.
//  
//  THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
//  IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
//  OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
//  IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
//  INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
//  NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
//  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
//  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
//  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
//  THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
//

#import <UIKit/UIKit.h>

#define CPBPressureNone         0.0f
#define CPBPressureLight        0.1f
#define CPBPressureMedium       0.3f
#define CPBPressureHard         0.8f
#define CPBPressureInfinite     2.0f

@interface CPBPressureTouchGestureRecognizer : UIGestureRecognizer <UIAccelerometerDelegate> {
    @public
    float pressure;
    float minimumPressureRequired;
    float maximumPressureRequired;

    @private
    float pressureValues[30];
    uint currentPressureValueIndex;
    uint setNextPressureValue;
}

@property (readonly, assign) float pressure;
@property (readwrite, assign) float minimumPressureRequired;
@property (readwrite, assign) float maximumPressureRequired;

@end


//
//  CPBPressureTouchGestureRecognizer.h
//  PressureSensitiveButton
//
//  Created by Anthony Picciano on 3/21/11.
//  Copyright 2011 Anthony Picciano. All rights reserved.
//
//  Redistribution and use in source and binary forms, with or without
//  modification, are permitted provided that the following conditions
//  are met:
//  1. Redistributions of source code must retain the above copyright
//     notice, this list of conditions and the following disclaimer.
//  2. Redistributions in binary form must reproduce the above copyright
//     notice, this list of conditions and the following disclaimer in the
//     documentation and/or other materials provided with the distribution.
//  
//  THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
//  IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
//  OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
//  IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
//  INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
//  NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
//  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
//  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
//  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
//  THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
//

#import <UIKit/UIGestureRecognizerSubclass.h>
#import "CPBPressureTouchGestureRecognizer.h"

#define kUpdateFrequency            60.0f
#define KNumberOfPressureSamples    3

@interface CPBPressureTouchGestureRecognizer (private)
- (void)setup;
@end

@implementation CPBPressureTouchGestureRecognizer
@synthesize pressure, minimumPressureRequired, maximumPressureRequired;

- (id)initWithTarget:(id)target action:(SEL)action {
    self = [super initWithTarget:target action:action];
    if (self != nil) {
       [self setup]; 
    }
    return self;
}

- (id)init {
    self = [super init];
    if (self != nil) {
        [self setup];
    }
    return self;
}

- (void)setup {
    minimumPressureRequired = CPBPressureNone;
    maximumPressureRequired = CPBPressureInfinite;
    pressure = CPBPressureNone;

    [[UIAccelerometer sharedAccelerometer] setUpdateInterval:1.0f / kUpdateFrequency];
    [[UIAccelerometer sharedAccelerometer] setDelegate:self];
}

#pragma -
#pragma UIAccelerometerDelegate methods

-(void)accelerometer:(UIAccelerometer *)accelerometer didAccelerate:(UIAcceleration *)acceleration {
    int sz = (sizeof pressureValues) / (sizeof pressureValues[0]);

    // set current pressure value
    pressureValues[currentPressureValueIndex%sz] = acceleration.z;

    if (setNextPressureValue > 0) {

        // calculate average pressure
        float total = 0.0f;
        for (int loop=0; loop<sz; loop++) total += pressureValues[loop]; 
        float average = total / sz;

        // start with most recent past pressure sample
        if (setNextPressureValue == KNumberOfPressureSamples) {
            float mostRecent = pressureValues[(currentPressureValueIndex-1)%sz];
            pressure = fabsf(average - mostRecent);
        }

        // caluculate pressure as difference between average and current acceleration
        float diff = fabsf(average - acceleration.z);
        if (pressure < diff) pressure = diff;
        setNextPressureValue--;

        if (setNextPressureValue == 0) {
            if (pressure >= minimumPressureRequired && pressure <= maximumPressureRequired)
                self.state = UIGestureRecognizerStateRecognized;
            else
                self.state = UIGestureRecognizerStateFailed;
        }
    }

    currentPressureValueIndex++;
}

#pragma -
#pragma UIGestureRecognizer subclass methods

- (void)touchesBegan:(NSSet *)touches withEvent:(UIEvent *)event {
    setNextPressureValue = KNumberOfPressureSamples;
    self.state = UIGestureRecognizerStatePossible;
}

- (void)touchesCancelled:(NSSet *)touches withEvent:(UIEvent *)event {
    self.state = UIGestureRecognizerStateFailed;
}

- (void)reset {
    pressure = CPBPressureNone;
    setNextPressureValue = 0;
    currentPressureValueIndex = 0;
}

@end

I imagine that the aluminium case and the table don't prevent from very small movements and that the sensor is very sensitive.

Or, Garage Band user experience is better with iPad standing on the Smart Cover.

Well simply do this experiment:

open with XCode the AccelerometerGraph tutorial app included in the XCode bundle. Then start the app (High Pass filter, better using Adaptive filtering): you will see the blue line changing according to the strength of the app. Of course this is affected by trembling in the table, which adds noise to the measure, but you can still filter it by checking the accelerometer data with the touch event.

So this kind of pressure detection is possible using the accelerometer.

Tap area size detection? (Harder tap, larger fingerprint) Tap dynamics?

Just thinking.

viggio24 is right. finger-area sensing works fine. (i have some posts here with the trivial line of code to get it). the only problem is that it's not clear what the consequences are if you ship with it enabled; we assume that it simply won't be approved in the best case.

Perhaps it's due to the more sensitive gyroscope? Combined with Accelerometer data, it's probably reasonably easy to determine even small changes in movement. Just a hunch. Unless there's something they're not telling us, which wouldn't be a first.

Even if the SDK exposes to us the touch area only, I believe the screen hardware is providing to the operating system much more info: touched pixels (used for area detection) and touch duration. Together (and experimenting) they can give you a good estimate of the applied pressure. It would be nice if Apple were releasing this empirical pressure estimation to developers.

Try it yourself with the AccelerometerGraph sample app. The device and the tabletop have finite stiffness, so you might see little blips in the graph.

ADDED:

If you compile Apple's AccelerometerGraph graph sample from source code yourself, you can turn up the gain on the vertical axis and see blips for smaller accelerations.