Body-Based Gestures

Just as touch can provide satisfaction in user interfaces, 3D cameras have opened up interaction design to include the whole body. I secured a collaboration agreement between the Primesense company and USC to work with Evan Suma to create open-source software that acted as a translator between skeletal gestures on a Microsoft Kinect or Primesense device and mouse and keyboard commands. We initially focused on creating applications for health and wellness such as stroke rehabilitation and breathing exercises, then introduced the software to the gaming community through some fun and well received videos, supporting the thousands of users who downloaded the application to control games, and created a prototype application exploring the value of such interfaces in places like operating rooms.

 

I find the design space of body-based interaction irresistible because it enables the mapping of the emotions of a gesture to the affordances of an interface. Working with Primesense on a consumer product reference design, I found that we needed to warp the coordinate space around a user to map it to meaning – for example, a far reach feels different than a short reach and pulling toward the body feels different then pushing away. I have also researched ways to detect subtle non-verbal behaviors such as breathing rate and fidgeting with 3D cameras while working on a user-sensing project for DARPA with Skip Rizzo.

Selected Publications

 Adapting User Interfaces for Gestural Interaction with the Flexible Action and Articulated Skeleton Toolkit. Computers & Graphics, 37(3), pp. 193-201, 2013.

 

Designing Informed Game-Based Rehabilitation Tasks Leveraging Advances In Virtual Reality. Disability and Rehabilitation 34(22), pp. 1863-1870, 2012.

 

Virtual Reality and Interactive Digital Game Technology: New Tools to Address Obesity and Diabetes. Journal of Diabetes Science and Technology 5(2), pp. 256-264, 2011.

 

Breath: A Game to Motivate Adherence of Breathing Exercises. Journal of Physical Therapy Education, presented at CSM (American Physical Therapy Association Combined), pp. 30-35, 2011.

 

Markerless Full Body Tracking: Depth-Sensing Technology within Virtual Environments. I/ITSEC, 8 pages, 2011.

 

Infinite Content Within Reach: Using 3D Body-Based Interaction to Enhance Content Discovery. 3DCHI, pp. 43-52, 2012. Workshop.

 

Unobtrusive Measurement of Subtle Nonverbal Behaviors with the Microsoft Kinect. IEEE VR Workshop on Ambient Information Technologies, pp. 10–13, 2012.