Newest sensors and MEMS enabling efficient context awareness for gaming and beyond

Over the last few years, gaming, supported by electronic devices, has expanded from a niche market to encompass age categories from toddlers to seniors. One enabler of the explosion of gaming adoption is an intuitive user-interface, as used by the Wii™ and motion games on phones or MP3 players. Intuitiveness of action through gesture is possible because of sensors like accelerometers and gyroscopes, which use Micro Electro Mechanical Systems (MEMS) technology with analogue mixed-signal chips in one package. They benefit from low-power consumption, small real estate, easy implementation and low price per unit.

The aggregation of different sensors like accelerometers, gyroscopes, ecompasses and altimeters with the adequate software enables the development of advanced games such as piloting a drone or interacting with a robot. However, the vast majority of the games played on portable devices use only one or two sensors: generally accelerometers and capacitive touch sensors. The rate of adoption by the end user is so high that it is common to see people touching the screen or shaking their new portable device when experiencing a game. The success of such games is based on the simplicity of control allowed by the sensors. Furthermore, the main sensor used in these games, the 3-axis accelerometer, has evolved over the last year. Analogue output motion sensors like Freescale semiconductor's MMA7361LC accelerometer, provide continuous information about user movement such as acceleration and are still used in console accessories. However, smarter devices with embedded functions are invading the portable and mobile electronic device space. Automatic sampling rate, interrupt control, pulse and jolt detection, low-power consumption, digital interface and high precision with a 14-bit like Freescale's MMA8451Q accelerometer are staples of embedded functions. This type of part enables very accurate gesture recognition without the penalties of power consumption. New games providing novel user experience by reflecting the complexity of hands gestures are starting to emerge thanks to these new capabilities.

Despite the considerable adoption of games using sensors in portable devices and games console accessories, the association of gaming and sensors is still in its early stages. Sensors are adding new dimensions which are unlocking innovative fields. For instance, Freescale has leveraged its medical know-how to adapt it to new games based on "emotion sensing." The platform uses a standard console game controller with the implementation of sensors like capacitive touch, humidity, accelerometer and pressure. The data of each sensor is processed through a 32-bit MCU with specific software developed based on neuroscience theories and experiments: heart rate, sweating, muscle contraction and attitude reflect a person's emotion. By analyzing these parameters in real time, the game can reflect the player's emotion with real life interaction. One of the demonstrated games is a sniper shooting game where the shooting becomes more difficult depending on the detected emotion of the player: for instance, with a shakier target if the player is getting upset or a lower visibility if the player is sweating. This type of interaction makes any game much more interesting since each session is adapted to the player and reflects the player's inner feelings more accurately.

This means that the number of gaming possibilities is exponentially proportional to the number of sensing elements; but also the complexity of creating new games.

Therefore, adding new kinds of sensors and combining the pre-processing and fusion of their data input is the current challenge for new applications. By anticipating our customers' challenge, Freescale has developed a new concept which combines a microcontroller with a motion sensor within the same package (the Freescale's MMA9550L motion sensor platform is part of the new Xtrinsic™ family). This motion sensor platform provides not only the extracted and pre-processed data of the motion sensor to the main application processor, but can also combine and conduct fusion of data together from external sensors. Consequently, specific sensor complexity like calibration, data filtering and specific application data is moved to this local MCU and only the required data is provided to the developer for easy implementation. A call to a high-level programming function can be the ultimate way for a developer to get access to the data he wants using this new product concept.

A simple example for a pointing device application (acting as an air mouse) is shown in figure 1. A standard solution uses a 3-axis accelerometer and a 3-axis magnetometer, like the MAG3110, which requires two different digital interfaces connected to another microcontroller or application processor. The magnetometer calibration (soft iron and hard iron) has to be managed by an external processing resource; the tilt compensation, using the accelerometer information, also has to be processed. So, 3360 bit/s of raw data needs to be calibrated and processed (assuming a sampling data rate of each sensor at 60 Hz). Using the Freescale's MMA9550L, the 3-axis accelerometer can be replaced by adding processing power and interfaces for external sensors; it also has the same real estate size: 3 by 3 mm. Therefore, still working with a presumed data rate of 60 Hz, the magnetometer calibration and tilt compensation can be locally processed which result of only 640 bit/s of data. The data provided can be used directly by the game developer. Beyond this gain in data processing which, in turn, saves power consumption; real time reactivity is also greatly improved since the latency is reduced, by avoiding synchronization issues and task scheduling, on top of data processing time saving.

Figure 1: MMA9550L with a 3-axis magnetometer (MAG3110) for a full 3D ecompass application

By extension, using sensors connected wirelessly is the way to a new generation of games where the real world can be viewed through wireless sensor networks (WSN) and visualized through virtual reality. The bottleneck of processing all the data coming from the sensors can be solved by using intelligent sensors like the MMA9550L, since local processing will greatly reduce the main application processor CPU usage and data loading. Re-creating a certain reality through different kinds of sensors like cameras, accelerometers, capacitive touch, and gyroscopes is going beyond simply adding a third dimension to the image, as it allows for interpretation of reality. Consequently, players will be able to enter an era of "contextual interaction" which will be rich in innovative games since the interaction with the real world can be controlled by the player and not the player trying to adapt to the game.

An illustration of richer sensor context in a special wireless gamepad is shown in figure 2. Processing the sensor data before transmitting is important for real time processing and to achieve the best trade-off for power consumption, communication and computation. The combination of a high accuracy accelerometer with the processing also allows adapting the sampling rate from 1 Hz, when no activity is detected, to a few kilo hertz of data sampling when user activity is at its maximum.

Figure 2: Example of a richer sensor context in a special wireless gamepad

With a wider adoption of sensors, games are spreading to a broader population with user friendly interaction. Adding more sensors means exponentially increasing the kind of games possible and unlocking innovations like the "emotion sensing" concept. However, the future is in contextual interaction supported by sensor networks allowing gamers to control a certain reality with new kinds of interaction.