TensorFlow for gesture recognition: It’s magic!

By Michael Parks, P.E., for Mouser Electronics

Introduction

Welcome to this multipart series of projects that will examine the convergence of embedded electronics and artificial intelligence (AI). That combination has enabled powerful, brain-like algorithms to escape the confines of powerful cloud-based server-grade hardware and find a place out on the edge of the cyber-physical interface. So-called “edge computing” allows billions of inexpensive embedded electronic systems to interact with the physical world in a near-instantaneous manner while contending with multiple dynamic inputs. Edge computing represents a revolution in automation, both in terms of scale and capabilities. With the costs of microcontrollers, memory, sensors, and associated development tools becoming more affordable than ever, building an intelligent AI-powered device is within the reach of just about everyone.

Project Materials and Resources

This first project in our TensorFlow Lite for Microcontrollers series will focus on NXP Semiconductor's MIMXRT1050 Evaluation Kit (Figure 1).

Figure 1: NXP Semiconductor's MIMXRT1050 Evaluation Kit (Source: Mouser)

Bill of Material

You can click this Mouser project share link to access the BOM along with the current pricing. As of the date this article , the BOM cost is about $104, before applicable taxes and shipping costs. Table 1 lists the items in the BOM.

Note that the next model up in this family of development boards (the i.MX RT1060) has the pads for needed FXOS8700 compass/accelerometer microchip, but the chip itself is not populated on the printed circuit board (PCB).

Resources

All source files for this project are located on Mouser's GitHub repository.

Hardware

This folder contains schematics of the i.MX RT1050 development board and the FXOS8700 6-axis compass microchip (Figure 2).

Figure 2: Schematic view of the FXOS8700 accelerometer and magnetometer microchip. (Source: NXP)

Software

We will have to modify or replace a few files (display and accelerometer interface) with our custom implementation in order to interact with i.MX RT1050 development board and associated components. Those files are:

• main_functions.cc
• output_handler.cc
• output_handler.h
• accelerometer_handler.cc
• accelerometer_handler.h

More details about these files can be found in the Software section below.

Tools

This project assumes that you have access to the following tools:

• Windows 10-based computer with at least 4GB RAM and 5GB of available storage.
• Internet connection

Documentation

A plethora of additional resources and documentation regarding NXP i.MX family of microcontrollers, the MCUXpress IDE, i.MX SDKs, and the TensorFlow Lite for Microcontrollers SDK are available. Some of the recommended documents to review include:

• Get Started with the MIMXRT 1050-EVK
  https://www.nxp.com/document/guide/get-started-with-the-mimxrt1050-evk:GS-MIMXRT1050-EVK
• MIMXRT1050-EVK: i.MX RT1050 Evaluation Kit Overview
  https://www.nxp.com/design/development-boards/i-mx-evaluation-and-development-boards/mimxrt1050-evk-i-mx-rt1050-evaluation-kit:MIMXRT1050-EVK
• MIMXRT1050 EVK Board Hardware User's Guide
  https://www.mouser.com/pdfdocs/NXP_MIMXRT1050-EVK_HUG.pdf
• NXP elQ™ Machine Learning Software Development Environment for i.MX Applications Processors
  https://www.nxp.com/docs/en/nxp/user-guides/UM11226.pdf

Machine Learning Technology Overview

The MIMXRT1050 Evaluation Kit from NXP (Figure 3) features a powerful i.MX RT1050 processor that is a custom implementation of an Arm® Cortex®-M7Core architecture.

Figure 3: Feature highlight of the i.MX RT1050/60 Development Boards. (Source: NXP)

Google offers several pre-trained neural network models that will work with the TensorFlow Lite (Figure 4) for Microcontroller SDK. They are available at: https://www.tensorflow.org/lite/microcontrollers/get_started

Figure 4: Google's TensorFlow Lite for Microcontroller website has many resources and pre-trained models ready for use. (Source: Google)

This project will go with the Magic Wand example. This is a gesture recognition model that will leverage the development kit's onboard 3-axis accelerometer that is found as part of the FXOS8700 compass microchip.

To deploy a TensorFlow model to a microcontroller, you will need to follow this development workflow to train Neural Network (NN) model (Figure 5) then convert it to a format that can work on a resource-constrained device such as a microcontroller.

1. Train the Neural Network: The TensorFlow framework allows a model to be trained on a desktop computer or using the horsepower of cloud-based services such as AWS or Google Compute Engine. The bottom line, the more processing power (CPUs and GPUs) you throw at training a NN, the faster or more robust the final model will be. The result of training a model with TensorFlow is a file with a .pb extension. Downloading a pre-trained model is another option.
2. Prepare the Model for Inferencing Hardware: The next step is to take the trained model and prepare it to run on the chosen endpoint device. The model must be made small enough to fit within the memory of the i.MX RT1050 after conversion. Additionally, the code can only use the functions that are supported by TensorFlow Lite for Microcontrollers. However, it is possible to use a function that is not currently supported if you write your own custom implementations.
   1. Convert the TensorFlow model to a TensorFlow Lite FlatBuffer: You will convert your model into the standard TensorFlow Lite format using the TensorFlow Lite converter. You might wish to output a quantized model because these are smaller in size and more efficient to execute.
   2. Convert the FlatBuffer to a C Byte Array: Models are kept in read-only program memory and stored in a C file. TensorFlow SDK provides tools can be used to convert the FlatBuffer into the appropriate C byte array format.
3. Integrate the TensorFlow Lite for Microcontrollers C++ Library: Write the “value-added” code for our particular implementation that will allow the project to interact with the accelerometer chip to collect acceleration data, perform inferencing utilizing the TensorFlow Lite for Microcontroller C++ library, make a prediction, and then display the results to the end-user via the serial terminal.
4. Deployment to the Edge: Build and deploy the program to your device using the MCUXpresso IDE.

Figure 5: The workflow to translate a TensorFlow model to a C byte array which can be handled by a memory-constrained device such as a microcontroller. (Source: NXP)

Building the Project

This project is predominately software-driven because the i.MX RT1050 development board already has the needed hardware onboard. You have multiple options when it comes to development environments, including:

• GNU toolchain for Arm® Cortex®-M with Cmake build system
• IAR Embedded Workbench
• Keil™ MDK-Arm

Figure 6: NXP Software Development Kit Builder Tool allow a developer to pick the libraries for their particular target board. (Source: NXP)

To begin, let us use the NXP MCUXpresso SDK Builder website to grab the needed SDK libraries for this project (Figure 6). If you are using a different toolchain, you can still use this site to select and download the libraries tailored for your preferred toolchain. The website for the SDK Builder is https://mcuxpresso.nxp.com/en/builder

Be sure to select the following SDKs:

• sdmmc: Contains the middleware for sd, mmc, sdio card stack
• elQ: This machine-learning SDK contains the ARM CMSIS-NN library (neural network kernels optimized for Cortex-M cores) and an inference engine for running TensorFlow Lite models
• ISSDK: Contains the IoT Sensing SDK (ISSDK), which provides the sensor drivers and reference examples for the onboard sensors, including the FXOS8700 magnetometer/accelerometer sensor.

Figure 7: MCUXpresso IDE is a convenient tool to edit your code and interact with the i.MX RT1050 development board. (Source: NXP)

For this project, we will leverage NXP's MCUXpresso IDE (Figure 7). If you intended to use the i.MX RT1050 development board, please be sure that you download version 11.1.1 of the IDE or greater. The MCUXpresso IDE can be downloaded here: https://www.nxp.com/design/software/development-software/mcuxpresso-software-and-tools/mcuxpresso-integrated-development-environment-ide:MCUXpresso-IDE

Figure 8: PuTTY allows a developer to interact with their target development board over a serial connection. (Source: MB Parks)

Lastly, we will download a small application called PuTTY (Figure 8) that will allow us to connect to and talk with the development board via a serial connection. The software can be downloaded here: https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html

Once we have all the software installed, we will begin by launching the MCUXpresso IDE (Figure 9). Then click on the button labeled Download and Install SDKs then select Install and Import Examples. This will open a screen to let us select SDKs we wish to install and the associated examples.

Next, we'll grab a copy of the source code from our GitHub repository. Open your command prompt, navigate to where you want to save the source code, then enter:

#git clone https://github.com/Mouser-Electronics/TensorFlowLite-NXP

Figure 9: The MCUXpresso IDE at the initial launch. (Source: NXP)

From the MCUXpresso IDE, click file > open and then navigate to where you cloned the repository and select the project folder. With that, we are already to examine the source code.

Software

Figure 10: Source code available on GitHub. (Source: Mouser)

The codebase for this project is written in C++. Twelve files of interest will be found in the Source folder of the project structure. Files with the .h extension are the header files that contain function definition and variable declarations. The files with the .cc extension contain the logic as defined by the various classes and functions found within each source code file. All source code is available on Mouser's GitHub repository (Figure 10). The list of key files include:

• main.cc*: This file contains the main loop that makes up this project. It is very simple and just calls the setup() function loop once and repeatedly calls the loop() function within an endless loop.
• main_functions.cc / .h: Contains the setup() and loop() functions responsible for most of the high-level functionality of this project.

The setup() function initialize the various data structures needed by the inferencing engine of the TensorFlow libraries. The loop() function has four major objectives:

• Read the accelerometer
• Send the data from the accelerometer to the inferencing engine
• Predict what gesture is being made
• Display the output prediction to the user over the serial port

You will notice the naming convention of setup() and loop() follows that of the Arduino programming environment because the TensorFlow examples are made to be compatible with various Arduino boards.

• constants.h: Sets various control points for the TensorFlow inferencing engine to find a balance between performance and accuracy. To see what these constants are and what they control, please see the subsection below titled Key Variable and Constants.
• output_handler.cc / .h*: Controls the text output to the user over the serial communications port.
• accelerometer_handler.cc / .h*: Contains the code to interface with the accelerometer found onboard the FXOS8700 microchip. Please note that acceleration data needs to be delivered to the NN model in a manner that requires the FXOS8700 to be configured in a mode that supports a FIFO data structure.
• gesture_predictor.cc / .h: Looks over the most recent predictions for each gesture and determines which is the most likely to have been performed by the user. Return that prediction to the calling function.
• magic_wand_model_data.cc / .h: Contains a C byte array representation of the TensorFlow model after being ran through a conversion process. The process takes a TensorFlow model and translates it to a TensorFlow Lite FlatBuffer and then ultimately creates a C byte array that is small enough to be efficiently handled by memory and storage constrained devices such as microcontrollers./li>

*These files have been replaced or modified with custom-written code for the i.MX RT1050 to handle the FXOS8700 accelerometer and serial communications.

**Also, if you download the original Magic Wand files from the TensorFlow Lite for Microcontrollers GitHub repository there are additional test files.

Key Variables and Constants

These are variables you might want to tweak depending on your particular design choices:

• const float kTargetHz = 25: Sets the sample rate of the accelerometer.
• constexpr int kGestureCount = 4: Sets the size of the rolling buffer of gestures to be used as part of the prediction algorithms.
• constexpr int kWingGesture = 0: A numerical value assigned to represent the wing gesture.
• constexpr int kRingGesture = 1: A numerical value assigned to represent the ring gesture.
• constexpr int kSlopeGesture = 2: A numerical value assigned to represent the slope gesture.
• constexpr int kNoGesture = 3: A numerical value assigned when no gesture is detected.

The following constants control the sensitivity of the gesture predictions based on the accelerometer data.

• constexpr float kDetectionThreshold = 0.8f: The score that the average gesture prediction score must surpass for the inference engine to say that a gesture has been made by the user.
• constexpr int kPredictionhistoryLength = 5: The number of gestures that must be detected to trigger a prediction.
• constexpr int kPredictionSuppressionDuration = 25: Sets a limit on how often a prediction is made. Larger the number, the longer the delay.

A note about the constant expression keyword (constexpr) that is used to define the variable a constant but does so at compile-time versus runtime as with the const keyword.

Building the Project

The NXP i.MX RT1050 development board has multiple operating nodes. To enable the functionality on the dev board needed for this project, a four-position DIP switch (SW7) must be set to off-off-on-off. Also, the device should be programmed and powered with the micro-USB port (debug port) that runs along the short, right side of the dev board.

Figure 11: Layout of the MCUXpresso IDE in edit mode. (Source: MB Parks)

Now it is time to build the project and then upload it to the development board. This is accomplished via the bottom left portion of the MCUExpresso IDE interface (Figure 11). To accomplish this first click on Build under the Quickstart Panel > Build your Project. Then, with the board plugged into the computer and the development board's debug USB port, click on Debug under Quickstart Panel > Debug our Project. This will open a dialog box that looks for the connected board. If found, it will populate the list found in the middle of the dialog box with your i.MX RT1050 product information. Click OK to proceed with flashing the board with your project .bin file.

Once the project is loaded, it will automatically start running.

Next, fire up PuTTY and establish a new serial connection. The settings for the connection should be as follows:

• Port: Check Windows Device Manager to determine which virtual com port has been assigned to the i.MX RT1050 after plugging it. For more information on how to do this, check out this link.
• Baud Rate: 115200
• Data Bits: 8
• Stop Bits: 1
• Parity: None
• Flow Control: None

Click Open and a serial terminal should be displayed. On the i.MX RT1050 development board click the Reset button. Physically move the development board in around. If successful, you will be greeted with the following prompt inside the terminal window (Figure 12).

Figure 12: Output will be displayed in a serial terminal. (Source: MB Parks)

Project in Action

Figure 13: Power the development board with a micro-USB cable plugged into the micro-USB port on the right side of the board. (Source: NXP)

With the board plugged in and the PuTTY terminal open, it is time to interact with our board. Be sure to use the longest micro-USB cable you have as the gesture movement might need to be a bit exaggerated. Remember to insert the USB cable into the debug port located along the right side of the development board (Figure 13) and not the two USB ports along the bottom of the board.

The model is trained to identify three different gestures. These include:

• Wing or the letter "W"
• Ring or a circle
• Slope which is basically two-line segments at a 45-degree angle

Moving the development board through X-Y plane (Figure 14) in a motion resembling the letter "W", a circle, or 45-degree angle will result in a triggering of the output to the serial terminal stating which gesture had been detected as seen in Figure 12.

Figure 14: Shows the i.MX RT1050 development board with an overlay of the X, Y, and Z axes as seen by the accelerometer onboard the FXOS8700. (Source: MB Parks)