Internet-Connected AI Face Tracker

The Seeed Studio Sipeed Maixduino Kit for RISC-V AI + IoT is a rapid development platform for quickly developing artificial intelligence (AI) and Internet of Things (IoT) applications using a dual-core 64-bit RISC-V CPU coupled with a 400MHz neural network processor. When used with a digital camera, it supports machine vision edge computing applications based on convolutional neural networks. The Medium One IoT Platform is a cloud-based IoT platform designed to help early-stage developers prototype IoT projects or connect existing hardware to the cloud. In this project, we'll set up an AI machine vision application using the Maxiduino to collect real-time face detection readings and send them to Medium One, where the data can be processed and viewed using programmable workflows and configurable dashboard widgets.

Project Materials and Resources

The project’s bill of materials lists components used in this project. Additional hardware and software development tools are also identified.

Project Bill of Materials (BOM)

• 713-110110044 - Seeed Studio Sipeed Maixduino Kit for RISC-V AI + IoT
• 349-M1IOTPROTO – One-year subscription to the Medium One cloud environment

Hardware

• 802.11 b/g/n Wi-Fi access point with a Dynamic Host Configuration Protocol (DHCP) server, internet connectivity, and without a firewall or proxy that blocks outgoing traffic to the internet
• Personal computer (PC) running Microsoft Windows^®, macOS or Linux
• USB Type-A to Type-C cable

Accounts and Software

• Web browser for accessing software download sites and portals
• Login account for the Medium One IoT Platform
• Sipeed MaixPy Integrated Development Environment (IDE) and libraries
• Sipeed MaixPy firmware update and vision models available from the Sipeed download site
• Application project files available in a GitHub repository
• A serial terminal program such as Tera Term on the PC
• Wi-Fi access point connection details, including Service Set IDentifier (SSID) and password

Project Technology Overview

Seeed Studio Sipeed Maixduino Kit for RISC-V AI + IoT

The Seeed Studio Sipeed Maixduino Kit for RISC-V AI + IoT (Figure 1) is a development board based on the Sipeed MAIX module featuring a dual-core 64-bit RISC-V CPU with floating-point unit coupled with a 400MHz neural network processor. The neural network processor enables AI applications using loadable neural networks either pre-built or custom-developed using several deep-learning toolsets. An onboard ESP32 Wi-Fi + Bluetooth module provides wireless connectivity, and camera and LCD connectors allow you to integrate a digital camera for machine-vision applications along with an LCD for images, text, and graphics. The onboard digital microphone supports speech recognition and sound processing applications, and an audio output connector allows a speaker to be connected for sound output. Power is provided through either a USB Type-C cable or a DC input connector.

Figure 1: The Seeed Studio Sipeed Maixduino Kit for RISC-V AI + IoT is shown. (Source: Mouser Electronics)
 

The board has an Arduino Uno form factor (Figure 2) with Arduino-compatible connectors allowing expansion using Arduino shields containing other types of sensors, actuators, or communications interfaces.

Software development is supported using MicroPython, Arduino IDE, PlatformIO IDE, OpenMV IDE, and deep-learning tools, including Tiny-Yolo, Mobilenet, and TensorFlow Lite.

More detailed information about the Maixduino can be found here.  

Figure 2: The Maixduino board layout with Arduino Uno Form Factor is shown (Source: Mouser Electronics)
 

MaixPy and MaixPy IDE

For this project, we'll use MaixPy to develop application software for the Maixduino. MaixPy is MicroPython ported to the K210 processor contained in the MAIX module. It contains pre-built library packages that support board operation, including initialization, input/output peripherals, and sensor data processing. Source code for MaixPy is available on GitHub and example MaixPy programs are available here.

The MaixPy IDE (Figure 3) provides an editor and downloader for developing and running MaixPy applications. It runs on Windows^®, macOS, or Linux PCs and contains built-in tools for observing the digital camera data stream.

Figure 3: The MaixPy IDE is shown. (Source: Mouser Electronics)

Project Application Source Code Files

For this project, the MaixPy IDE created a set of project source code files that initialize the Maixduino, connect to Wi-Fi, run the face detection algorithms, and send face-detection information to the Medium One IoT Platform through Wi-Fi.

Medium One IoT Prototyping Platform

The Medium One IoT Platform (Figure 4) is designed to help early-stage developers prototype their IoT project or connect their existing hardware to the cloud. It offers an IoT Data Intelligence platform enabling customers to build IoT applications with less effort quickly. Programmable workflows allow you to quickly build processing logic without having to create own complex software stack. Configurable dashboards will enable you to visualize application data and view real-time data in various formats. Medium One’s iOS and Android apps allow you to build simple mobile app dashboards to communicate with your devices through the IoT platform.

Figure 4: The Medium One IoT platform is shown (Source: Mouser Electronics)

IoT devices can exchange data with Medium One through either a REST API or MQTT. More detailed information about the Medium One IoT Platform can be found here and on the Medium One site.

The Setup (Hardware)

While setting up the hardware, remember that electronic components are static-sensitive, so handle accordingly.

Personal Computer (PC)

Power up the PC and allow it to boot up.

Wi-Fi Access Point

Make sure your Wi-Fi access point runs with an active connection to the internet and a DHCP server that assigns IP addresses. You’ll need to know the access point SSID and security credentials to be used later when configuring the Wi-Fi connection.

Maixduino Kit

Unbox the Maixduino Kit and lay out the components. On the main Maixduino board, look for a four-digit revision number next to the USB-C connector. This project was developed using board revision 2737, and you might have the same or later revision that has additional features such as an onboard RGB LED.

Connect the LCD and camera flat ribbon cables to the Maixduino connectors following the orientation shown in Figure 5. Note that the camera is facing down, and the LCD is facing up. Remove the protective covering from the camera lens and remove the protective cover on the LCD if desired.

Figure 5: Connecting the LCD and camera to Maixduino board is shown. (Source: Mouser Electronics)

Connect a USB Type-A to Type-C cable from the PC to the Maixduino. The board should power up, and you should see activity on the LCD.

The USB C interface has two virtual serial ports that appear to the PC as "Sipeed-Debug (Port 1)" and "Sipeed-Debug (Port 2)." Port 1 is connected to the K210 CPU chip, and Port 2 is connected to the ESP32 module. Port 1 is used to interact with the K210 processor to load and run programs and to update the firmware. Port 2 is used for updating firmware on the ESP32 module.

Start your serial terminal program (Tera Term) and open "Spieed-Debug (Port 1)" at 115200,N,8,1. Press the enter key, and you should see a ">>>" prompt, which is the built-in MicroPython interpreter. Press the Maixduino RESET button, and you see startup messages on the serial port as the board reboots.

Update MaixPy Firmware

Follow the instructions here to update the MaixPy firmware running on the board. You can upload new firmware using either the kflash_gui tool (GUI based) or the kflash command-line tool. Install one or both of the tools on your PC. MaixPy firmware releases are available on the Sipeed download site in the folder named MAIX/MaixPy/Release/master. Multiple releases of the MaixPy firmware are available, and for this project, maixpy_v0.6_1_gfa51290.bin was used. Download the .bin file and save it to your computer, then upload it to the Maixduino using either kflash or kflash_gui as described in the instructions. If the firmware upload fails with timeout errors, try again using a slower baud rate such as 115200. You can verify the new version of firmware by connecting your serial terminal program and pressing the RESET button to print the startup banner.

The kflash command line for loading maixpy_v0.6_1_gfa51290.bin on a Linux PC was this, using a lower baud rate:

$ kflash -p /dev/ttyUSB0 -b 921600 -B maixduino maixpy_v0.6_1_gfa51290.bin

Load Face Recognition Model

This step loads a pre-built neural network face recognition model into Maixduino flash memory so the neural network processor can use it. On the Sipeed download site, navigate to the MAIX/MaixPy/model folder and download the face_model_at_0x300000.kfpkg file. Program the file onto the Maixduino using either kflash or kflash_gui. The kflash command line for loading face_model_at_0x300000.kfpkg on a Linux PC was this, using a lower baud rate:

$ kflash -p /dev/ttyUSB0 -B maixduino -b 115200 face_model_at_0x300000.kfpkg

The Setup (Software)

MaixPy IDE

Follow the instructions here to download MaixPy IDE and install it on your PC. Note that the instructions also refer to a LED blink program. However only later hardware revisions of the Maixduino board contain an onboard RGB LED.

After setting up Medium One, we'll come back to MaixPy.

Set Up the Medium One IoT Cloud

Web browse to the Medium One IoT Platform and log in, after which you should see an initial dashboard resembling Figure 4. If you have trouble logging in, check your Medium One credentials to make sure you're using the latest URL provided with your sandbox subscription. Click Setup > Manage Users > Add New User. Set Username to mydevice, create a password of your choosing and enter it in both password fields, then click Save. In the Manage API Users list, you should see a new user account having Login ID = mydevice and an auto-generated user MQTT ID like Figure 6.

Figure 6: The newly created User ID with auto-generated MQTT ID is shown. (Source: Mouser Electronics)

Click Setup > MQTT, and you should see a Project MQTT ID and a set of port numbers like Figure 7.

Figure 7: Project MQTT ID (Source: Mouser Electronics)

Medium One uses MQTT usernames and passwords for authentication. The MQTT username is created by combining the Project MQTT ID, a forward slash, and the user MQTT ID. For example, if the Project MQTT ID is “ZHxxxxxxx0Y” and the user MQTT ID is “sTxxxxxxx8w” the corresponding MQTT username would be “ZHxxxxxxx0Y/sTxxxxxxx8w”.

Next, we’ll create the MQTT password. Navigate to Setup > Manage API Keys > Add New API Key. Set the description to mydevice, make sure Enabled is check-marked, and click Save. The result should look like Figure 8.

Figure 8: The newly created API key is shown. (Source: Mouser Electronics)

The MQTT password is created by combining the API Key, a forward slash, and the mydevice user password. For example, if the API Key is “PZxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxMBQ” and the mydevice user password is “AaaaBbbb3” the corresponding MQTT password would be “PZxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxMBQ/AaaaBbbb3”.

The MQTT topic has the following format: “0/Project MQTT ID/User MQTT ID/Device ID”.

The Device ID field can be anything, and we’ll use "mydevice" as the Device ID. For example, if the Project MQTT ID is “ZHxxxxxxx0Y” and the user MQTT ID is “sTxxxxxxx8w” the corresponding MQTT topic would be “0/ZHxxxxxxx0Y/sTxxxxxxx8w/mydevice".

The MQTT username, MQTT password, and MQTT topic strings will be added to the next step’s project source code.

Download and Open the Project Application Source Code File

Web browse to https://github.com/IoTDevLabs/ma21_files and find the Maixduino_MediumOne_1.0.0.zip file. Download the file to your computer and unzip it to produce file find_face_mediumone.py.

Open find_face_mediumone.py in the MaixPy IDE and update these string variables with your Wi-Fi SSID and password along with your own Medium One MQTT username, password, and topic strings as described earlier:

• WIFI_SSID
• WIFI_PASSWORD
• MQTT_USERNAME
• MQTT_PASSWORD
• MQTT_PUB_TOPIC

Save the modified file.

Run the Application

Click Tools > Select Board and checkmark Sipeed Maixduino. Click the green Connect button (chain-link icon) in the lower-left corner of the MaixPy IDE window to connect to the Maixduino board through the USB-C serial cable (Port 1). Click the Serial Terminal label along the bottom of the MaixPy IDE window to open a serial terminal monitor to view Maixduino status messages. Click the green Run button underneath the link icon to start program execution.

You should see status messages in the Serial Terminal window about connecting to Wi-Fi, pinging google.com, and an initial message published to Medium One. The LCD should show what the camera is seeing along with the frame processing time (t:XXms) and the number of faces detected (f:X) within the camera field of view. Point the camera at one or more faces, and you should see the face count increase along with boxes outlining each detected face. Photographs of people's faces should also work.

How the Application Program Works

This program is derived from the demo_find_face.py example program provided in the MaixPy examples. It was extended to add Wi-Fi and MQTT classes, the logic for connecting to Medium One, and message publishing when the face count changes. The high-level program flow includes:

• Connect to Wi-Fi
• Ping google.com to verify internet connectivity
• Connect to Medium One
• Initialize hardware and camera
• Load neural network model
• Periodically read camera and face detection model
• If the number of detected faces changed since last time, publish a JSON formatted MQTT message to Medium One with the new face count
• Display the camera image and overlay information on the LCD

NOTE: The MQTT code currently uses unencrypted MQTT, so the information sent through the internet is not protected by encryption.

MQTT Payload Format

MQTT messages are formatted as JSON strings according to the Medium One MQTT payload specification. Here’s an example message:

{"event_data":{"face_count":1,"timestamp":31684}}

Fields:

• face_count = number of faces detected within the camera field of view
• timestamp = milliseconds since the board started

View Data in the Medium One Dashboard

In the Medium One dashboard, navigate to Data Viewer > Data Streams and click raw Events. You should see raw messages (Figure 9) being received from the Maixduino. Click the “+” sign to view message details.

Figure 9: The raw message display is shown. (Source: Mouser Electronics)

Click Dashboard on the top left, then click Add Widget > Single User Real Time Events Stream to add an event stream widget to the dashboard.

In the Select user dropdown, select mydevice. You should now see messages appearing in the Real Time Events Stream widget (Figure 10).  Click the save icon in the upper right corner to save your modified dashboard.

Figure 10: Real-time events stream widget display is shown. (Source: Mouser Electronics)

Add More Widgets

To display more widgets, we need to enable specific data fields contained in the message payload. Navigate to Config > Data Streams and click on raw Events. The Schema Map should be pre-populated with fields detected in the incoming messages. However, they are currently disabled. Check-mark the Active box on raw.face_count and raw.timestamp, then click Save Data Stream. These fields are now available for use in other dashboard widgets.

Back on the dashboard, click the Single User Last Value Table widget and select the widget’s mydevice user. Click the widget’s Tag Config icon to the right of the mydevice user selection and check-mark raw:face_count and raw:timestamp, then click Save. The Last Value Table should now populate with the most recent received values for each field (Figure 11). Click the save icon toward the upper-right corner to save the updated dashboard.

Figure 11: Last value table widget display is shown. (Source: Mouser Electronics)

Now let’s add a dashboard widget for the face_count. Click Single User Real Time Line Chart and select the mydevice user. Click the widget’s Tag Config icon and check-mark the raw:face_count row, then click Save. The updated dashboard should look like Figure 12. Click the dashboard save icon to save the updated dashboard. Put one or more faces in front of the camera to see the face_count value change on the real-time graph.

Figure 12: The real time line chart widget is added to dashboard (Source: Mouser Electronics)

At this point, your Maixduino board is running continuously reading the camera, running the face detection model, and transmitting face detection counts to the Medium One cloud. Figures 13 and 14 show the completed project running with the USB-C cable connected and a piece of tacky putty holding the camera in position. Two faces are being detected in Figure 14.

Remember to power off the Maixduino board when you’re done. Otherwise, the board can continue to send messages to Medium One and consume daily message allotments.

Figure 13: Completed project shows the board, camera, and LCD. (Source: Mouser Electronics)

Figure 14: The completed project shows two faces detected on the LCD. (Source: Mouser Electronics)

Troubleshooting Tips

• Make sure the camera and LCD cables are connected properly.
• Make sure the protective covering has been removed from the camera, and the camera lens is focused.
• Make sure there is adequate lighting in the camera field of view.
• Make sure the MaixPy firmware has been updated.
• Make sure the Wi-Fi and Medium One settings are correct, including the MQTT port number.
• View the camera image on the LCD or in the MaixPy IDE to see what it's seeing.
• View information messages on the MaixPy Serial Terminal serial port.

Conclusion and Where to Go Next

This project created an internet-connected AI face detector that sends real-time sensor data to the Medium One IoT Platform. It can be modified and extended in several ways, and here are a few examples:

• Dive deeper into the application code and board hardware by reading the Sipeed documentation and studying the source code.
• Explore other pre-built image recognition models available on the Sipeed site.
• Try the Arduino or PlatformIO IDE development environments.
• Connect an Arduino-compatible sensor or actuator shield and integrate it into your application.
• Add more widgets to the Medium One dashboard.
• Learn about the Medium One Workflow Studio, which lets you create data processing workflows to transform your sensor data.
• Experiment with the Medium One mobile apps.