Authors: Jordan Rose, Field Application Engineer at Rutronik, and Stephan Menze, Head of Global Innovation Management at Rutronik
Multimodal interface based on PSOC
Touch, voice, and gestures: a demonstrator shows how modern human-machine interaction can be implemented in confined spaces. Embedded intelligence combined with flexible interfaces enables intuitive and highly responsive handling concepts.
The way people interact with machines is evolving rapidly. Although touchscreens are already part of everyday life, contactless operation concepts are becoming increasingly important, especially in applications where hygiene, environmental conditions, or physical limitations play a significant role.
The demonstrator presented (Figure 1) shows how different sensing principles—radar, voice, and touch—can be combined in a single embedded system to create a robust human-machine interface (HMI). The aim was to demonstrate the reliability of the recognition system against external influences such as bright sunlight, rain, ambient noise, or situations where hands are dirty or gloves are worn. Developed within the framework of internal and customer projects, the demonstrator serves as a practical basis for transferring knowledge about hardware, software, and algorithms for gesture recognition using radar and voice control.
General system architecture
The demonstrator brings together all the essential elements of a modern multimodal human-machine interface in a compact design. The integration of gesture and voice control, motor control, and graphical visualization requires a carefully coordinated architecture capable of managing diverse sensor and actuator interfaces, as well as supporting real-time parallel processing.
One of the main challenges was integrating heterogeneous components with different electrical interfaces—from high-speed display connections to latency-critical sensor inputs—onto a single microcontroller platform. The Infineon PSoC Edge provides the processing power and peripherals necessary for signal processing and control. The FreeRTOS real-time operating system coordinates individual tasks and manages data flows and control commands via the internal AHB bus.
| Alternative | Property | Function |
| BLDC motor with Hall sensors | Control via PWM and GPIO (Hall sensors) | Speed and direction of rotation adjustment; rotor position feedback |
| Infineon IM69D130 XENSIV Digital MEMS Microphone | Connection via PDM digital interface | Audio signal capture for keyword spotting |
| Raystar 1024×600 IPS TFT LCD Touch Screen | MIPI DSI connection, capacitive touch technology | System status visualization and touch operation |
| Infineon BGT60TR13C 60 GHz Radar | Connection via SPI and GPIO | Gesture recognition through magnitude, distance, and azimuth analysis |
| Infineon IFX007T Motor Control Board | Triple half-bridge module | BLDC motor power control |
| Infineon PSOC Edge Evaluation Board | Multicore microcontroller with NPU (Neural Processing Unit) | Centralized processing of radar, audio, visualization and motor control signals |
Table 1. Main components of the demonstrator
Gesture recognition using 60 GHz radar
Gesture recognition is the demonstrator's core control element. Left and right movements, which increase or decrease the motor's speed, are detected, as well as a "push" gesture to stop it. The system's core is a 60 GHz FMCW radar sensor for motion detection. All signal processing is performed on the PSoC Edge's Cortex-M55 core. Since no machine learning is used, development time is reduced and training is eliminated.
Gesture recognition process (Figure 2):
• Input data: magnitude and AoA (angle of arrival), using only the azimuth, since currently only the left and right directions are distinguished.
• Capture: 60 GHz FMCW radar with one transmitting and three receiving antennas, providing an independent signal per antenna.
• Motion detection: FFT Doppler for each antenna signal in order to identify moving targets and suppress static objects.
• Direction determination: calculation of the azimuth angle from the phase differences between the receiving antennas.
• Gesture classification: analysis of the temporal evolution of the azimuth angle to detect movements such as “swipe left”, “swipe right” or “click”.
Figure 2. Gesture recognition using the Infineon BGT60TR13C radar. (Source: Rutronik System Solutions)
The latency is approximately 10 ms after the movement ends. Gestures can be recognized at distances between 5 and 30 cm, or even greater if the system is properly configured, even under complex conditions of reflection and typical environmental factors such as sunlight. Adding elevation (vertical angle) would allow for the recognition of additional gestures such as upward or downward movements.
Gesture recognition is complemented by voice control based on keyword spotting. Voice signals are captured by a MEMS microphone, preprocessed in the PSoC Edge's Cortex-M55, and evaluated by a trained neural network. This network, composed of multiple convolutional layers, is optimized to recognize a limited set of clearly defined keywords, such as "start" or "stop."
The model was developed in Python using Keras and TensorFlow and subsequently adapted to the PSOC Edge using Infineon's ML Configurator. Inference is performed on the Cortex-M55 using an optimized TensorFlow Lite Micro.
Keyword spotting process (Figure 3):
• Audio capture: The digital MEMS microphone (16 kHz) provides PDM data.
• Preprocessing: conversion to MEL filter bank by time segmentation (≈530 µs).
• Inference: evaluation of MEL spectra using a multi-layer convolutional CNN.
• Result: The recognized keyword is transmitted as a control command to the motor or other system functions.
Figure 3. Keyword spotting process. (Source: Rutronik System Solutions)
BLDC motor control with Hall sensors
A brushless DC motor (24 V, maximum 4.800 rpm), controlled directly by the microcontroller, provides an immediate response to gesture and voice commands. Thanks to integrated Hall sensors, the system detects the current speed and adjusts it according to the commands received.
The Infineon IFX007T triple half-bridge module is used for control, driven by PWM signals and digital control lines. Speed regulation is performed at a sampling frequency of 1 kHz, ensuring fast and precise changes.
Touchscreen user interface
System status is displayed via a 7-inch capacitive touchscreen with a resolution of 1.024 × 600 pixels. Among other data, the display shows motor speed, recognized gestures, and voice control status.
The graphical interface is generated directly on the microcontroller using the open-source LVGL library. Efficient use of memory and processing resources allows for the simultaneous execution of the graphical interface, gesture recognition, and real-time speech processing. The refresh rate of approximately 10 FPS is sufficient for displaying statuses and providing operational feedback.
Specific challenges and lessons learned
A key feature of the demonstrator is the direct comparison between two approaches: conventional signal processing and machine learning. In the case of gesture recognition using 60 GHz radar, machine learning was deliberately omitted because it was not functionally required. This enables robust, fast, and training-free detection, as well as high immunity to external light, acoustic interference, and variations in hand position.
In contrast, voice control using keyword spotting employs a neural network that has been pre-trained and optimized for a limited set of keywords. In this area, machine learning unleashes its full potential, enabling accurate responses to clearly defined, recurring events. The algorithms rely on publicly available datasets to enhance recognition stability.
This hybrid approach—using classical signal processing where it provides speed and robustness, and machine learning where it improves recognition capabilities—demonstrates how different methods can be combined to develop a practical and versatile HMI solution.
Another objective was to demonstrate that all functions—gesture recognition, voice control, motor control, and graphical visualization—can be fully implemented on a single microcontroller. Achieving this required seamless hardware and software integration, as well as real-time processing of multiple sensor data streams within the limited resources of an embedded platform. This involved coordinating different interfaces, minimizing latency, and efficiently prioritizing data. The modular architecture and strict functional separation allow the system to be flexibly adapted to different applications and provide developers with a ready-to-use foundation.
Perspectives and transfer to real-world applications
Combining radar, audio, and engine control into a single system not only demonstrates technological feasibility but also provides a practical platform for knowledge transfer. Customers benefit from ready-to-use software samples that allow them to conduct their own tests or rapidly develop custom applications, significantly reducing implementation times.
The demonstrator can be used as a reference platform and adapted to specific projects. It is also suitable for special environments such as cleanrooms or glove boxes. The hardware and software platform, along with supplementary resources—sample code, electrical schematics, application notes, and instructions—can be provided upon request. Specific adaptations are also possible, for example, by integrating new functions, modifying speech recognition, or expanding gesture recognition capabilities.
An example of transferability is the implementation of a neural network on an RDK2 platform in conjunction with a RAB3 radar. Once the principles of data acquisition, training, and deployment of neural networks are understood, these methods can be easily transferred to other platforms. Infineon's ecosystem of tools supports this process and simplifies portability.
Furthermore, future expansions in the area of radar technology are planned to develop new use cases and broaden the functional range. Therefore, the demonstrator is not only a current technology platform but also an open foundation for the development of intelligent, sensor-based HMI solutions for embedded environments.
