Guest blog: Driving on the edge

Gianfranco Comune, senior field applications engineer with Arrow Electronics

Gianfranco Comune from Arrow Electronics explains how machine learning can help with semi-autonomous driving

With machine learning and the broader field of artificial intelligence generating so much attention, we may be sharing the road with self-driving cars sooner than expected. However, the idea of being a passive passenger isn’t for everyone.

For many, driving isn’t just a task. It’s a passion, an expression of individual freedom and personal skill. But it is also a privilege and it is not available to everyone. This is especially true for people with physical disabilities.

Fortunately, the future of driving doesn’t have to be all or nothing. Technology allows for an intermediate option where a human still calls the shots. The significant difference is how it happens. This is a new category known as semi-autonomous control.

A semi-autonomous vehicle is still directly driven by computers, defined as embedded systems, just like an autonomous vehicle. However, a human operator provides the driving input rather than an algorithm.

There are different elements that add up to driving, and each requires its own method under the semi-autonomous title.

Acceleration and braking can be reliably executed using a simple breath control system known as sip and puff. A pressure sensor measures either the intensity of exhalation (puffing) through a tube to accelerate or inhalation (sipping) to brake. The sensor then sends a signal to a microcontroller which communicates the driver’s commands to the vehicle’s embedded computer.

Sip and puff technology has drawbacks. Hard or continuous operation taxes the operator’s breathing, resulting in driver fatigue and safety concerns. On top of that, fluid saturation often occurs within 20 minutes of use.

More advanced technologies are in development, such as wireless mouth pads, designed to fit inside the mouth and operated by tongue gestures.

Steering is another fundamental element of driving. For the past decade, semi-autonomous steering has been controlled by head movements – specifically yaw angle – through infra-red cameras positioned inside the vehicle. The cameras process the yaw angles of the operator’s head by tracking reflective sensors placed on a helmet or sunglasses. The inputs from the cameras are relayed to a processing system which in turn relays the commands to the vehicle’s steering system in real time.

Like the sip and puff, this system has drawbacks. First, the processing system is a proprietary and costly system. Moreover, multiple cameras are required to triangulate the infra-red sensors and calculate the angle, which reduces the driver’s field of view.

The cameras require a fixed body with sensors to track. So not having that means no driving. Finally, the sensors must be calibrated often, especially for multiple drivers, to meet the stringent precision and real-time requirements for safe vehicle operation. This results in an expensive and high-maintenance system.

Engineers began developing an innovative version of semi-autonomous steering in 2022 based on AI. The idea seemed simple; replace the bulky, costly, proprietary IR steering system with easily accessible off-the-shelf technology and a specific AI technique, called deep learning. Being a first time-ever use case for this technology, there was no pre-trained model, nor a public dataset to train the model.

The effort began by finding an open-source head pose model that could learn the functionalities needed for steering. Engineers optimised the model through meticulous training to achieve real-time responsiveness, just like the previous infra-red based system. The training process required a dataset with the precise yaw steering angles. To build the dataset, the team created a 3D model of a human head, and then built a complex motor-control system to capture the yaw angle as the ground truth data.

The framework was built on OpenCV. For the inferencing, a popular face detection model to isolate the driver’s head and a modified public head-pose model for yaw angle were incorporated.

Selecting hardware was easier with the decision falling on the Nvidia Jetson. Nvidia GPUs are a huge force in machine learning and the ideal off-the-shelf option. The prototype was developed around the Jetson Xavier AGX development kit, which provided the necessary processing power for such a compute-intensive system. The IR cameras were replaced with an off-the-shelf, high-resolution, high frame rate stereo camera, connected through a Gigabit Multimedia Serial link, or GMSL, cable directly to the Jetson.

Initial driver testing was successful at low to medium speed and, as planned, it removed the need for a fixed body and continuous calibration, allowing anyone to drive the vehicle. For improved accuracy (mostly for helmet steering), the engineers improved the head pose model, expanded the dataset with helmet data, and improved jitter filtering. Also, to increase performance, the engineers deployed a rugged Nvidia Orin AGX based system. It enables the system to execute sixty frames per second inferencing, with high accuracy and, more importantly, better real-time response.

Through these optimisations, an operator can steer using head controls at high speeds with or without a helmet. This also makes the system potentially suitable for auto racing, flying, boating and other performance activities.

The concept of semi-autonomous mobility was introduced for a US professional racer who suffered a severe spinal cord injury to be able to drive again. Engineers – including myself –worked to build accessible open-source semi-autonomous control systems and tested them in high performance cars.

However, this technology extends beyond driving a car. It can be adapted to semi-autonomous operation of farm and construction machines, warehouse equipment such as forklifts and box carts, or large industrial human-machine interface (HMI) systems. This would allow disabled workers to work in sectors that were historically inaccessible to them. It also would allow able-bodied workers to control machines with head controls, freeing their hands for other job tasks while the equipment is operating safely.

Gianfranco Comune is senior field applications engineer with Arrow Electronics

www.arrow.com

Recommend to Colleagues: