Driver Assistance: Put a SoC in it

Architectural blocks for multiple camera system

Aaron Behman from Xilinx and consultant Adam Taylor discuss considerations for adas and the advantages of programmable SoCs

Increases in processing capability and the development of cmos image sensors and other sensor technologies have helped vehicle manufacturers introduce advanced driver assistance systems (adas). Adas enhances drivers’ awareness of the environment around them reducing the chances of collision. Some systems can also monitor drivers and alert them should they become sleepy, for instance. 

Increasingly adas also takes control or provides information to autonomous driving systems, providing assistance to the driver with capabilities such as parking assist, lane assist and adaptive cruise control.

It is therefore no surprise that the adas market is predicted to be worth $4bn a year by 2021 and is experiencing a ten per cent compounded annual growth rate.

Adas use a wide spectrum of sensors encompassing embedded vision, radar and lidar. Often, to extract the information required they use a sensor fusion approach combining information from several sensors. Within the embedded vision sphere, adas can be split further into two categories: external monitoring addresses aspects such as lane departure, object detection, blind spot detection and traffic sign recognition; while internal systems monitor aspects such as driver drowsiness and eye detection. Both internal and external adas applications bring with them challenges to address in implementing the image processing algorithms.

These challenges range from the ability to implement the algorithms required for the application, to complying with the correct automotive standards.

Many adas applications also require sensor fusion to combine the inputs from several sensors, significantly increasing the required processing power. Sensor fusion can be homogeneous where the multiple sensors of the same type are used, or heterogeneous where different sensor types are used to extract the information required.

Many applications use an all-programmable SoC or FPGA to implement the system due to the flexibility provided, both to implement the required algorithms but also due to the ability to interface with different sensor types and networks.

Along with performance, adas applications also come with several difficulties that may not be obvious at first. Vehicle manufacturers have stringent pollution standards to achieve and therefore the weight and power consumption of the overall system is important.

The cost is also critical due to the many tens or hundreds of thousands of vehicles being produced. While security and safety of the system are very critical and governed by standards, using a SoC or FPGA can help us address a number of these.

 

System architecture

The development of embedded vision that monitors both external and internal cameras can be seen to be one of the more difficult adas implementations. This needs to interface to several cameras around the vehicle, process the images and provide the information to the occupants.

Many camera systems use point-to-point LVDS wiring to transfer the data, however this brings with it additional cost and weight in the cabling required. There are though alternative approaches that are increasingly gaining ground, pushing some functionality into the camera itself. If the image output by the camera is compressed and not the raw image, then network based architectures are possible. These networks could be based around commonly used automotive buses such as:

 

  • Most (Media Oriented Systems Transport): A high speed network that can be implemented in either optical or electrical physical layers;
  • IDB-1394: High speed network implemented over an electrical physical layer and implemented in a daisy chained topology; and
  • Ethernet AVB (audio visual bridging): This provides the ability to route image data and other data around the vehicle as needed.

 

When using a network, the system architect must ensure the necessary bandwidth is available to transfer the image data between the camera and adas core with the required latency for its application.

The data generated by the adas may need to be shared with other systems within the car, for instance adaptive cruise control or parking assist. Therefore, the adas must be able to interface to other commonly used automotive interfaces such as Can or Flexray.

At the architectural level, the use of an all-programmable SoC approach provides several advantages. If point-to-point wiring is used to interface to the cameras, the camera receivers can be implemented in the programmable logic prior to the image processing chain. If a programmable SoC were to be used, the flexibility of the processor system side would allow for the easy inclusion of a Can, Ethernet and other protocols such as Flexray when combined with logic in the programmable logic and an external phy if necessary. The combination of dual-core processors and programmable logic allows for a very low power per pixel, as the system is very tightly integrated.

 

SoC architecture

Where the SoC comes into its own is the use of programmable logic to implement camera interface and image processing chain, while the processing system of the SoC provides the communications, control and additional algorithmic processing as required. The image processing pipeline can be generated using IP libraries and more specialist IP cores.

To develop additional algorithms, high-level synthesis tools can accelerate the development. Rather than developing the IP using a traditional hardware description language, a higher level language such as C or C++ can reduce the time to market.

Further acceleration of the algorithm can be achieved using open source frameworks. Algorithms developed in this way can be mapped into a high-level synthesis video library. This eases the transition from proof of concept and demonstration to the algorithms running in the target hardware for characterisation and qualification.

Many implemented architectures will use the processor DDR memory as a frame buffer; this enables the processor to access the images as necessary for further onwards transmission if using an Ethernet based system or PCIe, for example. The processor can perform additional image processing algorithms on the images stored within the DDR before re-insertion into the image processing chain.

This provides a very interesting capability that the SoC itself can form its own prototype and demonstration platform. Using common embedded vision development frameworks running on the cores within the processor, this provides for a down to size prototype system. This system can then be optimised for performance using the programmable logic side of the SoC.

 

Safety and security

The very nature of adas is that it contributes to automotive safety, and as such systems must be developed in line with a standard as with many other high reliability systems.

For adas, the applicable standard is ISO 26262. This defines a number of automotive safety integrity levels (Asils) that define the time to failure in hours. There are four Asils with D being the highest and most difficult to achieve and A the lowest. 

Achieving these requirements requires an integrated engineering delivery lifecycle to ensure not only the target Asil is achieved, but the data pack generated also demonstrates this by containing the necessary evidence.

Automotive applications are subject to harsh environments, and as such the developers need to ensure they use automotive grade components, for instance one certified to AEC-Q100, which has been manufactured and qualified to a higher standard than commercial components.

They must also consider the security of the system that is preventing unauthorised people from making modifications to the system, as this could potentially lead to catastrophic results.

Using a programmable SoC provides the ability to use its secure boot capability to prevent unauthorised programmes and bit streams from being configured into the system.

 

Conclusion

At both the system and device level, the use of a programmable SoC offers a number of benefits providing for greater levels of integration, allowing for a more weight and power efficient system. Modern high-level synthesis tools enable the rapid development of image processing functions that can be used within many open-source commonly used frameworks.

Aaron Behman is director of strategic marketing at Xilinx and Adam Taylor is an embedded systems consultant

www.xilinx.com

 

Recommend to Colleagues: