Functional Safety: Making a stand

In the light of the ISO 26262 functional safety standard, William McCaffrey from Vector Software discusses building an automotive test stand for software
Rigorous stand-alone testing of the many physical components of automobiles is done as a standard part of the development process to allow manufacturers to work out the potential problems of a model before it goes into full production. After all, it is conventional wisdom that it's much less expensive to eliminate a problem before mass production begins than to discover a problem and have to fix it afterwards.
When developing a key structural component of a vehicle such as an engine, it is customary for an automotive manufacturer to use a standard testing framework in the form of an engine test stand to develop, characterise and test those engines. For example, in the development phase most automobile OEMs are interested in designing engines with optimal fuel efficiency, durability and that are in compliance with all relevant legislation on emissions and so on. An engine test stand allows the engine to operate in many different modes and offers measurement, control and recording of the associated physical variables – a process required to catalogue the many parameters that affect the engine’s design and performance.
Because there are many factors potentially involved, it is difficult to take them all into account during development, and a large number of tests are required to determine what may be needed to improve the engine’s performance. What makes all this testing possible is that engine test stands are automated test platforms that run specified tests to achieve the goals of the manufacturer.
Automated tests can run quickly and more frequently, which is also cost effective and has been proven to improve the development process.
A sophisticated engine test stand framework typically consists of actuators that allow control of the engine state, sensors that measure physical variables of interest and data acquisition systems to log those data. The same types of test stands are also built for other key systems such as transmissions. Would a car manufacturer skip this step and not fully test an engine or transmission prototype? Of course not, that would be a recipe for disaster.
Today’s automobiles have evolved from simple mechanical devices into incredibly complex machines, with software now powering all major systems of the vehicle including engine control, powertrain, transmissions, steering, braking and entertainment. This powerful software is as important – if not even more important – as the vehicles’ metal.
Yet statistics show that more than 50 per cent of auto recalls are now due to software bugs, not mechanical issues. Consider that the industry average is eight bugs per thousand lines of code, and the average modern high-end car has ten million lines of code. Because of this, shouldn't the components of the software systems in vehicles be isolated in similar frameworks? Software quality needs to be at the top of the list for automotive OEMs – and the vehicle’s software also needs the analogous test stand to simulate the real world environment.
Automated software testing can essentially provide exactly this type of test stand or simulation environment for software, automatically enabling companies to build test frameworks efficiently for all of their application components.
ISO 26262
One such framework to work with as a basis for automotive testing is ISO 26262, which covers the full engineering development cycle and defines functional safety requirements for automotive electrical and electronic systems. Functional safety in accordance with ISO 26262 encompasses all systems and subsystems in vehicles, including software, ensuring the avoidance of unreasonable risks and hazards caused by potential malfunctions in electrical and electronic systems.
ISO 26262 has gained increasing importance, but even despite the improved efforts to provide safety-compliant systems, the automotive industry still lacks global compliance standards. According to Gartner’s 2015 Hype Cycle for Automotive Electronics, the key issue that needs to be addressed is the gap in cross-functional synergy within competencies, the root cause of most product development issues. The report states that the lack of system and vehicle-level experts capable of defining the system and product requirements and deriving the functionality and development plan for all competencies are the biggest impediments to adoption of the standard.
The occurrence of life-threatening, safety-related incidents in the recent past only shines the spotlight on the requirements for functional safety standards and the need for automotive electrical and electronics components, as well as for suppliers to comply with this standard. This means that a well-structured mechanism should be in place. Most importantly, it should begin with implementing a training plan to build the specialist talent or skills required.
Framework in action
System design, functional safety assessment and safety validation are all key elements of ISO 26262 compliance. Most bugs are actually traced to weak requirements. Requirement and test rigor is correlated to risk impact.
ISO 26262 testing corresponds to the Automotive Safety Integrity Level (Asil), the automotive-specific risk-based approach for determining product risk classes. There are three facets that need to be combined to consider the Asil. First, the probability of exposure must be determined. Next, it must be determined if that exposure can be controlled, and what the effect of a failure would be in terms of how severe it is. When exposure, controllability and severity are added, the net combination is the Asil. The higher the integrity level, the more testing must be done.
The Asil is best determined at the beginning of the development process, with a safety assessment to determine the requirements. Automated test tools can support the capture and reporting of structural code coverage for all Asils required by ISO 26262.
Part six of ISO 26262 addresses the recommendations for software testing and verification. Recommended activities include unit level and system level testing such as functional tests (requirement-based tests and partition tests) and structural coverage tests.
During unit testing, developers assess each unit individually for its ability to function properly and meet operational requirements. Once each unit is tested in isolation, the developer moves on to integration testing, where a small collection of units that make up a functional process is tested to make sure the communications between units are working properly.
Building a testing framework to run unit and integration testing automatically supports advanced functionality to ease the testing of code with any complexity. If software is more thoroughly tested, as in this scenario, fewer bugs are likely to be introduced.
Making it work
To improve testing, one must understand what is being tested. Source code coverage analysis is an easy-to-implement way to measure the completeness of existing tests. Measuring code coverage for all types of testing and combining this into a single view provides valuable insight into undertested and overtested areas of the application. By understanding gaps in existing tests, users can improve testing efficiency and, over time, fill these gaps with new tests.
Code coverage analysis might have the highest return on investment of any development process change. Modern tools make it easy to implement code coverage as part of the build process, allowing coverage measurement with no changes to the test process. However, one of the most misunderstood issues with code coverage is its relevance to software quality. One hundred per cent code coverage should not be the goal of software testing; it should be the result of complete testing.
Gaps in code coverage might point to poorly implemented tests, which don’t adequately test the requirements, but the gaps might also be the result of extra code, for example the requirement indicates there are 100 message IDs that should be handled, but the developer added ten new ones and the requirement never got updated.
Just as many groups use coding standards to ensure consistency and portability of source code, testing standards are needed to ensure that good quality tests are developed. All test cases should capture preconditions, test values and expected results – no special knowledge should be required to run a test or to determine if it passed or not.
For code to be testable, it must be easy to understand. In the early days of software development when computer systems had limited CPU clock speed and limited memory, developers were held in high regard if they could make software small and fast. Engineers needed to be clever to fit code onto the device. Programming tricks abounded, as did really hard to understand code.
Very seldom is this the case now; even small embedded systems often have plenty of processing and memory resources, and the emphasis should instead be building code that is easy to understand and maintain. Fortunately, this dramatically improves testability. Every engineer will have their own idea of what simple means, so it is important that development groups implement coding rules and formalise enforcement with automated checks or peer review checklists.
A common testing platform makes it easy to implement correctness and robustness tests. Too often, tests are compartmentalised, with the vast majority being developed and run by the software quality assurance (SQA) team. If developers do not have access to these tests, then it is not surprising that bugs are found by SQA team late in the release cycle – after new features are integrated and these tests get run for the first time. Sharing tests offers a variety of benefits to the whole team:
- • Bugs will be fixed on the day they are introduced, not weeks later;
- • Code changes that break existing tests will not be integrated;
- • Team members will focus on edge cases versus nominal path testing; and
- • SQA team will have more time to write new tests.
Test sharing will provide all stakeholders a sense of ownership for application quality.
Automated and continuous
A robust testing infrastructure requires that tests are small and run fast, execution is automated and dependable, and results are easy to understand. This will enable any team member to run any test, at any time, on any version of the code and to interpret the results easily and, most importantly, it must be easy for developers to debug failing tests.
Automated and continuous testing will only work if a common testing platform is implemented that allows team members to run existing tests and integrate new tests easily. Too often, testing is performed periodically with manual processes required, rather than constantly and incrementally with complete automation. If testing is not continuous, there will be a significant lag between when a bug is introduced and when it is found. The longer this lag time, the more difficult it is to find the cause, and the harder it is to fix.
Conclusion
Most industries move fast, but the pace of innovation in the automotive industry over the past five years or so is nearly unprecedented. However, as many critical functions are now controlled by embedded software, the automotive industry still lacks global quality compliance standards for software quality, as opposed to the rigorous testing performed on the many physical components of automobiles, where testing is done as an ongoing part of the development process, allowing manufacturers to work out potential problems before full production.
The ISO 26262 standard has gained increasing importance, but statistics still show that more than 50 per cent of auto recalls are due to software bugs rather than mechanical issues. With an industry average of eight bugs per thousand lines of code, and the average modern high-end car having ten million lines of code, software quality needs to be a top priority for automotive OEMs. Automated software testing can provide the same type of test stand or simulation environment for software as a vehicle’s physical components, automatically enabling companies to build test frameworks efficiently for all a vehicle’s software application components.
William McCaffrey is co-founder and chief operating officer of Vector Software
