Quality crisis

John Paliotta shows how software testing tools can be used to drive engineering excellence
We live in a software-powered world, and it is falling apart around us. This is not a rant; just take a look at the headlines, and you’ll see a steady stream of news about software bugs. In particular, several automotive companies have been affected by very public bugs.
Toyota may have gotten the most publicity during the past three years, with two high-profile recalls: 1.9 million Prius vehicles were recalled recently because of “a programming error which could cause the gas-electric hybrid systems to shut down”. Before that, close to nine million Toyota vehicles were recalled for the well-publicized sudden acceleration problem. This is a problem that recent analysis suggests was caused by stack-overflow as a result of deep recursion.
The point is that automotive recalls because of malfunctioning electronic systems are increasing rapidly. This increase in recalls signals a crisis in quality that is affecting the entire software industry, from the healthcare.gov web site to applets on your mobile phone.
Quality standards
To help improve the quality of electronic systems in vehicles, the automotive industry has supported the creation of the ISO 26262 functional safety standard. This standard, which defines a set of recommended activities related to the development of electronic systems for automobiles, is being rapidly implemented by OEMs and tier-one suppliers in an effort to improve quality.
It is important to realise that although ISO 26262 is relatively new, it is actually an adaptation of the IEC 61508 standard that has been in use since 1999 in industries such as industrial and process control, railway, and atomic energy. As such, the guidance is based on a solid foundation of best practices that have been successfully applied within these industries for many years. Specifically for software-based systems, ISO 26262 recommends testing of the software components prior to integration to prove correctness, and the analysis of the code coverage achieved during testing to prove testing completeness.
This is not the first initiative for improving software quality within the automotive industry. Since the early 1990s, most automotive software has been developed in conformance with the Misra standard. Misra is a set of “rules” that prohibit the use of problematic coding patterns that can lead to software bugs. For example, although the C programming language allows you to write: “if (a=4)” most programmers really intend “if (a==4)”, so Misra prohibits the former. The Misra rules are easily implemented in static analysis tools, which parse source code looking for violations of the rules, and the standard is now widely accepted as a model for good programming practice. “Misra C++: 2008 Guidelines for the use of the C++ Language in Critical Systems” defines similar rules for the C++ language.
Misra is not enough
Obviously, conforming to the Misra coding rules can help quality, but code that is Misra complaint is not necessarily correct or tested thoroughly; these two objectives are addressed by ISO 26262. Dynamic testing activities recommended by ISO 26262 include unit level, system level and functional tests, where test cases are based on the software requirements, and code coverage is measured during testing.
So the questions facing developers of automotive software systems are: how best to implement an ISO 26262 conformant development process and what will be the cost? Fortunately, the path to improved quality is well understood, and the costs are less than might be expected. Think of where the time goes during software development. There are three basic activities: design, coding and testing. Normally, they each take about the same amount time in the project schedule, but what does “test” mean, and when are we done testing. ISO 26262 answers this question exactly, as it prescribes a risk-based process for determining the criticality of a system and based on that criticality recommends particular code coverage metrics that should be recorded during testing. The criticality levels range from Asil D for the most critical systems (air-bags, ABS), to Asil A for the least critical systems (entertainment systems).
Competing goals
The goals of improving quality while reducing cost and time-to-market may seem to be in conflict, but really improving software quality can also help you reduce cost. The cost of producing and selling a product is not just the R&D, manufacturing and selling, it is also the life-cycle support and maintenance cost of the deployed product. The key to reducing post-sales support and warrantee costs is to improve quality, and the easiest way to improve quality is to employ a holistic approach to testing. Testing cannot be the responsibility of a single group of engineers that are involved for one third of the development process; it must be the responsibility of everyone during the full life cycle.
The key to improved quality is a repeatable and automated testing process that enables each member of the team to access every test that the organisation owns. This allows every engineer to test a software change thoroughly at any point in the development cycle, without specialised testing knowledge. Quality becomes everyone’s job
Poor quality
No organisation sets out to release poor quality software, certainly automotive OEMs do not, but system complexities and time to market pressures result in a lot of sub-optimal software being released. So what are some of the direct and indirect costs associated with poor quality for OEMs?
• Repair costs: Manufacturers budget hundreds of millions of euros per year for warrantee repair and recall costs.
• Brand damage and lost sales: Consumers have unprecedented access to quality and dependability rankings for every car brand, and they act on this information.
• Enterprise value: Investors have been quick to punish companies when they suffer reduced earnings from recall campaigns, or reduced market share from brand damage.
Best practices
So how do you start on the path of creating a quality culture? What are some best practices?
- Build high quality requirements: During the design phase it is critical to build high quality and consistent requirements that do not just address the nominal features, but also edge cases, functional data ranges and error handling.
- Prototype while building requirements: Ensure that the design is valid by creating prototypes of critical functionality. Will your algorithm scale from ten data items to ten million?
- Build testable software: No one would design a widget that could not be manufactured and tested efficiently, but engineers consistently build software that is hard to test. Small modules with well-defined pre and post conditions make testing easier.
- Coding style: Adhere to organisation standards for consistency and industry standards for dependability.
- Build complete test cases: Correctness tests that map to requirements, robustness tests that stress functional ranges, completeness tests that stress combinations of values.
- Meaningful peer review: Focussed on the architecture, edge cases and test cases.
- Automated testing infrastructure: Any test, run any time, by any user.
- Culture of quality: Quality is everyone’s responsibility, not just the QA department.
Software tools
The good news is that with an investment in a handful of software development tools, you can greatly reduce the effort of implementing the best practices described above. Regardless of the tools selected, the workflow implemented must be practical and repeatable.
Conclusion
More than ever before, software controls the users’ satisfaction with your products. It is truly the face of a product. Few people care about the CPU or integrated circuits that are used in their car’s navigation system. What they do care about is dependable operation, ease of use and flexibility.
Building quality into your software applications is a process. There are no silver bullets to improving quality, no push-button solution, and no shortcut your organisation can take to achieve this goal. As you work to improve quality, pay attention to development tools and processes.
John Paliotta is co-founder and chief technology officer of Vector Software
