Testing complex software

Traditional V shaped software development model shows the engineering process and corresponding test environment available from Vector Software

Steve Rogerson interviews William McCaffrey, chief operating officer at Vector Software, on the importance of test-driven development

Most companies are already using static analysis tools in software development. Many have evangelised about the importance of static analysis and that message seems to have hit home. But as software becomes more important in embedded devices with the growth of the so-called internet of things, reducing defects in the software needs to move beyond just static analysis.

Some of the figures speak for themselves with about a third of organisations saying they release software with too many defects and even more saying they lack an adequate software quality assurance programme. Yet, more than a quarter of organisations do not conduct any formal quality reviews.

The impact of this on software developers is that many spend up to 80% of their time and costs identifying and correcting defects. And nearly three-quarters of these defects (bugs) are introduced in the specification and design stage of software development.

The traditional V model of software development has been much criticised as outdated but is still widely used, and today still provides a framework for building requirements-based critical software. One company though claims to have a tool set that can bring all stages of this framework together and help detect the bugs and defects earlier in the lifecycle using what is known as test-driven development.

While the tools from Vector Software are designed to work on the right-hand side of the V – the testing phase – its products link with all the major tools on the left-hand development phase to allow what it regards as crucial in modern software development, and that is testing as software is developed, not after it is developed.

"The integration with the cross-compilers on the right that we work with is what makes us stand out from other tools," said William McCaffrey, chief operating officer of Vector Software. "We work in the second half of the V and provide integration links to all the tools on the left.”

And the interest in this has moved well beyond the traditional mission and safety critical applications. The growth of microcontrollers in consumer devices and the cost of failures in the field have led to these areas too taking an interest in test-driven development.

"We have even had interest from companies building white goods," said McCaffrey. "Once people didn't have microprocessors in washing machines and dish washers, but now they do. We have a couple of major appliance manufacturers that are using our tools. We are relevant wherever there are requirements to test. We have now had interest from some robotics companies and finance companies. People we don't normally talk with are now seeking us out."

Part of this is because more companies are beginning to realise the true cost of poor quality software. Whereas this is obvious in areas such avionics, it is becoming more apparent in other areas. Much publicised recalls in automotive are not only expensive but damaging to a car manufacturer's reputation. In the medical device industry, faulty equipment can lead to loss of life, and here McCaffrey is amazed at how little testing is required for the software.

"Medical device manufacturers have no hard requirements to prove that embedded software has been tested," he said. "They may do static analysis, which is fine, but they should make sure the device's requirements are also being dynamically tested. This means actually building tests to exercise the code. Testing and code coverage are recommended but not required. I don't know the reasoning behind not having a requirement to show all test artefacts."

Regulated industries test their software because they have to, and many use static analysis tools because they do find issues with the code. They are simple to use and people benefit from their use. However, they check the code without tying into the design or the requirements of the application. Dynamic testing, on the other hand does check to see the results of running the software are as expected.

"There is obviously a place for static analysis," said McCaffrey, "but you should do both. It is more labour intensive to do dynamic testing and some companies think they can release fully tested applications without it. Everyone wants the ability to have a continuously tested application. To have the ability to run unit, integration and system tests on a regular or daily basis like Google does offers the gold standard in software testing. The companies we are selling to are already doing it but others need to see how this is possible with automated tools."

There are multiple reasons for low-quality software. A common one is that the company's processes are not up to date. Here, management has to decide that things are done correctly and enforce that. Many projects start with existing code, and that can bring its own problems. But bug fixes themselves can cause problems if care is not taken to see what effects the changes might have on different parts of the software. And with distributed teams, care has to be taken to make sure they are all working in unison as different engineers and developers can have their own styles and test strategies.

"This has to come from management," said McCaffrey. "We are working with some big automotive manufacturers and they are mandating that this kind of testing will be done. The goal is continuous test and differential testing so that when one source file changes you can see the other places that have been affected so you only need to test those."

This is key as, for example, some avionics companies have test suites that can take three weeks for a full run through and you do not want to do that every time something changes. Differential testing, where only the parts that have changed or been affected are tested, is thus vital.

"They have to re-test," said McCaffrey, "but having differential testing is a big time saver for them."

One thing that is prevalent in almost all organisations is that more attention is placed on fixing problems than it is on preventing them, leading to bug fixers having a higher status than bug preventers.

"Unfortunately, the problems that get the most visibility are the current bugs and the superstars are the ones that go in and fix them," he said. "But the key is to prevent them before they crop up. Continuous testing and tools such as ours are considered to be bug preventers."

And the situation is going to become worse as the amount of code in even simple devices increases. Larger devices too are becoming more complex and more connected and so the amount of code increases. A good example is to consider the original Apollo guidance computer, which had a mere two thousand lines of code. An average car today has ten million lines of code and that is predicted to grow to a hundred million within ten years.

"With a car, everything is so tied together that a single problem can have multiple effects," said McCaffrey. "The complexity in cars is going to increase greatly over the next few years."

But as this is happening, the number of software engineers coming out of education is reducing.

"It is very difficult to hire software engineers," he said. "As the complexity grows, the resources to deal with that complexity reduce. So we have to develop tools to deal with this."

This he said was leading to companies wanting people to come in and show them how to set up the processes.

"Companies are very open to being educated on how to implement this continuous test environment," he said. "It will be up to us to drive the adoption of the tools. They want us to show them how to integrate our tools into their processes. People want help to set up a continuous test environment."

www.vectorcast.com

Recommend to Colleagues: