Software solutions play an ever-increasing role in safety-critical and safety-related systems, with the result that software malfunctions now represent liabilities and a real threat in terms of injury, loss of life, the interruption of essential services, or damage to the environment. As a result, international standards organisations such as ISO and the International Electrotechnical Commission (IEC) have published widely-recognised and widely-adopted standards against which software developers can certify the safety of their software. Examples include ISO 26262 (Road vehicles – Functional safety) for automotive, EN 50128 (Communication, signaling and processing systems – Software for railway control and protection systems) for rail transport, and IEC 61508 (Functional Safety of Electrical / Electronic / Programmable Electronic Safety-related Systems) for industrial applications.
The responsibility for demonstrating that application software – and the software methods, processes, and toolchains used to develop it – comply with the relevant functional safety standards lies firmly with the application developer. However, it remains a fact that significant parts of the toolchain lie outside the developer’s control. This is one of the reasons why compiler validation has become a key issue for developers of safety-critical systems. Virtually no compiler is bug-free, so it is extremely important to know where a compiler malfunctions so that compiler errors can be avoided.
It is also true that a significant part of the code that becomes part of the complete application is likely to be compiled with a different applications, set of compiler options and compilation environment from those being used by the developer. This is because part of the code that typically ends up in an application comprises pre-compiled library functions, such as those in the C Standard Library (libc) that is often supplied in binary format as part of a software development kit (SDK).
Contrary to the commonly-held belief that because a library is supplied in binary format it is insensitive to any particular applications, i.e. the code is invariant, in practice this is not the case. The inclusion of macros and type-generic templates frequently makes library components use-case sensitive. So even if the library was pre-qualified by the SDK supplier using the same compiler delivered with the SDK, the matching use-case, compiler options and target hardware environment requirements are almost certain not to have been met, making it difficult to demonstrate functional safety standard compliance.
To overcome this limitation, Solid Sands has introduced a new library qualification tool called the SuperGuard C Library Safety Qualification Suite – a requirements-based test suite for the C Standard Library with full traceability from individual test results back to requirements derived from the ISO C language specification. SuperGuard can be used to support qualification of C Standard Library implementations for safety-critical applications both for unmodified third-party library implementations and self-developed or self-maintained implementations. Its role in the V-Model for software development is shown in Figure 1.
The aim of qualification
Qualification of a software library is critical because code from the library is linked into the application and installed onto the target device. If a library component is defective, the functional safety of the entire application can be jeopardised. Every functional safety standard has its own specific objectives when it comes to the use of software libraries, but, in general, they all share a common goal: to verify that the library implementation complies to its specification. ISO 26262 provides two routes to library qualification, detailed separately in ISO 26262 Part 8 and ISO 26262 Part 6. The SuperGuard C Library Safety Qualification Suite can be used in both cases.
How SuperGuard tests are developed
When implementing the requirements-based testing recommended in Part 8 and Part 6 of the ISO 26262 functional safety standard, the main issue with the C and C++ standard library specifications is that although they provide a detailed behavioral description for each function, neither of them defines a clear set of requirements. The necessary requirements for each function must therefore be created from the behavioral descriptions.
The SuperGuard C Library Safety Qualification Suite incorporates the proven test suite for the C Standard Library already included in Solid Sands’ world-leading SuperTest compiler test and verification suite, which tracks the (ISO) language specifications. However, SuperGuard goes much further than SuperTest in terms of its reporting capabilities, documenting requirements, individual tests and test results in accordance with functional safety standards such as ISO 26262, EN 50128 and IEC 61508.
The tests in SuperGuard’s test suite are designed according to the following principles, making them suitable for a wide range of development environments:
SuperGuard tests are behavioural tests — i.e. they verify that the behaviour of the implementation complies with the library specification. Each test executes the construct or function under test and compares the results of execution with the expected (‘model’) results defined in the library specification. The test itself reports success or failure to the test driver.
In order to check the behaviour of the implementation, these tests are compiled and executed in an execution environment, which means that the entire toolchain, including the target processor, is involved in each test. This makes SuperGuard suitable for hardware-in-the-loop verification of the library.
The tests for the freestanding part of the library (typically used in bare-metal systems) require minimal resources. Most SuperGuard tests can run on systems with less than 4K memory, making it possible to use SuperGuard on very small embedded systems.
To implement requirements-based testing, SuperGuard provides a detailed breakdown of the C Standard Library specifications into testable implementation requirements together with test specifications describing how each requirement is tested. By linking individual test execution results back to the corresponding test specification, test requirement, and standard library function, SuperGuard provides the full traceability needed for requirements-based testing. To provide evidence of completeness, it provides close to 100% structural code coverage for more than 80% of the functions, with high Modified Condition/Decision Coverage (MC/DC). Note that this addresses the library implementation itself, not the underlying OS layer.
Header files and function-like macros
The C language throws in one more spanner to complicate the tester’s life. Which is that not all functions in the C Standard Library are only implemented as pre-compiled binaries. Many are also heavily dependent on information contained in source header files.
These header files, which define things such as types, global variables and macros, are as much a part of the library as the (pre-compiled) library functions. Many functions are implemented both as a real function and as a macro, and for speed and efficiency it is common practice to use the macro implementation. Both are tested by SuperGuard.
Unlike the corresponding binaries, function-like macros are not pre-compiled. They are compiled by the SDK’s compiler together with the application source code. It is therefore important that, together with other content in the header files, they are verified for the specific use-case of a given safety-critical application. In the C++ library, the use of macros is elevated to an even higher level through the use of type-generic templates that only exist in the headers.
Safety-critical applications require software developers to do everything in their power to ensure that their development processes, toolchains, and application code pose no risk of injury, loss of life, the interruption of essential services, or damage to the environment. When using third-party and/or commercial off-the-shelf (COTS) tools and components, such as compilers and standard libraries, developers should not assume that these tools and components are error-free, or that pre-qualification implies that to be the case. Qualification is only truly valid if it is carried out in precisely the same development environment and under exactly the same use-case scenario as used in the application.
Employing the same library test suite included in the Solid Sands SuperTest compiler test and verification solution, its SuperGuard C Library Safety Qualification Suite adds the traceability needed to relate the results of requirements-based tests – the recommended method of testing in functional safety standards such as ISO 26262 – back to the C Standard Library specification. Full traceability is provided by breaking down the ISO C Standard Library functional specifications into clearly defined requirements, developing suitable test specifications to check those requirements and implementing them in accordance with ISO 26262 recommendations. In addition, it allows software developers to perform these tests in the same development environment, under the same use-case conditions, and on the same target hardware used in their application, with close to 100% structural code coverage. By generating a comprehensive qualification report tailored to the needs of ISO 26262 certification organizations, SuperGuard alleviates much of the burden of demonstrating the integrity of library components used in safety-critical applications.