Introduction
Software solutions play an increasingly important role in security-critical and security-related systems in general, so software malfunction carries liability and a real threat that can manifest itself in the form of injury, loss of lives, interruption of essential services or damage to the environment. As a result, international standardization bodies such as ISO (International Organization for Standardization) and IEC (International Electrotechnical Commission) have published widely recognized and adopted standards that help software developers certify the security of their software. Some examples are ISO 26262 (Road vehicles – Functional safety) for the automobile, EN 50128 (Communication, signaling and processing systems – Software for railway control and protection systems) for rail transport and IEC 61508 (Functional safety of electrical, electronic and programmable electronic systems) for industrial applications.
The responsibility for demonstrating that the application software and the software methods, processes and toolchains used for its development meet the corresponding functional safety standards clearly rests with the application developer. However, it is a fact that significant parts of the toolchain are outside of the developer's control. This is one of the reasons why compiler validation, a segment in which Solid Sands is already a world leader, has become a critical issue for developers of security-critical systems. In practice, no compiler is error free, so it is extremely important to know where a compiler fails in order to avoid compilation errors.
It is also true that a significant portion of the code that becomes part of the full application will likely be compiled with different use cases, compiler option sets, and build environments than those used by the developer. This is because some of the code that often ends up in an application is made up of precompiled library functions, such as the Standard Library of C (libc), which is often supplied in binary form within a development kit. Software (SDK).
Contrary to what is usually thought, that a library supplied in binary format is insensitive to any specific use case, that is, that the code is invariant, in practice it is not so. Incorporating generic type templates and macros often makes library components sensitive to the use case. Thus, even if the library were pre-validated by the SDK vendor using the same compiler supplied with the SDK, the use case requirements, compiler options, and target hardware environment would almost certainly not have matched, and this makes it difficult to demonstrate compliance with the functional safety standard.
To overcome this limitation, Solid Sands has introduced a new library validation tool called the SuperGuard C Library Safety Qualification Suite. It is a requirements-based test package for the C Standard Library with full traceability from the results of each test to the requirements derived from the ISO C language specification. SuperGuard can be used to validate implementations of the C Standard Library in security-critical applications, both in unmodified and self-developed and maintained third-party library implementations. Its role in the Model V for software development is shown below.
Validation objective
Validation of a software library is critical because the library code is linked to the application and installed on the target device. If a library component is faulty, the functional security of the entire application will be compromised. Each functional safety standard has its own specific goals when it comes to using software libraries, but in general they all share a common goal: verifying that the library implementation conforms to its specification. ISO 26262 offers two ways to validate the library, detailed separately in ISO 26262 Part 8 and ISO 26262 Part 6. SuperGuard C Library Safety Qualification Suite can be used in both cases.
ISO 26262 Part 8, Clause 12: Validation of software components
For classification as COTS (commercial off-the-shelf) products, libraries are covered by Part 8, Clause 12: “Validation of software components”. This clause addresses the need to validate existing software components, such as libraries, "in order to demonstrate that they are suitable for reuse." It specifically mentions “third-party supplied software libraries”, which clearly includes the standard libraries supplied with commercial SDKs. The clause also applies to reused internal software and open source software.
Part 8, Clause 12 states that a prerequisite for validation is a statement of the software component requirements. It also suggests that testing that a software component meets these requirements should "refer primarily to requirements-based tests", which can be accomplished through the "application of a specialized validation test package" that it should “cover both normal operating conditions and the response if a failure occurs” and should not “show errors that could lead to non-compliance with safety requirements”.
Fortunately the library specification for the C and C++ languages exists and is publicly accessible, so there is already a starting point for rigorous requirements-based testing. In fact, the existence and testability of both the language and the library specification are one of the reasons why the C and C++ programming languages are so widespread in the developer community. Now, the language specification is not written as a list of requirements. A key feature of the Solid Sands SuperGuard test suite is that all of its library tests are firmly based on the requirements derived from the ISO C language definition.
To meet the requirement that the test package cover both normal and fault conditions, you should implement each function both within and outside of its boundary conditions, and check the error handling of the function. Among the requirements derived from the ISO C specification to create SuperGuard are verification of the required failure response as defined in the specification.
Reliable proof of the absence of errors that non-compliance with safety requirements may cause depends not only on effective requirements-based testing but also on structural code coverage to ASIL D (Automotive Safety Integrity Level D), the highest level of integrity for automotive applications. To ensure proof of integrity, SuperGuard provides high structural code coverage and high MC/DC (Modified Condition/Decision Coverage).
SuperGuard also incorporates equivalence class and limit value analysis and checking, as well as error estimation depending on the best available knowledge and experience of responding to a library function.
ISO 26262 Part 6: Product development at the software level
ISO 26262 Part 6 takes a parallel approach by treating library code in the same way as other application code executed on the target. Interestingly it doesn't mention libraries as a separate category of code and this might be why the need for library validation is often overlooked.
Validation of the C Standard Library as detailed in ISO 26262 Part 6 is more comprehensive than that detailed in Part 8 because Part 6 addresses all phases of software development.
Part 6, Table 7: “Software Unit Verification Methods” includes requirements-based verification as one of its recommendations for all ASIL levels. As noted above for “software component validation” (ISO 26262 Part 8, Clause 12), SuperGuard assumes all requirements-based testing, although in practice it is recommended that developers use more than one of the methods. indicated in the table to verify an implementation of the C Standard Library.
SuperGuard also employs all the methods listed in Part 6, Table 8: “Methods from Test Cases for Testing Software Units”. Method 1a, “Requirements Analysis” is used to decompose the C Standard Library specification into testable requirements. The description of the C Standard Library is a mixture of (sometimes implicit) definitions and restrictions on the use of functions, as well as function response definitions. In SuperGuard they have also become testable requirements that form the basis of the test package, including the requirements for handling abnormal cases defined in the language standard.
Methods 1b, “Generation and analysis of equivalence classes”, and 1c, “Limit value analysis”, refer to the partitioning of the domains of the input values of the functions and are covered in the SuperGuard test specifications. , which provide the link between requirements and tests.
Method 1d, “Error estimation from knowledge and experience”, is based on Solid Sands' knowledge of difficult implementation areas for the C library, as well as regression tests performed since the initial development of the package. of testing and verification of the Solid Sands SuperTest compiler over 30 years ago.
With this approach, SuperGuard plays a prominent role on the right side of the Model V for software development. In relation to ISO 26262 Part 6, Clause 9: “Software unity verification”, it covers the following:
- Fulfillment of the C standard library implementation according to your requirements
- Verification of the hardware-software interface by testing on the target hardware
- Confidence in the absence of unexpected functionality by checking failure cases and monitoring code coverage
- Verification of the necessary resources through tests on the target hardware
How tests are developed with SuperGuard
When implementing the requirements-based checks recommended in Part 8 and Part 6 of the ISO 26262 functional safety standard, the main problem with the C and C++ Standard Library specifications is that while they provided a detailed description of the response for each function, none of them define a clear set of requirements. Therefore, the necessary requirements for each function must be created from the descriptions of the answers.
SuperGuard C Library Safety Qualification Suite incorporates the proven test suite for the C Standard Library already included in SuperTest, the world's leading Solid Sands compiler testing and verification suite, which has followed the language specifications (ISO) for more than 30 years. However, SuperGuard goes much further than SuperTest in its reporting capabilities, documentation requirements, individual tests, and test results in compliance with functional safety standards such as ISO 26262, EN 50128, and IEC 61508.
The tests in the SuperGuard test package are designed according to the following principles, making them suitable for a wide variety of development environments.
SuperGuard's tests are response-based, that is, they verify that the implementation responds by meeting the library specification. Each test executes the tested construct or function and compares the results of the execution with the expected results (“model”) defined in the library specification. The test itself indicates whether it has been passed to the controller.
These tests are compiled and executed in an execution environment in order to check the response of the implementation, which means that the entire toolchain, including the target processor, is called upon for each test. This makes SuperGuard suitable for library loopback hardware verification.
Tests for the stand-alone part of the library (usually used on bare metal systems) require minimal resources. Most SuperGuard tests can be run on systems with less than 4K of memory, so it is possible to use SuperGuard on very small embedded systems.
To implement requirements-based testing, SuperGuard provides a detailed breakdown of the C Standard Library specifications into testable implementation requirements along with test specifications that describe how each requirement is tested. By linking the results of each test execution to the corresponding test specification, test requirement, and standard library function, SuperGuard provides the full traceability needed for requirements-based testing. To demonstrate that it has been completed, it provides close to 100% structural code coverage for more than 80% of the functions with a high MC/DC (Modified Condition/Decision Coverage). Note that this addresses the library implementation itself, not the underlying OS layer.
An example
Each library test in SuperGuard is developed according to a consistent methodology as illustrated by the function strncpy shown below. This is the specification in Section 7.21.2.4 of the C99 language definition:
7.21.2.4 The function strncpy
Synopsis
1. #includestring.h>
char * strncpy(char * restrict s1, const char * restrict s2, size_t n);
Description
2. The function strncpy copy no more than n characters (characters after a null character are not copied) from the array pointed to by s2 to the matrix pointed to by s1. If the copy takes place between overlapping objects, the response is undefined.
3. If the matrix pointed to by s2 is a string with less than n characters, null characters are added to the copy in the array pointed to by s1 until they have been written n characters.
Result
4. The function strncpy delivers the value of s1.
The most curious aspect about this specification is that Paragraph 2 does not specify a lower bound of on the number of characters it copies. strncpy(). He says like this: “copy no more than n characters”. The specification does not require at any point that any character be copied from s2 a s1. In Paragraph 3 an action is indicated, specifically that s1 is padded with null characters. Taken literally, a correct implementation following this specification would simply consist of writing n null characters in s1.
In effect, the function was not created for this nor is it what it is expected to do. The general interpretation is that the function copies as many characters as possible from s2 a s1 until the chain s2 o n they have been consumed. There is no confusion about it and apparently no one has complained about this phrase since ANSI C89 because the same words are still present in C18. But to define the requirements we have to be a little more precise.
The first step in our test development process is to extract a set of requirements (REQ) from this description taking into account what the feature is actually intended for:
REQ-copystring: Si s2 points to a string with length 'l2' (as defined by strlen()) which is less than n, strncpy() will copy l2 characters, in order, from array s2 until the womb s1.
REQ-copyn: Yes s2 does not point to a string with a length less than n, strncpy() will copy the first n characters, in order, from array s2 until the womb s1.
REQ-shorter: Si s2 points to a string with a length less than n, strncpy() will add null characters ('\0') after the characters copied into the array s1 until they have been written n characters altogether.
REQ-nomore: strncpy() will not write to destination array s1 after the first n characters.
REQ-nochange: strncpy() will not modify the array s2.
REQ-return: strncpy() will deliver the value of s1.
The requirement REQ-nochange follow the statement s2 as a matrix challenge, but the declaration alone does not guarantee that an implementation of strncpy() don't write on s2.
A test specification has been developed for each of these requirements. The test specification defines how a test verifies whether the requirement is true. A single test specification typically leads to a number of use cases covering the input and output domains of the function. Test cases are implemented by the test. The test specification links the requirement to the tests.
For example, the test specifications for the requirements of REQ-copystring y REQ-number are the following:
Test specification for REQ-copystring: apply function strncpy() with different parameter values n (included n==0) equal to and greater than the length of the source string. Check that the source string has been copied to the destination array up to the last null character.
Test specification for REQ-nomore: In all cases, check that the character with index n in destination array s1 has not been modified. If this agrees with the test, check that no characters have been changed after n.
In this case the test specification is implemented REQ-nomore in the same test file as the other test cases to strncpy(). Since the requirement must be unconditionally maintained each time it is applied strncpy(), instead of creating new tests for this test specification is implemented by simply doing an additional check on each test case for the other requirements instead of creating new tests for it.
Handling startup files and function-type macros
The C language complicates the tester's life in several ways. This is why not all functions in the C Standard Library are implemented only in precompiled binary form. Many also rely heavily on the information contained in the boot source files.
These startup files, which define things like types, global variables, and macros, are part of the library just as much as the (precompiled) library functions. Many functions are implemented both as a real function and as a macro, and to increase their speed and efficiency it is common to use their implementation as a macro. Both cases are checked by SuperGuard.
Unlike the corresponding binaries, function-type macros are not precompiled. They are compiled by the SDK compiler along with the application source code. It is therefore important that, along with other content of the startup files, they are checked for the specific use cases of a security-critical application. In the C++ library, the use of macros is taken to an even higher level by using generic type templates that only exist in the header.
When requirements-based testing fits ISO 26262 Part 8 Clause 12
Thanks to these requirements-based test methodologies, SuperGuard ensures the availability of the following fundamental elements included in ISO 26262: Part 8, Clause 12.4.1 to consider a software element validated:
12.4.1 a) the software component specification
The C and C++ Standard Libraries specification is based on publicly available ISO standards, and SuperGuard adds clear functional requirements to the response descriptions that contain these standards.
12.4.1 b) proof that the software component meets its requirements
By converting the functional requirements extracted from the library specification response descriptions into test specifications and test designs, and uniting the resulting tests into a comprehensive set of executable tests, SuperGuard provides the demonstration that an implementation of the library C Standard Library meets the extracted requirements and thus the response description.
12.4.1 c) proof that the software component is fit for its intended use
When SuperGuard is used to test an implementation of the C Standard Library, it generates a detailed report of the PASS/FAIL status of all tests with full traceability back to the functional requirements extracted from the specification response descriptions.
SuperGuard also meets the requirements of ISO 26262 Part 8, Clause 12, paragraphs 12.4.2.2, 12.4.2.3 and 12.4.2.4, which are related to 12.4.1b above. Clause 12.4.2.2 states that verification of a software component can be done using a requirements-based check and a specialized test package, which is what SuperGuard provides. It also states that verification must “cover both normal operating conditions and the response if a fault occursand shouldn't "display errors that could lead to breaches of the security requirements assigned to this software component." SuperGuard checks the functional requirements for all malfunctions defined in the ISO C language standard.
Clause 12.4.2.3 defines an added requirement for the use of software components in ASIL-D applications and states that the structural coverage must be measured in order to assess that the test cases have been completed.
Clause 12.4.2.4 states that the verification process detailed in Clause 12 can only be applied to an unchanged implementation of the Software Component. While some developers replace library functions with their own specialized implementations, these modifications typically only apply to a few functions. Since the C Standard Library is highly modular and virtually all function implementations are independent of the rest of the library, most functions in the library can still be verified as provided in Part 8, Clause 12, thus allowing the application developer to concentrate on validating only those functions that have been modified in accordance with the provisions of Part 6.
Code coverage analysis
Throughout the creation of the SuperGuard test package, special attention was paid to code coverage so that a mature and well-known open source implementation of the C Standard Library would meet the ASIL D requirement in Clause 12.4.2.3. SuperGuard offers 100% coverage for many library functions, plus high MC/DC coverage. The areas where SuperGuard has less coverage are often those related to the implementation-defined response, for which there are no requirements that can be derived from the language specification.
Although each implementation of the library is different, ultimately all of them should handle parsing of similar cases. This means that SuperGuard's high code coverage benefits code coverage for all C Standard Library implementations.
anomalous cases
SuperGuard can handle abnormal cases in two ways. The first is related to the response defined from an anomalous input; For example, when entering a negative number in the function sqrt() the result should be the value NaN (assuming IEC 60559 arithmetic). Although this is an abnormal case, the function's response is fully defined and can be checked like any other.
The second is related to the requirements that can be verified by the compiler. For example, if a function should return a result of type emptiness, a test may attempt to use the supplied value with the expectation that it will generate a compiler error. A test of this kind is called a x-test in SuperGuard and the file name of these tests starts with a x. The
X-Test PASS if the compiler indicates a compile error and FAIL if it does not. X-Tests are never run.
Summary and conclusions
Security-critical applications require software developers to do everything in their power to ensure that their development processes, toolchains, and application code do not pose any risk of injury, loss of life, disruption of essential services. or damage to the environment. When using third-party and/or commercial off-the-shelf (COTS) tools and components, such as compilers and standard libraries, developers should not assume that these tools and components are bug-free or that their prevalidation implies that they are. . Validation is only truly valid if it is performed in the same development environment and under the same use case conditions as in the application.
Using the same test package included in SuperTest, Solid Sands' solution for compiler testing and verification, its SuperGuard C Library Safety Qualification Suite adds the necessary traceability to link test results based on requirements, recommended method for testing functional security requirements such as ISO 26262, against the C Standard Library specification. Full traceability is achieved by decomposing the ISO C Standard Library functional specifications into clearly defined requirements, developing appropriate test specifications to verify such requirements and implementing them in accordance with the recommendations of ISO 26262. It also allows software developers to perform these tests in the same development environment under the same use case conditions and on the same target hardware used in their application, with close to 100% structural code coverage . By generating a comprehensive validation report specially tailored to the needs of ISO 26262-certified organizations, SuperGuard makes it much simpler to demonstrate the integrity of library components used in security-critical applications.
