Creating Interchangeable Test Cases Across a Diverse Tool Chain with Multiple Vendor Platforms

Testing is a fundamental exercise performed by engineering teams to validate the functionality, reliability, and operational safety of products. Many industries, including aerospace and automotive manufacturers and suppliers, use a mixture of different vendor-specific tools in the test process. While these tools certainly help developers achieve a high level of productivity, they also have inherent limitations that impact the workload. One of the biggest drawbacks is that it is difficult, and in some cases impossible, to exchange test cases between the tools because they do not speak the same language.

Wouldn’t it be great if test cases could be freely exchanged during different development stages not only between different departments but also between companies or suppliers? The amount of redundant effort – and in particular the code that has to be generated for testing – could certainly be reduced, as well as the overall expense of implementing tests. For larger complex systems that require operational safety, such as aircraft systems, this kind of freedom would go a long way.

One way to overcome this problem is through standardization. By standardizing interfaces between test benches and automation tools and standardizing exchange formats for test cases, a multi-vendor tool chain could not only facilitate the reuse of test cases, but the exchange of test cases between different departments and even different companies or suppliers. Additionally, engineers would also be able to use their local tool chain to easily debug and inspect failed test variants in their own familiar environment.

I recently co-wrote a paper, “Interfacing and Interchanging – Reusing Real-Time Tests for Safety-Critical Systems”, in cooperation with my dSPACE colleague Rainer Rasche and business partners Volker Meyer from Airbus Operations GmbH and Marco Franke and Klaus-Dieter Thoben from the Bremen Institute of Industrial Technology and Applied Work Science (BIBA). We looked into these challenges, and I am happy to share a summary of our findings. 

We looked at two different approaches that can potentially solve the problem. The first focuses on enabling interfacing between test automation tools and test benches. The second facilitates interchanging, which makes it possible for users to exchange test cases and definitions between different test automation tools. 

Standardized Interfacing

By using standardized interfaces between automation test tools and test benches, users can combine their software and simulation hardware freely. It also gives them the flexibility to switch tools and test benches, if required. There are two common approaches: Standardized interfacing at the protocol level and at the API level. For simulation purposes, standardization at the API level is more frequently used.

The Association for Standardisation of Automation and Measuring Systems (ASAM) has developed XIL API as a generic simulator interface for the communication between test automation tools and test benches. It enables users to choose products freely according to their requirements, as well as independently of the vendor or the development stage.

XIL API 2.0 is the latest version of the standard. Cross-tests between different vendors and their products demonstrate a good interoperability of test benches and test automation tools that use XIL API. This allows end users to combine the most suitable test software with the most suitable test hardware. The standard is technology-independent and can be easily extended to other languages, such as Java or C++, if desired. 

Standardized Interchanging

Companies that use diverse, multivendor test systems often end up with a mix of different test procedure languages with their own syntax and semantics. By having standardized exchange formats, users would be able to interchange test cases and definitions between different automation tools.

Some common test procedure languages are: Testing and test control notation (TTCN-3), Automatic Test Mark-up Language (ATML), and the signal description definitions within XIL. Typical test procedure language addresses the set-up environment, describes how a fault is injected into the system under test, and defines potential failure states, as well as expected outcomes (relevant to specific parameter value assignments), but there is a disconnect in semantics.

The semantics of specific test bench functions, such as logging, are defined outside of the test procedure language. This complicates the interchange of test cases. To enable the interchange of test bench functions, either the semantics must be extracted from the test-bench-specific implementation and represented in the state chart or similar test-bench-specific functions have to be mapped between the various test benches. In either case, there is no guarantee that the functions of one test bench will be made available on another.

To create a test procedure that is interchangeable, a two-step approach is required:

1. The general transformation of states and transitions (i.e., field data integration, cross compiling)
2. A mapping mechanism to map test system functions between each other

The goal of both methods is to extract the syntax and semantics from one format and transform it to the other syntax covered the required semantics.

To learn more about standardized interfacing and interchanging approaches, as well as methods for implementing these solutions, download the technical paper: Interfacing and Interchanging – Reusing Real-Time Tests for Safety-Critical Systems. This paper offers a proof of concept.