Efficient Continuous Testing with dSPACE

Agile software development processes are gradually conquering the automotive industry. One of the major benefits are the continuous development and testing processes to improve development efficiency and software quality. Establishing these processes typically works quite well within the development departments. However, including additional integration levels and test stages from SIL into HIL in agile processes remains as major challenge for the automotive industry. In the following blog entry, I want to introduce you to a typical continuous integration process and demonstrate how the dSPACE tool chain enables agile continuous processes by bringing the automation to the required level.

First, I would like to describe the typical steps of a continuous integration process. A developer working on a development branch of a certain software unit or module wants to commit changes to the master branch of the ECU software. In a continuous integration process, the publication of changes automatically triggers associated tests to the changed software unit. First, the publication triggers a set of unit tests on the changed code or changed model in model-based development. If the tests are successfully completed, the continuous integration process then automatically triggers the integration of the changed software unit as well as the other units into the ECU software. This can be a virtual ECU (V-ECU), for instance, to perform basic functional tests in a SIL environment (such as dSPACE VEOS).

Immediate Feedback on Code Quality

These basic functional tests typically consist of automated tests associated with a set of functional requirements for the software module that is tested. The SIL environment also gives the developer the option to perform manual tests in the fully operational software that take into consideration the changes in an appropriate simulation environment. If the unit tests, the basic functional tests, and any additionally performed manual tests or automated code reviews are successful, the changes are finally merged to the master branch of the ECU software development. This means the modified and tested software unit can be incorporated into the next ECU software version. The main benefit is that the developer gets immediate feedback on the quality of the changes and faulty code is identified early and thus not merged to the master branch.

Benefit from a Pure Software Testing Environment

In contrast to standard software development, in embedded software development it is useful if the integration pipeline can generate two software artefacts of the next ECU software version in parallel, a virtual ECU (V-ECU) and a real ECU (ECU) software image. The V-ECU image is ideally suited for a full set of functional tests running over night in SIL. This way, you can benefit from the scaling options provided by a pure software testing environment. The ECU image is used on highly automated HIL simulators to perform a set of basic tests limited by the amount of available hardware simulators and ECU hardware prototypes. These basic tests can be a cross-section of different software-hardware integration tests, e.g., smoke tests, functional tests, and performance tests.

As the generation of V-ECUs and ECUs are done on the same code basis, the functional behavior is the same. Due to the consistency in the dSPACE tool chain, you can use tests on both the SIL stage and the HIL stage. Back-to-back tests are one method to validate that these SIL tests do not have to be repeated in a HIL environment for final validation. As a result, you can save HIL simulation time for performance tests or tests with electrical error stimulation. This means, you use the full power of a hardware-in-the-loop simulator. During the subsequent test stages, systems of various ECUs are tested, from domain ECU integrations for body or powertrain, for example, to full vehicle ECU integrations.

Decreased Feedback Time leads to Higher Software Quality

The dSPACE SIL platform VEOS and the SCALEXIO HIL offer ideal platforms to support all levels of integration from a functional test of a single software module to release testing of a virtual vehicle that includes the full ECU network found in a car. In the spirit of continuous integration and testing all the different test stages will be continuously and automatically executed. Naturally, the sheer number of tests it will make it impossible to run all required tests overnight. Nevertheless, with the scalability of SIL at least the functional testing is achievable overnight. In addition, a high automation of HIL and an intelligent selection of test sets not only increases the amount of testing, the feedback time to the developers also drastically decreases which results in higher software quality.

So, how can we achieve all this? Powerful test platforms such as VEOS for SIL and SCALEXIO for HIL are a good basis for running the (V-)ECU software in an appropriate simulation environment. In addition, an efficient test automation tool can help to quickly develop a large set of automated test cases, which is required in such a continuous integration pipeline. Basically, every test run can be seen as a regression test and the goal must be to increase the test catalog as well as the software quality from day to day. For this, we offer various test implementation methods, e.g., step-based or key-word-driven testing, signal-based testing or completely free-programmable tests. For testing of ADAS/AD driving functions or perception algorithms, dSPACE also offers scenario-based testing where automated tests are modeled by means of realistic traffic scenarios as described, e.g., in the Pegasus project. The dSPACE tools ModelDesk and AutomationDesk will help you to implement your tests efficiently.

Test Orchestration becomes Crucial

As previously mentioned, in continuous integration processes the feedback time to the developer is essential to gaining the benefit of increased software quality. In such a complex test process with multiple test stages, the orchestration of the tests becomes a crucial part. The orchestration consists of three parts, planning the test, executing the test, and collecting the results as well making them available to the developers. In more detail, this means the selecting and collecting test cases for execution plans (e.g., by means of queries) from a central database to configure which tests will be executed for every build, or every night for the nightly build, or for the build over weekends.

This requires remote control of the test platforms and test automation tools to automatically distribute the tests to the available SIL and HIL platforms, execute the tests, and retrieve the results to the central database. From the database, the testers can triage the results and publish them to make them accessible to developers. To cut out manual human interaction, the test platforms also require an automated preparation, i.e., automatically flashing and potentially calibrating the ECU software, downloading and parameterizing the simulation model, and checking out required test data, e.g., test implementations or stimulus data for open-loop tests. And after the test execution, performing a test system clean-up to be ready for the next test run.

SYNECT for Highly Automated Test Pipelines

For this set of tasks around a highly automated testing pipeline, dSPACE offers SYNECT, its central test and data management platform. SYNECT not only provides a central database for test cases and test results, it also enables managing simulation models and their parameterizations. Furthermore, the tool offers several off-the-shelf integrations for testing automation tools, such as dSPACE AutomationDesk, Vector CANoe/vTESTstudio, NI TestSTand, and recently also Tracetronic ECU-TEST. In addition, it offers seamless integrations to the ALM environments, such as IBM ELM, Siemens Polarion ALM, Attlassian JIRA or Microsoft TFS. Therefore, the highly automated test pipeline with SYNECT works hand in hand with your development pipeline.

Bring Your Testing to the Next Level

In summary, dSPACE provides you with the full tool chain to bring your testing to the next level, from test platforms to test automation and test orchestration. All this is available not as a closed environment but as an open integrated environment that can be extended and complemented by the third-party tools you require to work efficiently.