In their efforts to create software-defined vehicles and automate more and more functions, car manufacturers are responding to the high complexity of an open world with increasingly sophisticated hardware and software solutions. However, this means that requirement- and specification-driven approaches are reaching their limits and the current set of methods for requirement-based engineering cannot cover these demands. With the term "Safety of the Intended Functionality" (SOTIF), an approach was created to meet these new challenges. The aim is to bring both sides of the V-model closer together. To achieve this, the activities in the verification and validation branches must be related to each other to create a solid basis for the safety argumentation.
The future of homologation: Where is it heading?
In February 2021, the UNECE (United Nations Economic Commission for Europe) presented the New Assessment/Test Method for Automated Driving (NATM) – a framework and potential game changer. It includes a scenario catalog and five validation methods. This emphasizes the fact that the homologation process will evolve over time. Even if future strategies and type approval for new vehicles and ADAS functions vary, one thing is certain: currently and in the future, it is necessary to combine the assessment of an Automated Driving System (ADS) and its ability to demonstrate safe behavior when operating in the real world by extensive use of simulative approaches meaningfully combined with real-world driving and physical certification.
The complex driving tasks and the search for the best approach require technical testing, which needs more than a static set of test cases. The complex automation systems also require more than a static series of checklist items in terms of architecture and components. In both cases, there is a need to test procedures and target fulfilment (e.g., through audits).
Current and future regulations in testing and type approval
The requirement to relate individual test results to the rest of the tests and results is expressed in the "multi-pillar" approach of the GRVA (UNECE Working Party on Automated/Autonomous and Connected Vehicles) and the efforts of the New Assessment/Test Method for Automated Driving (NATM) of the UNECE (United Nations Economic Commission for Europe).
This suggests that there will be a more extensive catalog of requirements in the current and upcoming regulations than in the past.
The following points are currently being discussed:
- Scenario catalog: A variety of tests including hazardous and regulation-defined scenarios to evaluate automotive control systems
- Simulation/virtual testing: Simulation at scale to ensure complete test coverage
- Track testing: Testing the reliability under real-world conditions and proofing the system's real-world performance
- Real-world testing: Evaluation of the system's performance on public roads and analysis of the system's capability to handle real-world traffic
- Audit/assessment: Evaluation of safety concepts, tools, and implemented precautions
- In-service monitoring and reporting: Fostering a safety culture that encourages continuous improvement while avoiding blame
There are issues here that can only be addressed by a combination of these points.
Some examples are:
- Trustworthiness (simulation/virtual testing + track testing + real-world testing)
- Qualification of tools (audit/assessment + testing + in-service monitoring)
- Coverage of hazard scenarios (audit/assessment + scenario catalog)
- Sufficient exploration of unknown hazardous scenarios (real-world testing + in-service monitoring + scenario catalog)
These objectives are the focus of inspections by technical service providers.
Possible implications of the study and the draft test strategy
1. The draft strategy can be used to argue the estimates of the residual risk, the scenarios carried out, the vulnerabilities, and the remedial actions considered. The above-mentioned goal of covering all hazard scenarios is, of course, linked to the need to evaluate the residual risk and assess whether the values achieved are acceptable. A key element in achieving the goal of assessing residual risk is an audit strategy approach that combines hazard analysis and scenario investigation with a review of analysis results and the effectiveness of implemented risk mitigation measures.
2. The draft test strategy defined by the ASAM Test Specification Study Group can serve as a basis for an evaluation of OEM/Tier 1 specific test strategies.
The challenge of verification and validation of automated driving is also due to the interdependence of different tests using different methods and test environments: The full strength of the verification and validation evidence/arguments only comes into play if the different results are related to each other. A draft test strategy could show the most important interrelationships in this respect. It could thus serve as a basis for the creation of a verification and validation strategy at the OEM or Tier-1 and as a basis for assessments and approvals.
3. The different test strategy methods can be verified in different ways, for example, using document review, witness testing, or re-testing at the technical service.
Assessing the test activities and the test results obtained for compliance with the verification and validation strategy and with the requirements of standards and regulations is likely to require several approaches. These approaches, which are very similar to the test methods themselves, need to balance effort and depth of investigation. Documentary checks can be used with little effort. These can be used, for example, to clarify that passed test results actually confirm safety and that failed tests have been properly fed back into the development cycle. To gain a deeper insight into the tests performed, witnessed tests are particularly valuable, especially when very specific and specialized test equipment is required. To achieve fully independent confirmation of test results, a fully independent review by a technical service can complete the range of testing activities.
Let’s talk about solutions:
“Automation has long since found its way into today's automobiles – and will do so to an even greater extent in the future. To cope with the associated complexity, the industry is developing ever more sophisticated hardware and software solutions. Existing requirement- and specification-driven approaches are reaching their limits and need to be rethought. This is where a testing strategy with a holistic approach comes into play. For this, dSPACE has both relevant process experience gained from industrial development projects and an integrated end-to-end solution portfolio for these demanding tasks. We are your partner along the entire value chain, helping you to introduce new approaches and processes efficiently and implement them reliably.
If you need a sparring partner for your challenges, you've come to the right place.”
Courtesy of ASAM e. V. Source: https://report.asam.net/ – Published August 2023