My 8 Tips for Efficient Data Logging

Lists can make our lives much easier. If you are getting started in data logging for the development of highly automated and autonomous vehicles, I can give you a list of eight essential tips for efficient data logging and for selecting the corresponding tools. These tips are based on the extensive experience of our dSPACE competence teams and can save you time and effort that would otherwise be wasted on incorrect assumptions. 

Data Logging: A Starting Point in Data-Driven Development for ADAS/AD

Before taking a closer look at data logging, let’s see how it ties into the overall development process for ADAS/AD: If we think of data-driven development as a data pipeline, then data logging can be considered one elementary part. It is the starting point of the pipeline and it is followed by other development steps, such as data ingestion, data analysis, data selection, data annotation, and, of course, software development accompanied by comprehensive testing and validation, including data replay tests. Data can also be logged during on-road tests. For us, this means two things: First, data logging is not an isolated process. Second, since data logging is the starting point of development, our actions in data logging will have a strong influence on all subsequent steps.   

The List: 8 Tips

Now to what you’re here for: my list of tips for efficient data logging. These 8 tips are certainly not the be-all and end-all, but they do address some main points.

# 1: Focus on Sensors

When you embark on a new data logging project, first check the properties of the sensors in detail. There is a wide variety of sensors, and unfortunately there is no standard on how to read information from them. This is especially true for camera sensors, so to be able to log data, first check whether the data logger you want to use supports the applicable interface and data formats. If it does not, select a different logger. For this, you will need detailed information from the sensor supplier, so clarify that you have access to this information.

# 2: Evaluate Tools Yourself, Do Not Buy into Marketing

You know best that time is of the essence in ADAS/AD development projects, and the market is very competitive. This means that you do not have a lot of time to try out tools – nor should you experiment with your budget. You will need a solution that works reliably right away and matches the intended use. Component failures should not occur as they would lead to delays. Synchronization has to work well. A comprehensive view is needed instead of data silos. So before selecting a tool, ask your supplier for references. Let the supplier prove that the solution works and can be integrated into your environment. In this context, it is best to start a proof of concept tailored to your requirements, which will make sure the solution fits.

# 3: Look for Integrated Solutions and Openness

Integrating tools that are not “made for each other” can be a challenge – and time-consuming. Integration of the data logging hardware and the corresponding software tools is particularly important. Look for solutions that offer a high level of integration out of the box, with open interfaces that make them easy to connect to your tools and processes. This way, you can accelerate your time to market and will not waste time.

# 4: Think Ahead (Scalability and Flexibility)

I know you have to solve the most pressing project issues first, so you will be looking for a data logging solution that works today. But you should always keep an eye on the future, with potentially stricter or different requirements. Whatever solution you choose, it has to be able to meet these requirements. Scalability is very important in this context, which means you can start with a small or medium-sized solution and then extend it to cover future demand. Relevant factors for scalability are interfaces and processing power (required to handle the data from the interfaces).

# 5: More Is More: Processing Power

Plan with a lot of processing power. Not only to achieve scalability, but also to cover future software requirements. The lion’s share of optimization potential in data logging lies in software. The more data you can analyze, the better your insight into the data, the more efficient your data cycle management. With sufficient processing power, you can perform intelligent data filtering in the vehicle to reduce the volume of data. Your overall goal should be to pick and store only the data relevant for your work further down the line.

# 6: Scale Up Power, Scale Down Complexity

Connecting a number of devices can increase the potential for errors. Look for a centralized data logging solution that consists of only a few components. Centralization will keep your data together. You will also need less space, which is especially relevant for the vehicle, where space for hardware equipment is limited.

# 7: Find Good Fleet and Data Management Software

The huge amount of data collected in data logging campaigns underlines the need for good data management software. No matter if you have 5 or 100 cars in the field recording traffic situations, you will want to know where the cars are and if everything is working. You will want to know if your campaign is progressing as expected. You have your plans, goals, and metrics, and you need to be able to track their fulfillment.

# 8: Think of the Complete Data Pipeline

As I mentioned before, data logging is always part of a larger context. The recorded data will serve a specific purpose further down the data pipeline, so think beyond the hardware and software, beyond the vehicle. Focus on the actual purpose of the collected data in the development process, in AI training, and in testing (e.g., in data replay tests). As for process efficiency: Always try to shorten the data cycle and ensure a seamless flow of data from the vehicle to the data scientists. Keep an eye out for consistent file formats and compatible interfaces.

Want more tips?

If you are looking for more tips, we at dSPACE will gladly help you master your data logging and data pipeline challenges – with technology, methods, and years of expertise.

Do you have any questions? Our data logging experts are always happy to help. Simply contact us at info@dspace.de.