Our newest product, aiData, became available to selected partners a few months ago. The product includes five tools that can help partners and customers, ranging from data capture to data processing. In this blog series, we will take a closer look at these tools, starting with aiRec, responsible for data recording.

The Data Challenge

Reliable neural networks that form the heart of the automated driving software stack - by sensing the environment around the vehicle - need large, diverse, and contextually rich annotated data sets for training and verification. Collecting and processing enough relevant data efficiently is the biggest challenge automated driving development is facing today. aiMotive has developed tools and is offering them to its customers to overcome this hardship, including solutions for data recording, labeling, synthetic data generation and benchmarking. This first part of a 5-part blogpost series will focus on aiRec, aiMotive’s solution for recording and synchronizing data – and much more - from multi-modal sensor setups.

Data Recording

Recording data ’the right way’ is a critical requirement for all further steps in a data processing  pipeline. What do we mean by ’the right way’? First of all, all the various sensors on the data collection vehicle have to be properly calibrated; their data streams have to be recorded in a synchronized manner (which is no trivial matter when using diverse sensor modalities that have different frame rates or even different ways of scanning the environment); and ideally some initial filtering of the recorded data should take place at this stage.

Calibration

aiRec is a tool for calibrating and validating various sensors and sensor modalities from pinhole and fisheye cameras to radars, Lidars, GPS, and IMU, including patented processes. It also supports the runtime detection of deterioration in the calibration and automatic re-calibration of the sensors. Proper calibration is especially critical in multi-modal sensor setups so that 3D annotations remain consistent across sensors and sensor modalities. Furthermore, to ensure the usability of the recorded data even when the calibration was incorrect during data collection, our post-process tool checks if the uploaded recordings have calibration data that is correct, and regenerates calibration data for recordings with erroneous calibration parameters. 

Synchronization

Once the various sensors on the data collection vehicle are calibrated, we must ensure that they are recorded in a synchronized manner, with respect to a common time base, in order to enable the subsequent annotation, be it manual or automatic. aiRec enables synchronized recording by sensors in a wide range of sensor setups consisting of different sensors, sensor modalities and sensor placements. 

Filtering

aiRec allows for the placement of manual and automatic flags (to mark the interesting parts of the footage) immediately during the recording stage, which makes filtering the raw data possible even before uploading. Manual flags can be placed by the operator of the vehicle using a simple interface, when for example they encounter an interesting scenario on the road, or they experience unexpected behaviour from the automated driving vehicle (when the automated driving software is running at the same time as the data collection is taking place).

The other method for placing flags automatically is to use the perception solution of the automated driving software stack itself (running in shadow mode, or utilizing parts of the full stack) to process the raw footage, and to mark those sections that pose a challenge, such as low confidence detections, erroneous detections, sudden track losses, hard example mining, as well as custom triggers defined by the developers of the perception stack.

Outlook

Collecting the data for training and testing neural-network-based automated driving solutions is already a quite complex task. In our next blogpost, we will introduce aiMotive’s automated annotation solution, and how it helps our customers reduce their costs associated with generating annotated data.