This project aims to develop solutions for vehicular perception through inertial sensor signals and Artificial Intelligence models. Vehicular perception comprises exteroception and proprioception. Exteroception aims to understand the environment outside the vehicle, recognizing the road features on which it travels. These features include transient events in the form of anomalies and obstacles, such as potholes, cracks, speed bumps, etc.; and persistent events, such as surface type, conservation condition, and the road surface quality. On the other hand, proprioception aims to understand vehicular movements to identify their own behavior. These identifications can also be transient in the form of driving events, such as lane change, braking, skidding, aquaplaning, turning right or left; and persistent, as a safe or dangerous driving behavior profile. This situational information (perceptions) has wide applicability in Intelligent Transport Systems (ITS) such as Advanced Driver Assistance Systems (ADAS) and autonomous vehicles.
For the development of this project, we collect nine datasets using GPS, camera, inertial sensors (accelerometers and gyroscopes), magnetometer, and temperature sensor. These data were produced with contextual variations including three different vehicles, driven by three different drivers, traveling through three different environments. To recognize and classify the vehicular perception patterns, we have developed several models based on Artificial Intelligence, among Classical Machine Learning and Deep Learning approaches. Below we describe the datasets produced, models developed and the results obtained, together with published scientific papers and source-codes.
- Vehicular Perception Research
- Passive Vehicular Sensors Dataset
- Data Classes
- Best Models
- Aplications
- How To Cite
- License
The project is active and we are currently developing new models for new perception pattern recognition. Below are described the research progress, in chronological order of research development. At the Research Gate you can also find the published scientific papers and request a full text for free.
In this paper, we describe the state-of-the-art vehicle perception produced through inertial sensors and Artificial Intelligence techniques. Through a literature review, we compiled the data extracted from the selected studies and described each paper in detail and chronological order of publication.
Access here
In this paper, we present a structured literature mapping of the state-of-the-art vehicle perception produced through inertial sensors and Artificial Intelligence techniques. We describe a structured, approach, and technologies-oriented panorama of this field.
Access here
Road Surface Type Classification Based on Inertial Sensors and Machine Learning: A Comparison Between Classical and Deep Machine Learning Approaches For Multi-Contextual Real-world Scenarios
In this research, we developed models for the road surface type classification, classifying between segments of dirt, cobblestone, and asphalt roads. We applied classical techniques of K-Means Clustering (KMC), Support Vector Machine (SVM), and K-Nearest Neighbors (KNN); and Deep Learning techniques based on Convolutional Neural Network (CNN), Long Short-Term Memory Network (LSTM), and hybrid LSTM-CNN. We analyzed in the time domain the data collected near and below the suspension, in addition to the impact of the data window size. We also evaluate the generability of model learning for unknown contexts, such as an unknown car, driver, or environment.
Access here
Multi-Contextual and Multi-Aspect Analysis for Road Surface Type Classification Through Inertial Sensors and Deep Learning
In this research, we developed models for the road surface type classification, classifying between segments of dirt, cobblestone, and asphalt roads. We applied Deep Learning techniques based on Convolutional Neural Network (CNN), Gated Recurrent Unit (GRU), and Long Short-Term Memory Network (LSTM). We analyzed various aspects, such as the influence of the vehicle data collection placement, the analysis domain, the model input features, and the data window size. We also evaluate the generability of model learning for unknown contexts, such as an unknown car, driver, or environment.
Access here
In this research, we developed models for the speed bump detection in segments of cobblestone and asphalt pavements. We applied Deep Learning techniques based on Convolutional Neural Network (CNN), Gated Recurrent Unit (GRU), Long Short-Term Memory Network (LSTM), Convolutional Neural Network Long Short-Term Memory (CNN-LSTM), and Convolutional Long Short-Term Memory (ConvLSTM). We analyzed some aspects, such as the influence of the vehicle data collection placement and the data window size. We also evaluate the generability of model learning for unknown contexts, such as an unknown car, driver, or environment.
Access here
In this research, we developed models to classify the road roughness condition in three quality levels: bad, regular and good road. We applied Deep Learning techniques based on Convolutional Neural Network (CNN), Gated Recurrent Unit (GRU), Long Short-Term Memory Network (LSTM), Convolutional Neural Network Long Short-Term Memory (CNN-LSTM), and Convolutional Long Short-Term Memory (ConvLSTM). We analyzed some aspects, such as the influence of the vehicle data collection placement and the data window size. We also evaluate the generability of model learning for unknown contexts, such as an unknown car, driver, or environment.
Access here
Master's thesis. Access here
Conference paper. Access here
Conference paper. Access here
The nine datasets collected are available for download at Kaggle and Arquivos UFSC. To collect and preprocess the raw data that resulted in the PVS datasets, the following projects were used:
-
- Source-code used to manipulate the sensors in the data collection.
-
MPU-9250 and GPS Raw Data Pre-Processing
- Source-code used for initial adjustments to inertial sensor raw data, such as combining with GPS data and creating data class labels.
-
MPU-9250 Data Plot Video Creator
- Source-code to create videos with data plot of speed and signals from inertial sensors.
For data collection, we use several passive approach sensors, detailed in the following table:
| Hardware | Sensor | Data | Sampling Rate |
|---|---|---|---|
| HP Webcam HD-4110 | Camera | 720p Video | 30 Hz |
| Xiaomi Mi 8 | GPS | Speed in m/s, latitude, longitude, etc. | 1 Hz |
| MPU-9250 | Accelerometer | 3-axis acceleration in m/s² | 100 Hz |
| MPU-9250 | Gyroscope | 3-axis rotation rate in deg/s | 100 Hz |
| MPU-9250 | Magnetometer | 3-axis ambient geomagnetic field in µT | 100 Hz |
| MPU-9250 | Temperature | Sensor temperature in ◦C | 100 Hz |
All the hardware equipment was attached to the vehicle as shown in the next figure. The camera was placed on the outside car roof (1), and the GPS receiver was placed internally on the dashboard (2). Two networks with MPU-9250 modules were distributed in the vehicle to get data coming from different points. Thus, each end of the front axle (right and left side) received one of the sensor networks, where a module was attached to the control arm (4), located below and near to the vehicle’s suspension system; another module was placed above and near the suspension system, attached to the body immediately above the tire (3); and a third module was attached to the vehicle’s dashboard (2), inside the cabin.
The data were produced in three different vehicles, with three different drivers, in three different environments in which there are three different surface types, in addition to variations in conservation state and presence of obstacles and anomalies, such as speed bumps and potholes. The following table details the data collection contexts.
| DataSet | Vehicle | Driver | Scenario | Distance |
|---|---|---|---|---|
| PVS 1 | Volkswagen Saveiro | Driver 1 | Scenario 1 | 13.81 km |
| PVS 2 | Volkswagen Saveiro | Driver 1 | Scenario 2 | 11.62 km |
| PVS 3 | Volkswagen Saveiro | Driver 1 | Scenario 3 | 10.72 km |
| PVS 4 | Fiat Bravo | Driver 2 | Scenario 1 | 13.81 km |
| PVS 5 | Fiat Bravo | Driver 2 | Scenario 2 | 11.63 km |
| PVS 6 | Fiat Bravo | Driver 2 | Scenario 3 | 10.73 km |
| PVS 7 | Fiat Palio | Driver 3 | Scenario 1 | 13.78 km |
| PVS 8 | Fiat Palio | Driver 3 | Scenario 2 | 11.63 km |
| PVS 9 | Fiat Palio | Driver 3 | Scenario 3 | 10.74 km |
Each dataset consists of the following files:
| File | Description |
|---|---|
| dataset_gps.csv | GPS data, including latitude, longitude, altitude, speed, accuracy, etc. |
| dataset_gps_mpu_left.csv | Inertial sensor data on the left side of the vehicle, combined with GPS data. |
| dataset_gps_mpu_right.csv | Inertial sensor data on the right side of the vehicle, combined with GPS data. |
| dataset_labels.csv | Data classes for each sample data in the dataset (for both sides). |
| dataset_mpu_left.csv | Inertial sensor data on the left side of the vehicle. |
| dataset_mpu_right.csv | Inertial sensor data on the right side of the vehicle. |
| dataset_settings_left.csv | Settings of the inertial sensors placed on the left side of the vehicle. Includes measurement range, resolution, etc. |
| dataset_settings_right.csv | Settings of the inertial sensors placed on the right side of the vehicle. Includes measurement range, resolution, etc. |
| map.html | Interactive maps with data classes. |
| video_dataset_left.mp4 | Video with data plotted from inertial sensors and speed, sampled on the left side of the vehicle. |
| video_dataset_right.mp4 | Video with data plotted from inertial sensors and speed, sampled on the right side of the vehicle. |
| video_environment.mp4 | External environment video. |
| video_environment_dataset_left.mp4 | Videos side by side from video_environment.mp4 and video_dataset_left.mp4 |
| video_environment_dataset_right.mp4 | Videos side by side from video_environment.mp4 and video_dataset_right.mp4 |
You can find the playlist of videos side by side on Youtube.
The data classes are available in the dataset_labels.csv file were built in one-hot-encoded form. The available labs are detailed below. In the folder Best Models is the Jupyter Notebook PVS - Data Exploration.ipynb in which you can explore the data through maps and tables with quantitative samples and distribution of data classes. Access here.
| Description | Label |
|---|---|
| Dirt Road | dirt_road |
| Cobblestone Road | cobblestone_road |
| Asphalt Road | asphalt_road |
| Description | Label |
|---|---|
| Paved Road | paved_road |
| Unpaved Road | unpaved_road |
| Description | Label |
|---|---|
| Good Road | good_road_left, good_road_right |
| Regular Road | regular_road_left, regular_road_right |
| Bad Road | bad_road_left, bad_road_right |
| Description | Label |
|---|---|
| No Speed Bump | no_speed_bump |
| Speed Bump in Asphalt | speed_bump_asphalt |
| Speed Bump in Cobblestone | speed_bump_cobblestone |
For those who want to use the best models produced in this research in a simplified way, they are available in the Best Models folder, access here. We provide the following best models:
- PVS - Data Exploration
- Road Surface Type Classification
- Road Roughness Condition Classification
- Speed Bump Classification
The recognized and classified patterns of vehicular perception have wide applicability. Below are some applications that can use this information:
- Autonomous Vehicles
- Advanced Vehicle Control Systems
- Advanced Driver Assistance Systems
- Advanced Public Transport Management Systems
- Advanced Traffic Management Systems
- Advanced Traveler Information Systems
- Vehicle Black Box
- Evaluation Software for Control and Quality Assurance in Civil Engineering
- Mobile Crowdsensing Applications
To cite PVS Datasets, use the reference below:
J. Menegazzo and A. von Wangenheim, "Multi-Contextual and Multi-Aspect Analysis for Road Surface Type Classification Through Inertial Sensors and Deep Learning," 2020 X Brazilian Symposium on Computing Systems Engineering (SBESC), Florianopolis, 2020, pp. 1-8, doi: 10.1109/SBESC51047.2020.9277846.
@INPROCEEDINGS{9277846,
author={J. {Menegazzo} and A. {von Wangenheim}},
booktitle={2020 X Brazilian Symposium on Computing Systems Engineering (SBESC)},
title={Multi-Contextual and Multi-Aspect Analysis for Road Surface Type Classification Through Inertial Sensors and Deep Learning},
year={2020},
volume={},
number={},
pages={1-8},
doi={10.1109/SBESC51047.2020.9277846}
}This project is under Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0). Please see License File for more information.