TY - GEN
T1 - Cost-effective Simulation-based Test Selection in Self-driving Cars Software with SDC-Scissor
AU - Birchler, Christian
AU - Ganz, Nicolas
AU - Khatiri, Sajad
AU - Gambi, Alessio
AU - Panichella, Sebastiano
N1 - Funding Information:
We gratefully acknowledge the Horizon 2020 (EU Commission) support for the project COSMOS (DevOps for Complex Cyber-physical Systems), Project No. 957254-COSMOS) and the DFG project STUNT (DFG Grant Agreement n. FR 2955/4-1).
Publisher Copyright:
© 2022 IEEE.
PY - 2022
Y1 - 2022
N2 - Simulation platforms facilitate the continuous development of complex systems such as self-driving cars (SDCs). However, previous results on testing SDCs using simulations have shown that most of the automatically generated tests do not strongly contribute to establishing confidence in the quality and reliability of the SDC. Therefore, those tests can be characterized as 'uninformative', and running them generally means wasting precious computational resources. We address this issue with SDC-Scissor, a framework that leverages Machine Learning to identify simulation-based tests that are unlikely to detect faults in the SDC software under test and skip them before their execution. Consequently, by filtering out those tests, SDC-Scissor reduces the number of long-running simulations to execute and drastically increases the cost-effectiveness of simulation-based testing of SDCs software. Our evaluation concerning two large datasets and around 12'000 tests showed that SDC-Scissor achieved a higher classification F1-score (between 47% and 90%) than a randomized baseline in identifying tests that lead to a fault and reduced the time spent running uninformative tests (speedup between 107% and 170%). Webpage & Video: https://github.com/ChristianBirchler/sdc-scissor
AB - Simulation platforms facilitate the continuous development of complex systems such as self-driving cars (SDCs). However, previous results on testing SDCs using simulations have shown that most of the automatically generated tests do not strongly contribute to establishing confidence in the quality and reliability of the SDC. Therefore, those tests can be characterized as 'uninformative', and running them generally means wasting precious computational resources. We address this issue with SDC-Scissor, a framework that leverages Machine Learning to identify simulation-based tests that are unlikely to detect faults in the SDC software under test and skip them before their execution. Consequently, by filtering out those tests, SDC-Scissor reduces the number of long-running simulations to execute and drastically increases the cost-effectiveness of simulation-based testing of SDCs software. Our evaluation concerning two large datasets and around 12'000 tests showed that SDC-Scissor achieved a higher classification F1-score (between 47% and 90%) than a randomized baseline in identifying tests that lead to a fault and reduced the time spent running uninformative tests (speedup between 107% and 170%). Webpage & Video: https://github.com/ChristianBirchler/sdc-scissor
KW - Self-driving cars
KW - Software Simulation
KW - Regression Testing
KW - Test Case Selection
KW - Continuous Integration
UR - http://www.scopus.com/inward/record.url?scp=85124278186&partnerID=8YFLogxK
U2 - 10.1109/SANER53432.2022.00030
DO - 10.1109/SANER53432.2022.00030
M3 - Conference contribution
SN - 978-1-6654-3786-8
T3 - Proceedings - 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2022
SP - 164
EP - 168
BT - Proceedings - 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2022
PB - IEEE
CY - 10662 LOS VAQUEROS CIRCLE, PO BOX 3014, LOS ALAMITOS, CA 90720-1264 USA
ER -