In this paper, we use Street View Imagery and InterpretML to study the subjective perception of driving in urban street environments to understand how drivers evaluate the environment and perceive safety. First, with the continuous development of urban road traffic in recent decades, it is increasingly recognized that traffic safety should not only be considered by measuring actual traffic accidents through indicators such as accident rate and fatality rate, but also by focusing on the changes in drivers’ perceptions of potential dangers and psychological stress experienced by drivers. To bridge this research gap, we collected subjective perception data from 500 drivers on different street images in Korea, applied semantic segmentation and object detection to extract visual features, and integrated socio-demographic variables into the analysis. The data were modeled using Explainable Boosting Machine (EBM) to understand how drivers evaluate the environment and perceive safety. The results show that factors such as vehicles and buildings significantly reduce drivers’ perceived safety, while open sky visibility enhances drivers’ perceived safety. In addition, the interaction effect reveals that we should pay attention to scenarios where road infrastructure and traffic users are present at the same time, and that social characteristics shape various perceptual responses to different road landscapes. The present study provides a comprehensive view of drivers’ subjective evaluation mechanisms, offering interpretable insights and practical suggestions for urban transportation infrastructure design. Data sources and data types should be expanded in future studies to further understand driver perception, as well as to compare modelling results from formal accident data to examine the gap between perceived and objective safety.