Driving behavior is how drivers respond to actual driving environments and a major factor for road traffic safety. Recent advances in in-vehicle sensors facilitate continuous monitoring of driving behaviors; large-scale driving data have been accumulated. This study develops a framework to evaluate large-scale driving records and to establish clusters that can be used to identify potentially aggressive driving behaviors. The framework employs three steps of data analytic methods: abrupt change detection to extract meaningful driving events from raw data, feature extraction using an auto-encoder, and two-level clustering. This framework is applied to real driving data that were obtained from 43 taxis in Korean metropolitan cities. The application shows that the framework can characterize driving patterns from large-scale driving records and identify clusters with high potential for aggressive driving. The findings imply that the outcome clusters represent the norm of driving behavior and thus can be used as a reference in diagnosing other drivers’ behavior.