Abstract: The unsafe behavior of the driver is one of the important causes of many incidents. This paper presents a method to recognize unsafe driving behaviors based on the convolutional neural network. Firstly, the characteristics of the real-time image are extracted by the specific convolutional neural network, and then three kinds of behaviors are classified into two categories in parallel. The data set of unsafe driving behaviors in a real scene is established. The test on this dataset illustrates the efficiency and good generalization of the method. The experimental results show that the method achieves 99.85%, 99.62% and 98.68% accuracy for calling, smoking and unbelting in the data set of about 100 000 images, which is comparable to the results obtained by recent advanced Inception-v3 and Xception models.