Abstract: To improve the fault diagnosis accuracy of pumping unit and reduce the storage memory of diagnosis model, a novel fault diagnosis method based on a lightweight attention convolutional neural network is designed to recognize the dynamometer card in this paper. The shape outline composed by the displacement and load of dynamometer card is transformed into the image. Regarding the model architecture, we leverage the depthwise separable convolution and propose a regularization attention module which can be embedded to the consecutive convolution layers. Each channel of the depthwise separable convolution layer is compressed and filtered by the mechanism provided by the model. It constructs the attention feature map, in which the feature is suppressed or enhanced. Regarding the model training, the attention-based loss function is presented to suppress the contribution of easy samples to the training loss. It makes the training to pay more attention to hard samples than easy ones. Finally, the proposed method is evaluated by the experiment. The experimental results show that the size of the model is only 5.4 Mb, while the diagnosis accuracy is 95.1%, which meet the requirements of fault diagnosis of the pumping unit.