Abstract: Infrared target classification has important values in target recognition. At present, convolutional neural network has achieved excellent performance in visible image classification. However, for infrared images, the available networks can't achieve satisfying results due to the small number of annotated samples and large imaging differences. In this paper, visible images are used as source domain, infrared images as target domain. Transfer learning is used to address the challenges in the deep learning framework. In the transfer learning, if the target domain network can represent the distribution of its domain well, the performance and generalization of the target domain network should be more effective. Therefore, the convolutional autoencoder is trained with a large number of unannotated infrared samples, which greatly enhances the feature representation in the infrared image domain. By reducing the feature distribution distance between the two domains, the feature distributions become similar. The classification performance in the source domain is transferred to the target domain. With the changes above, the accuracy rate is improved by 11.27% compared with the method based on the visible images fine-tuning.