Abstract: In order to make convolutional neural network get optimal connection automatically without experienced guidance and improve the optimizing effectiveness for parameters of convolutional neural network, a new method using both particle swarm optimization algorithm and discrete particle swarm optimization algorithm is proposed to optimize parameters and feature maps connecting structure of convolutional neural network. The particle swarm optimization is applied to optimize the weights of convolutional neural network at first, and then the discrete particle swarm optimization is applied to optimize feature maps connections between sub-sampling layer and convolutional layer. The method is applied to MNIST database and CIFAR-10 database, compared to convolutional neural networks of other connecting structures and other recognition methods, results shown that this method can optimize the parameters and structure of the network effectively, accelerate network convergence and improve the recognition accuracy.