Abstract: This paper presents a method based on multi-modal convolution neural networks to segment the brain CT angiography image (CTA) for brain vessel. This method firstly processes the original image by adopting the Gaussian and Laplacian filter, respectively, and constructs the multi model image as the input by combining processed images with the original image together. Next, these multi model images will be respectively segmented through a number of parallel convolutional neural networks. Finally, all the segmentation results are fused by employing the linear regression to extract the brain vessel. By evaluating the experiment with the real data acquired from the hospital, it can be proved that the convolution neural network is an effective method for segmenting the cerebral vessels. Moreover, the final result shows that our proposed segmenting method is more accurate than the existing algorithms.