Abstract: Objective Ultrasound (US) is one of a primary imageological examination and preoperative assessment for breast nodules. However, in the field of ultrasound diagnosis, it relies heavily on the experience of physicians due to the overlapping image expression of benign and malignant breast nodules. Computer-aided medical diagnosis has gradually become a hot spot of current research. In this paper, a heterogeneous multi-branch network (HMBN) is presented for benign and malignant classification of the breast ultrasound images. In HMBN, the image information includes ultrasound images and contrast-enhanced ultrasound (CEUS) images while non-image information includes patients’ age and other six pathological features. On the other hand, a fusion loss function suitable for this heterogeneous multi-branch network is also proposed. This loss function uses the minimum hyperspherical energy (MHE) based on additive angular margin loss to improve the classification accuracy. Experimental results show that on the breast ultrasound data set of 1303 cases collected, the classification accuracy of the proposed heterogeneous multi-branch network is 92.41%, which is 7.11% higher than the average diagnostic accuracy of doctors with five years of experience, and ranks among the best in diagnostic accuracy in comparison with other latest research results. It is proved that the accuracy and robustness of breast diagnosis are greatly improved by incorporating medical knowledge into the optimization process and adding contrast-enhanced ultrasound images and non-image information to the network.