Abstract: Automatic recognition of metal coating defects has significant value in realistic applications. As deep learning makes breakthrough in surface defect segmentation for a variety of materials, most of deep convolutional neural network segmentation models are trained in an end-to-end manner. However, it is difficult to exploit prior knowledge about metal coating defects in end-to-end deep learning and adapt to the variable scale of the defects and the limited training data. This paper proposes a defect segmentation algorithm based on prior knowledge about metal coating defects to unify U-Net, a deep learning segmentation model for automatic metal coating defect recognition. This anomaly segmentation is based on Hue channel distribution and edge response. Being trained in a knowledge driven manner, the model can exclude outliers from training data and effectively avoid over-fitting. On a metal coating defect image dataset with four defect types, including crack, blister, rusting and flaking, the proposed method achieves 81.24% mIoU, which is advantageous over end-to-end deep learning. The experiment shows that knowledge-driven model can boost the performance of deep learning models in metal coating defect segmentation.