Abstract: A fresco restoration model under multi-stage stacking fusion is proposed to solve the problem of difficult restoration of broken frescoes at Dunhuang. The Unet structure is used for coarse repair in the first stage and for maximizing the extraction of the required feature information, and the adversarial network (LsGan) is added to enhance the repair effect in this stage. In the second stage, the restoration is refined, mainly by restoring textures and refining occluded areas, introducing and improving the multi-head connection and multi-scale branching stacking modules, extracting and fusing multi-stage information from the murals. Finally, in order to consider the global details of the fix, the codec structure of the big feeling field is used and the mobile ViT module is introduced. The channels of the mural images are also separated and analyzed, introducing polarization attention that is insensitive to the channels. The experimental results show that the model can solve the difficult problem of fresco texture and detail restoration well. Compared with the optimal data of the selected restoration algorithm, the peak signal-to-noise ratio improved by 3.312 at a mask area of 5% to 20%, and the average peak signal-to-noise ratio is improved by 1.02 at a mask area ranging from 5% to 80%.