Feature-adaptive fusion network for multimodal relation extraction

To address the challenges of insufficient multimodal feature fusion and irrelevant multimodal data in practical applications faced by existing multimodal relation extraction (MRE) models, this paper proposes a cross-modal adaptive fusion network (CAFNeT), aiming to enhance MRE performance. Firstly, novel image data is generated using diffusion generative models and combined with the original text data to form new image-text pairs serving as auxiliary information for the network. Secondly, a multi-level cross-modal interaction module incorporating an adaptive multi-head cross-attention mechanism is designed. This mechanism facilitates coarse- and fine-grained cross-modal feature interactions at both the sentence and word levels, enabling the deep capture of semantic dependencies between modalities. Finally, a multimodal adaptive fusion module is introduced, featuring an image-text irrelevance correction mechanism based on auxiliary information. This mechanism dynamically adjusts the contribution proportions of the original image-text pair information and the auxiliary information in the final representation through an adaptive regulation mechanism. This effectively suppresses the influence of irrelevant image-text data while avoiding the loss of critical information. Experimental results on public MRE datasets demonstrate that CAFNeT achieves precision, recall, and F1-scores of 91.30%, 90.16%, and 90.72%, respectively, showing improvements over baseline models. Further ablation studies confirm the effectiveness of the proposed method.