跨模态自适应融合网络用于多模态关系抽取

Feature-adaptive fusion network for multimodal relation extraction

  • 摘要: 针对现有多模态关系抽取(MRE)模型难以有效解决多模态特征融合不充分和实际应用下多模态数据不相关的问题,提出一种跨模态自适应融合网络(CAFNeT),旨在提升多模态关系抽取的性能。首先基于扩散生成模型生成新的图像数据,并与原始文本数据组成新的图文对作为网络的辅助信息;其次在多层次跨模态交互模块中设计了一种自适应多头交叉注意力机制,分别从句子层面和词汇层面实现粗细粒度的跨模态特征交互,深入地捕捉模态间的语义依赖关系;最后,在多模态自适应融合模块中设计了一种基于辅助信息的图文不相关修正机制,通过自适应调节机制动态调整原始图文对信息与辅助信息在最终表示中的贡献比例,有效抑制图文不相关数据信息的影响,同时避免关键信息丢失。在公开多模态关系抽取数据集上的实验结果表明,CAFNeT在精确率、召回率和F1值上分别达到91.30%、90.16%与90.72%,相较于基准模型均有一定提升,进一步的消融研究证实了所提方法的有效性。

     

    Abstract: To address the challenges of insufficient multimodal feature fusion and irrelevant multimodal data in practical applications faced by existing multimodal relation extraction (MRE) models, this paper proposes a cross-modal adaptive fusion network (CAFNeT), aiming to enhance MRE performance. Firstly, novel image data is generated using diffusion generative models and combined with the original text data to form new image-text pairs serving as auxiliary information for the network. Secondly, a multi-level cross-modal interaction module incorporating an adaptive multi-head cross-attention mechanism is designed. This mechanism facilitates coarse- and fine-grained cross-modal feature interactions at both the sentence and word levels, enabling the deep capture of semantic dependencies between modalities. Finally, a multimodal adaptive fusion module is introduced, featuring an image-text irrelevance correction mechanism based on auxiliary information. This mechanism dynamically adjusts the contribution proportions of the original image-text pair information and the auxiliary information in the final representation through an adaptive regulation mechanism. This effectively suppresses the influence of irrelevant image-text data while avoiding the loss of critical information. Experimental results on public MRE datasets demonstrate that CAFNeT achieves precision, recall, and F1-scores of 91.30%, 90.16%, and 90.72%, respectively, showing improvements over baseline models. Further ablation studies confirm the effectiveness of the proposed method.

     

/

返回文章
返回