GCTR:粒度统一的跨模态文本行人检索网络模型

GCTR:A Granularity-Unified Cross-Modal Text-person Retrieval Model

关键词:  跨模态检索  图文检索  行人检索  视觉语言预训练  粒度特征增强

Key words:  cross-modal retrieval  text-image retrieval  person retrieval  visual language pre-training  granularity feature enhancement

摘要:

现有的文本行人检索网络模型在检索任务中缺乏对图文语义联系的关注，且容易忽略文本与图像特征之间的粒度差异，针对这两大问题，本研究提出一种粒度统一的跨模态文本行人检索网络模型(Granularity-unified Cross-modal Text-person Retrieval model，GCTR)。首先，GCTR利用具备跨模态迁移知识能力的视觉语言预训练模型来获取具有基础关联性的文本和图像特征；其次，本研究提出一个跨模态粒度特征增强模块(Cross-Model Feature Enhancement module，CMFE)，它利用跨模态特征增强码表(Enhanced Cross-modal Feature Codebook，ECFC)获取具有统一粒度的图像文本特征，解决了图文特征粒度差异的问题；最后,结合局部和全局的匹配损失策略完成模型的训练。GCTR在CUHK-PEDES、ICFG-PEDES和RSTPReid 3个公开数据集上的表现均优于现有的主流模型，证明了GCTR在跨模态文本行人检索任务上的优越性。

Abstract:

The existing text-image person retrieval tasks-lack the attention to the semantic connection between text and image and neglects the granularity difference between text and image features.In view of the problems above,a Granularity-unified Cross-Modal Text-person Retrieval model (GCTR) is proposed.Firstly,GCTR utilizes the contrastive language-image pre-training model,which possesses the cross-modal transfer learning capability,to obtain text and image features with basic relevance.Secondly,a Cross-model Feature Enhancement module (CMFE) is proposed,which utilizes the Enhanced Cross-modal Feature Codebook (ECFC) to obtain text and image features with unified granularity,solving the problem of granularity difference between text and image features.Finally,a combination of local and global matching loss strategies is employed to optimize the model training process.The retrieval evaluation metrics achieved by GCTR on three public datasets CUHK-PEDES,ICFG-PEDES,and RSTPReid surpass those of existing state-of-the-art methods,demonstrating the effectiveness of the proposed GCTR model in cross-modal text-image person retrieval tasks.