A news image captioning approach based on multimodal pointer-generator network


News image captioning aims to generate captions or descriptions for news images automatically, serving as draft captions for creating news image captions manually. News image captions are different from generic captions as news image captions contain more detailed information such as entity names and events. Therefore, both images on news and the accompanying text are the source of generating caption of news image. Pointer-generator network is a neural method defined for text summarization. This article proposes the Multimodal pointer-generation network by incorporating visual information into the original network for news image captioning. The multimodal attention mechanism is proposed by splitting attention into visual attention paid to the image and textual attention paid to the text. The multimodal pointer mechanism is proposed by using both textual attention and visual attention to compute pointer distributions, where visual attention is first transformed into textual attention via the word-image relationships. The multimodal coverage mechanism is defined to reduce repetitions of attentions or repetitions of pointer distributions. Experiments on the DailyMail test dataset and the out-of-domain BBC test dataset show that the proposed model outperforms the original pointer-generator network, the generic image captioning method, the extractive news image captioning method, and the LDA-based method according BLEU, METEOR, and ROUGL-L evaluations. Experiments also show that the proposed multimodal coverage mechanisms can improve the model, and that transforming visual attention to pointer distributions can improve the model.

Publication DOI: https://doi.org/10.1002/cpe.5721
Divisions: College of Engineering & Physical Sciences > School of Informatics and Digital Engineering > Computer Science
College of Engineering & Physical Sciences > Systems analytics research institute (SARI)
College of Engineering & Physical Sciences
Additional Information: This is the peer reviewed version of the following article: Chen, J, Zhuge, H. A news image captioning approach based on multimodal pointer‐generator network. Concurrency Computat Pract Exper. 2020;e5721, which has been published in final form at https://doi.org/10.1002/cpe.5721.  This article may be used for non-commercial purposes in accordance With Wiley Terms and Conditions for self-archiving.
Uncontrolled Keywords: image captioning,multimodal summarization,pointer-generator network,text summarization,Software,Theoretical Computer Science,Computer Science Applications,Computer Networks and Communications,Computational Theory and Mathematics
Publication ISSN: 1532-0634
Full Text Link:
Related URLs: http://www.scop ... tnerID=8YFLogxK (Scopus URL)
https://onlinel ... 0.1002/cpe.5721 (Publisher URL)
PURE Output Type: Article
Published Date: 2020-04-06
Published Online Date: 2020-04-06
Accepted Date: 2020-02-15
Authors: Chen, Jingqiang
Zhuge, Hai (ORCID Profile 0000-0001-8250-6408)



Version: Accepted Version

| Preview

Export / Share Citation


Additional statistics for this record