基于神经网络的图像描述方法研究

doi:10.16182/j.issn1004731x.joss.18-0310

系统仿真学报 ›› 2020, Vol. 32 ›› Issue (4): 601-611.doi: 10.16182/j.issn1004731x.joss.18-0310

基于神经网络的图像描述方法研究

孔锐¹, 谢玮¹, 雷泰²

1. 暨南大学智能科学与工程学院,广东珠海 519070;
2. 暨南大学信息科学技术学院,广东广州 510632

收稿日期:2018-05-24 修回日期:2018-09-26 出版日期:2020-04-18 发布日期:2020-04-16
作者简介:孔锐(1964-),男,安徽合肥,博士,教授,研究方向为人脸识别和机器学习;谢玮(1994-),男,湖南娄底,硕士生,研究方向为计算机视觉和深度学习。
基金资助:
广东省科技计划(产学研合作)(2016B0909 18098)

Research on Image Description Method Based on Neural Network

Kong Rui¹, Xie Wei¹, Lei Tai²

1. School of Intelligent Systems Science and Engineering, Jinan University, Zhuhai 519070, China;
2. College of Information Science and Technology, Jinan University, Guangzhou 510632, China

Received:2018-05-24 Revised:2018-09-26 Online:2020-04-18 Published:2020-04-16

摘要/Abstract

摘要： 自动识别和描述图像的内容是人工智能中一个重要的研究方向,它涉及计算机视觉和自然语言处理技术。针对这一难题,提出了一种由深层神经网络模型生成自然语言句子来描述图像内容的方法。该方法提出的模型由卷积神经网络(Convolution Neural Network,CNN)和循环神经网络(Recurrent Neural Network,RNN)组成,其中,CNN用来提取输入图像的特征生成固定长度的特征向量,该特征向量初始化RNN来生成句子。在MSCOCO图像描述数据集上的实验结果表明了该模型所生成句子的语法准确性和语义准确性,且优于先前的基线模型。

关键词: 图像描述, 神经网络, 语言模型, 深度学习

Abstract: The automatic recognition and automatically describing image content is an important research direction to the artificial intelligence to connect the computer vision and the natural language processing. A method of describing the image content is proposed to generate the natural language by using the deep neural network model. The model consists of a convolutional neural network (CNN) and a recurrent neural network (RNN). The CNN is used to extract features of the input image to generate a fixed-length feature vector, which initializes the RNN to generate the sentences. Experimental results on the MSCOCO image description dataset show the syntactic accuracy and the semantic accuracy of the sentences generated by the model is superior to the previous baseline model.

Key words: image description, neural networks, language model, deep learning

中图分类号:

TP391.9

孔锐, 谢玮, 雷泰. 基于神经网络的图像描述方法研究[J]. 系统仿真学报, 2020, 32(4): 601-611.

Kong Rui, Xie Wei, Lei Tai. Research on Image Description Method Based on Neural Network[J]. Journal of System Simulation, 2020, 32(4): 601-611.

参考文献 20

[1]	石祥滨, 房雪键, 张德园, 等. 基于深度学习混合模型迁移学习的图像分类[J]. 系统仿真学报, 2016, 28(1): 167-173,182.Shi Xiangbin, Fang Xuejian, Zhang Deyuan, et al.Image classification based on mixed deep learning model transfer learning[J]. Journal of System Simulation, 2016, 28(1): 167-173,182.
[2]	许锋, 卢建刚, 孙优贤. 神经网络在图像处理中的应用[J]. 信息与控制, 2003, 32(4): 344-351.Xu Feng, Lu Jiangang, Sun Youxian.Application of neural network in image processing[J]. Information and Control, 2003, 32(4): 344-351.
[3]	Farhadi A, Hejrati M, Sadeghi M A, et al.Every picture tells a story: Generating sentences from images[C]// European conference on computer vision. Heidelberg: Springer, Berlin, 2010: 15-29.
[4]	Li S, Kulkarni G, Berg T L, et al.Composing simple image descriptions using web-scale n-grams[C]// Proceedings of the Fifteenth Conference on Computational Natural Language Learning. Stroudsburg: Association for Computational Linguistics, 2011: 220-228.
[5]	Kulkarni G, Premraj V, Ordonez V, et al.Babytalk: Understanding and generating simple image descriptions[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence (S0162-8828), 2013, 35(12): 2891-2903.
[6]	张红斌, 姬东鸿, 尹兰, 等. 基于梯度核特征及 N-gram 模型的商品图像句子标注[J]. 计算机科学, 2016, 43(5): 269-273, 287.Zhang Hongbin, Ji Donghong, Yin Lan, et al.Product image sentence annotation based on gradient kernel feature and N-gram model[J]. Computer Science, 2016, 43(5): 269-273, 287.
[7]	Xu K, Ba J, Kiros R, et al.Show, attend and tell: Neural image caption generation with visual attention[C]// International Conference on Machine Learning. Lille Grand Palais: International Machine learning Society, 2015: 2048-2057.
[8]	Karpathy A, Fei-Fei L.Deep visual-semantic alignments for generating image descriptions[C]// Proceedings of the IEEE conference on computer vision and pattern recognition. Los Alamitos: IEEE Computer Society, 2015: 3128-3137.
[9]	Jia X, Gavves E, Fernando B, et al.Guiding the long-short term memory model for image caption generation[C]// Proceedings of the IEEE International Conference on Computer Vision. Piscataway: IEEE, 2015: 2407-2415.
[10]	Vinyals O, Toshev A, Bengio S, et al.Show and tell: A neural image caption generator[C]// Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on. Piscataway: IEEE, 2015: 3156-3164.
[11]	Vinyals O, Toshev A, Bengio S, et al.Show and tell: Lessons learned from the 2015 mscoco image captioning challenge[J]. IEEE Transactions on Pattern Analysis And Machine Intelligence (S0162-8828), 2017, 39(4): 652-663.
[12]	Szegedy C, Ioffe S, Vanhoucke V, et al.Inception-v4, inception-resnet and the impact of residual connections on learning[C]// Menlo Park: AAAI, 2017, 4: 12.
[13]	Mikolov T, Chen K, Corrado G, et al.Efficient estimation of word representations in vector space[J]. arXiv preprint arXiv (S2331-8422), 2013: 1301.3781.
[14]	Hochreiter S, Schmidhuber J.Long short-term memory[J]. Neural Computation (S0899-7667), 1997, 9(8): 1735-1780.
[15]	Lin T Y, Maire M, Belongie S, et al.Microsoft coco: Common objects in context[C]// European conference on computer vision. Cham: Springer, 2014: 740-755.
[16]	Zaremba W, Sutskever I, Vinyals O.Recurrent neural network regularization[J]. arXiv preprint arXiv (S2331-8422), 2014: 1409.2329.
[17]	Chen X L, Fang H, Lin T Y, et al. Microsoft COCO caption evaluation[EB/OL].2015. https://github.com/ tylin/ coco-caption.
[18]	Mao J, Xu W, Yang Y, et al.Deep captioning with multimodal recurrent neural networks (m-rnn)[J]. arXiv preprint arXiv (S2331-8422), 2014: 1412.6632.
[19]	Fang H, Gupta S, Iandola F, et al.From captions to visual concepts and back[C]// Proceedings of the IEEE conference on computer vision and pattern recognition. Los Alamitos: IEEE Computer Society, 2015: 1473-1482.
[20]	Devlin J, Cheng H, Fang H, et al.Language models for image captioning: The quirks and what works[J]. arXiv preprint arXiv (S2331-8422), 2015: 1505.01809.

基于神经网络的图像描述方法研究

Research on Image Description Method Based on Neural Network

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献 20

相关文章 15

编辑推荐

Metrics

本文评价

[1]	张森, 张孟炎, 邵敬平, 普杰信. 基于随机策略搜索的多机三维路径规划方法[J]. 系统仿真学报, 2022, 34(6): 1286-1295.
[2]	张立峰, 王会忍. 基于卷积神经网络及有限元仿真的电容层析成像图像重建[J]. 系统仿真学报, 2022, 34(4): 712-718.
[3]	康旭, 张晓峰. 基于生成对抗神经网络的雷达遥感数据增广方法[J]. 系统仿真学报, 2022, 34(4): 920-927.
[4]	周思锦, 陈棣成, 涂耿, 姜大志. 基于个性化和记忆机制的多模态情感计算模型[J]. 系统仿真学报, 2022, 34(4): 745-758.
[5]	孙晓安, 栾小丽, 刘飞. 基于智能优化灰色模型的电子固废预测[J]. 系统仿真学报, 2022, 34(3): 536-542.
[6]	魏娟, 游磊, 郭阳勇, 唐志海. 基于小波神经网络的多楼层疏散模型[J]. 系统仿真学报, 2022, 34(2): 269-277.
[7]	敖邦乾, 杨莎, 令狐金卿, 叶振环. 基于级联神经网络疲劳驾驶检测系统设计[J]. 系统仿真学报, 2022, 34(2): 323-333.
[8]	魏腾飞, 潘庭龙. 基于改进PSO优化LSTM网络的短期电力负荷预测[J]. 系统仿真学报, 2021, 33(8): 1866-1874.
[9]	林硕, 安磊, 高治军, 单丹, 尚文利. 结合栈式自编码及长短时记忆的入侵检测研究[J]. 系统仿真学报, 2021, 33(6): 1288-1296.
[10]	程文聪, 史小康, 王志刚. 基于生成对抗网络的仿真卫星云图生成方法[J]. 系统仿真学报, 2021, 33(6): 1297-1306.
[11]	赵瑛, 陆耀, 张健, 梁启弟, 龙炜. 基于深度神经网络的多视角人体动作识别[J]. 系统仿真学报, 2021, 33(5): 1019-1030.
[12]	刘晓琳, 吴竟祎. 飞机舵机电液加载系统控制补偿器设计与仿真[J]. 系统仿真学报, 2021, 33(5): 1031-1038.
[13]	于广宇, 董学平, 王祥民, 甘敏. 弹性网下基于LSTM的分解炉出口温度预测[J]. 系统仿真学报, 2021, 33(5): 1078-1085.
[14]	方群, 陈绪凯, 何昕, 祝玉军, 刘毅杨, 方阳阳, 李贺举. 基于SWIPT无线协作网络的中继选择算法研究[J]. 系统仿真学报, 2021, 33(4): 801-808.
[15]	仝卫国, 庞雪纯, 朱赓宏. 基于卷积神经网络的气液两相流流型识别方法[J]. 系统仿真学报, 2021, 33(4): 883-891.