基于个性化和记忆机制的多模态情感计算模型

doi:10.16182/j.issn1004731x.joss.20-0899

摘要/Abstract

摘要：

随着情感计算研究的不断深入，记忆、个性化、情感之间的密切联系逐渐引起研究者的重视。已有方法在机器情感的感知、理解和表达方面仍存在着诸多不足。提出一种集情感感知、理解和表达于一体的情感计算模型。该模型是一个面向记忆机制的、接受多模态输入（视觉、听觉、词汇）的深度网络感知模型，并应用一种模糊化的情感集成决策来实现不确定性情感的理解。实验证明：该模型在各类多模态的情感计算中都有着较好的表现。

关键词: 情感计算, 计算情感模型, 深度神经网络, 情感集成决策, 个性化

Abstract:

With the development of affective computing, the correlation of memory, individuation and emotion is more and more important. Focus on the machine emotion shortcomings in the perception, understanding and expression, an emotion computing model integrating the emotion perception, understanding and expression is proposed. The model is a memory-oriented deep network perception model that accepts multiple modal inputs (visual, auditory, lexical) and applies a fuzzy emotion integration decision to realize the understanding of uncertain emotions. The simulation experiments prove that the model has a good performance in all kinds of multimodal affective computing.

Key words: affective computing, computational model of emotion, deep neural network, integrated decision, personalization

中图分类号:

TP311

周思锦, 陈棣成, 涂耿, 姜大志. 基于个性化和记忆机制的多模态情感计算模型[J]. 系统仿真学报, 2022, 34(4): 745-758.

Sijin Zhou, Dicheng Chen, Geng Tu, Dazhi Jiang. Multi-modality Affective Computing Model Based on Personality and Memory Mechanism[J]. Journal of System Simulation, 2022, 34(4): 745-758.

图/表 18

图1

图2

图3

图4

图5

图6

图7

图8

图9

表1

表2

表3

表4

表5

表6

图10

图11

图12

参考文献 39

1	Minsky M. Society of Mind[M]. New York: Simon and Schuster, 1988.
2	Picard R W. Affective Computing[M]. London: MIT Press, 1997.
3	Richards J M, Gross J J. Personality and Emotional Memory: How Regulating Emotion Impairs Memory for Emotional Events[J]. Journal of Research in Personality(S0092-6566), 2006, 40(5): 631-651.
4	Armony J. Computational Models of Emotion[C]//IEEE International Joint Conference on Neural Networks. Piscataway: IEEE, 2005: 1598-1602.
5	Keltner D, Oatley K, Jenkins J M. Understanding Emotions[M]. London: Wiley Global Education, 2018.
6	Davidson R J, Abercrombie H, Nitschke J B, et al. Regional Brain Function, Emotion and Disorders of Emotion[J]. Current Opinion in Neurobiology(S0959-4388), 1999, 9(2): 228-234.
7	Scherer K R, Bänziger T. Emotional Expression in Prosody: A Review and an Agenda for Future Research[C]//Speech Prosody. Lisbon: International Conference, 2004: 359-366.
8	Irie G, Satou T, Kojima A, et al. Affective Audio-visual Words and Latent Topic Driving Model for Realizing Movie Affective Scene Classification[J]. IEEE Transactions on Multimedia(S1520-9210), 2010, 12(6): 523-535.
9	Kang H B. Affective Content Detection Using HMMs[C]//The 11th ACM International Conference on Multimedia. New York: ACM, 2003: 259-262.
10	Kang H B. Emotional Event Detection Using Relevance Feedback[C]//International Conference on Image Processing (Cat. No. 03CH37429). Piscataway: IEEE, 2003: I- 721.
11	Zhang S, Huang Q, Jiang S, et al. Affective Visualization and Retrieval for Music Video[J]. IEEE Transactions on Multimedia(S1520-9210), 2010, 12(6): 510-522.
12	Zhang S, Tian Q, Jiang S, et al. Affective MTV Analysis Based on Arousal and Valence Features[C]// IEEE International Conference on Multimedia and Expo. Piscataway: IEEE, 2008: 1369-1372.
13	Spaulding S, Breazeal C. Towards Affect-Awareness for Social Robots[C]//AAAI Fall Symposia. Palo Alto: AAAI, 2015: 128-130.
14	Reisenzein R, Hudlicka E, Dastani M, et al. Computational Modeling of Emotion: Toward Improving the Inter-and Intradisciplinary Exchange[J]. IEEE Transactions on Affective Computing(S1949-3045), 2013, 4(3): 246-266.
15	Teng S, Wang Z, Wang L, et al. Affective Computing Model Based on Markov Chain[J]. Computer Engineering(S1000-3428), 2005, 31(5): 17-19.
16	滕少冬. 应用于个人机器人的人工情感模型研究[D]. 北京: 北京科技大学, 2006.
	Teng Shaodong. Research on Artificial Psychology Model Applied in Personal Robot[D]. Beijing: Beijing University of Science and Technology, 2006.
17	Xiaolan P, Lun X, Xin L, et al. Emotional State Transition Model Based on Stimulus and Personality Characteristics[J]. China Communications(S1637-5447), 2013, 10(6): 146-155.
18	Yi W, Zhi-Liang W, Wei W. Research on Associative Memory Models of Emotional Robots[J]. Advances in Mechanical Engineering(S1687-8132), 2014, 6(1): 208153.
19	Masuyama N, Loo C K, Seera M. Personality Affected Robotic Emotional Model with Associative Memory for Human-robot Interaction[J]. Neurocomputing(S0925-2312),
	2018, 272(10): 213-225.
20	Ibanez R V, Keysermann M U, Vargas P A. Emotional Memories in Autonomous Robots[C]// The 23rd IEEE International Symposium on Robot and Human Interactive Communication. Piscataway: IEEE, 2014: 405-410.
21	Tiedens L Z, Linton S. Judgment Under Emotional Certainty and Uncertainty: the Effects of Specific Emotions on Information Processing[J]. Journal of Personality and Social Psychology(S0022-3514), 2001, 81(6): 973-988.
22	Eich E, Metcalfe J. Mood Dependent Memory for Internal Versus External Events[J]. Journal of Experimental Psychology: Learning, Memory, and Cognition(S0278-7393), 1989, 15(3): 443-455.
23	Scherer K R. Towards a Prediction and Data Driven Computational Process Model of Emotion[J]. IEEE Transactions on Affective Computing(S1949-3045), 2019, 12(2): 279-292.
24	Frijda N H, Bower G H, Hamilton V. Cognitive Perspectives on Emotion and Motivation[M]. Abingdon: Taylor & Francis, 1988.
25	Jiang D, Wu K, Chen D, et al. A Probability and Integrated Learning Based Classification Algorithm for High-Level Human Emotion Recognition Problems[J]. Measurement(S0263-2241), 2020, 150(1): 107049.
26	Salway A, Graham M. Extracting Information about Emotions in Films[C]//The 11th ACM International Conference on Multimedia. New York: ACM, 2003: 299-302.
27	Ekman P, Dalgleish T, Power M. Handbook of Cognition and Emotion[M]. New Jersey: Basic Emotions Chapter, John Wiley & Sons, Ltd, 1999.
28	Chan C H, Jones G J F. Affect-based Indexing and Retrieval of Films[C]//The 13th Annual ACM International Conference on Multimedia. New York: ACM, 2005: 427-430.
29	Huang S J, Gao N, Chen S. Multi-Instance Multi-Label Active Learning[C]// IJCAI. San Francisco: Margan Kaufmann, 2017: 1886-1892.
30	Arifin S, Cheung P Y K. A Computation Method for Video Segmentation Utilizing the Pleasure-Arousal-Dominance Emotional Information[C]//The 15th ACM International Conference on Multimedia. New York: ACM, 2007: 68-77.
31	Arifin S, Cheung P Y K. Affective Level Video Segmentation by Utilizing the Pleasure-Arousal-Dominance Information[J]. IEEE Transactions on Multimedia(S1520-9210), 2008, 10(7): 1325-1341.
32	Canini L, Benini S, Migliorati P, et al. Emotional Identity of Movies[C]// The 16th IEEE International Conference on Image Processing (ICIP). Piscataway: IEEE, 2009: 1821-1824.
33	Lev G, Klein B, Wolf L. In Defense of Word Embedding for Generic Text Representation[C]//International Conference on Applications of Natural Language to Information Systems. Berlin: Springer, 2015: 35-50.
34	韩晶, 解仑, 刘欣, 等. 基于Gross认知重评的机器人认知情感交互模型[J]. 东南大学学报(自然科学版), 2015, 45(2): 270-274.
	Han Jing, Xie Lun, Liu Xin, et al. Robot Cognitive Emotional Interaction Model Based on Gross Cognitive Reevaluation[J]. Journal of Southeast University (Natural Science), 2015, 45(2): 270-274.
35	Boleda G, Gulordava K, Aina L. Putting Words in Context: LSTM Language Models and Lexical Ambiguity[C]// The 57th Annual Meeting of the Association for Computational Linguistics. Pennsylvania: ACL, 2019: 3342-3349.
36	Li H, Xu H. Video-based Sentiment Analysis with hvnLBP-TOP Feature and bi-LSTM[C]//The AAAI Conference on Artificial Intelligence. Palo Alto: AAAI, 2019: 9963-9964.
37	Busso C, Bulut M, Lee C C, et al. IEMOCAP: Interactive Emotional Dyadic Motion Capture Database[J]. Language Resources and Evaluation(S1574-020X), 2008, 42(4): 335-339.
38	Zadeh A A B, Liang P P, Poria S, et al. Multimodal Language Analysis in the Wild: Cmu-mosei Dataset and Interpretable Dynamic Fusion Graph[C]//The 56th Annual Meeting of the Association for Computational Linguistics. Pennsylvania: ACL, 2018: 2236-2246.
39	Masuyama N, Loo C K, Seera M. Personality Affected Robotic Emotional Model with Associative Memory for Human-Robot Interaction[J]. Neurocomputing(S0925-2312), 2018, 272(19): 213-225.

分类器	IEMOCAP			CMU-MOSEI
分类器	LSTM	Bi-LSTM	LMFDB-LSTM	LSTM	Bi-LSTM	LMFDB-LSTM
A	0.412 5	0.452 7	0.474 0	0.545 8	0.574 6	0.573 9
L	0.582 1	0.602 2	0.618 3	0.579 5	0. 5 861	0.601 4
V	0.321 5	0.339 7	0.325 2	0.535 7	0.526 2	0.558 1
AL	0.600 5	0.612 4	0.621 6	0.585 8	0.586 6	0.583 7
AV	0.402 2	0.425 3	0.428 9	0.570 2	0.575 4	0.576 6
TV	0.603 2	0.614 3	0.627 6	0.593 6	0.600 3	0.604 9
AVL	0.608 4	0.618 0	0.627 5	0.583 1	0.599 7	0.604 1

情绪	精确率	召回率	F1值
积极	0.887	0.812	0.846
消极	0.690	0.785	0.733
中性	0.793	0.720	0.759

情绪	精确率	召回率	F1值
积极	0.805	0.534	0.647
消极	0.655	0.913	0.752
中性	0.856	0.841	0.850

情绪	精确率	召回率	F1值
积极	0.905	0.872	0.887
消极	0.882	0.880	0.880
中性	0.867	0.900	0.882

情绪	精确率	召回率	F1值
积极	0.899	0.910	0.903
消极	0.966	0.873	0.914
中性	0.861	0.930	0.900