Multi-modality Affective Computing Model Based on Personality and Memory Mechanism

doi:10.16182/j.issn1004731x.joss.20-0899

Abstract

Abstract:

With the development of affective computing, the correlation of memory, individuation and emotion is more and more important. Focus on the machine emotion shortcomings in the perception, understanding and expression, an emotion computing model integrating the emotion perception, understanding and expression is proposed. The model is a memory-oriented deep network perception model that accepts multiple modal inputs (visual, auditory, lexical) and applies a fuzzy emotion integration decision to realize the understanding of uncertain emotions. The simulation experiments prove that the model has a good performance in all kinds of multimodal affective computing.

Key words: affective computing, computational model of emotion, deep neural network, integrated decision, personalization

CLC Number:

TP311

Sijin Zhou, Dicheng Chen, Geng Tu, Dazhi Jiang. Multi-modality Affective Computing Model Based on Personality and Memory Mechanism[J]. Journal of System Simulation, 2022, 34(4): 745-758.

Figures/Tables 18

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig.6

Fig. 7

Fig. 8

Fig. 9

Table 1

Table 2

Table 3

Table 4

Table 5

Table 6

Fig. 10

Fig.11

Fig.12

References 39

1	Minsky M. Society of Mind[M]. New York: Simon and Schuster, 1988.
2	Picard R W. Affective Computing[M]. London: MIT Press, 1997.
3	Richards J M, Gross J J. Personality and Emotional Memory: How Regulating Emotion Impairs Memory for Emotional Events[J]. Journal of Research in Personality(S0092-6566), 2006, 40(5): 631-651.
4	Armony J. Computational Models of Emotion[C]//IEEE International Joint Conference on Neural Networks. Piscataway: IEEE, 2005: 1598-1602.
5	Keltner D, Oatley K, Jenkins J M. Understanding Emotions[M]. London: Wiley Global Education, 2018.
6	Davidson R J, Abercrombie H, Nitschke J B, et al. Regional Brain Function, Emotion and Disorders of Emotion[J]. Current Opinion in Neurobiology(S0959-4388), 1999, 9(2): 228-234.
7	Scherer K R, Bänziger T. Emotional Expression in Prosody: A Review and an Agenda for Future Research[C]//Speech Prosody. Lisbon: International Conference, 2004: 359-366.
8	Irie G, Satou T, Kojima A, et al. Affective Audio-visual Words and Latent Topic Driving Model for Realizing Movie Affective Scene Classification[J]. IEEE Transactions on Multimedia(S1520-9210), 2010, 12(6): 523-535.
9	Kang H B. Affective Content Detection Using HMMs[C]//The 11th ACM International Conference on Multimedia. New York: ACM, 2003: 259-262.
10	Kang H B. Emotional Event Detection Using Relevance Feedback[C]//International Conference on Image Processing (Cat. No. 03CH37429). Piscataway: IEEE, 2003: I- 721.
11	Zhang S, Huang Q, Jiang S, et al. Affective Visualization and Retrieval for Music Video[J]. IEEE Transactions on Multimedia(S1520-9210), 2010, 12(6): 510-522.
12	Zhang S, Tian Q, Jiang S, et al. Affective MTV Analysis Based on Arousal and Valence Features[C]// IEEE International Conference on Multimedia and Expo. Piscataway: IEEE, 2008: 1369-1372.
13	Spaulding S, Breazeal C. Towards Affect-Awareness for Social Robots[C]//AAAI Fall Symposia. Palo Alto: AAAI, 2015: 128-130.
14	Reisenzein R, Hudlicka E, Dastani M, et al. Computational Modeling of Emotion: Toward Improving the Inter-and Intradisciplinary Exchange[J]. IEEE Transactions on Affective Computing(S1949-3045), 2013, 4(3): 246-266.
15	Teng S, Wang Z, Wang L, et al. Affective Computing Model Based on Markov Chain[J]. Computer Engineering(S1000-3428), 2005, 31(5): 17-19.
16	滕少冬. 应用于个人机器人的人工情感模型研究[D]. 北京: 北京科技大学, 2006.
	Teng Shaodong. Research on Artificial Psychology Model Applied in Personal Robot[D]. Beijing: Beijing University of Science and Technology, 2006.
17	Xiaolan P, Lun X, Xin L, et al. Emotional State Transition Model Based on Stimulus and Personality Characteristics[J]. China Communications(S1637-5447), 2013, 10(6): 146-155.
18	Yi W, Zhi-Liang W, Wei W. Research on Associative Memory Models of Emotional Robots[J]. Advances in Mechanical Engineering(S1687-8132), 2014, 6(1): 208153.
19	Masuyama N, Loo C K, Seera M. Personality Affected Robotic Emotional Model with Associative Memory for Human-robot Interaction[J]. Neurocomputing(S0925-2312),
	2018, 272(10): 213-225.
20	Ibanez R V, Keysermann M U, Vargas P A. Emotional Memories in Autonomous Robots[C]// The 23rd IEEE International Symposium on Robot and Human Interactive Communication. Piscataway: IEEE, 2014: 405-410.
21	Tiedens L Z, Linton S. Judgment Under Emotional Certainty and Uncertainty: the Effects of Specific Emotions on Information Processing[J]. Journal of Personality and Social Psychology(S0022-3514), 2001, 81(6): 973-988.
22	Eich E, Metcalfe J. Mood Dependent Memory for Internal Versus External Events[J]. Journal of Experimental Psychology: Learning, Memory, and Cognition(S0278-7393), 1989, 15(3): 443-455.
23	Scherer K R. Towards a Prediction and Data Driven Computational Process Model of Emotion[J]. IEEE Transactions on Affective Computing(S1949-3045), 2019, 12(2): 279-292.
24	Frijda N H, Bower G H, Hamilton V. Cognitive Perspectives on Emotion and Motivation[M]. Abingdon: Taylor & Francis, 1988.
25	Jiang D, Wu K, Chen D, et al. A Probability and Integrated Learning Based Classification Algorithm for High-Level Human Emotion Recognition Problems[J]. Measurement(S0263-2241), 2020, 150(1): 107049.
26	Salway A, Graham M. Extracting Information about Emotions in Films[C]//The 11th ACM International Conference on Multimedia. New York: ACM, 2003: 299-302.
27	Ekman P, Dalgleish T, Power M. Handbook of Cognition and Emotion[M]. New Jersey: Basic Emotions Chapter, John Wiley & Sons, Ltd, 1999.
28	Chan C H, Jones G J F. Affect-based Indexing and Retrieval of Films[C]//The 13th Annual ACM International Conference on Multimedia. New York: ACM, 2005: 427-430.
29	Huang S J, Gao N, Chen S. Multi-Instance Multi-Label Active Learning[C]// IJCAI. San Francisco: Margan Kaufmann, 2017: 1886-1892.
30	Arifin S, Cheung P Y K. A Computation Method for Video Segmentation Utilizing the Pleasure-Arousal-Dominance Emotional Information[C]//The 15th ACM International Conference on Multimedia. New York: ACM, 2007: 68-77.
31	Arifin S, Cheung P Y K. Affective Level Video Segmentation by Utilizing the Pleasure-Arousal-Dominance Information[J]. IEEE Transactions on Multimedia(S1520-9210), 2008, 10(7): 1325-1341.
32	Canini L, Benini S, Migliorati P, et al. Emotional Identity of Movies[C]// The 16th IEEE International Conference on Image Processing (ICIP). Piscataway: IEEE, 2009: 1821-1824.
33	Lev G, Klein B, Wolf L. In Defense of Word Embedding for Generic Text Representation[C]//International Conference on Applications of Natural Language to Information Systems. Berlin: Springer, 2015: 35-50.
34	韩晶, 解仑, 刘欣, 等. 基于Gross认知重评的机器人认知情感交互模型[J]. 东南大学学报(自然科学版), 2015, 45(2): 270-274.
	Han Jing, Xie Lun, Liu Xin, et al. Robot Cognitive Emotional Interaction Model Based on Gross Cognitive Reevaluation[J]. Journal of Southeast University (Natural Science), 2015, 45(2): 270-274.
35	Boleda G, Gulordava K, Aina L. Putting Words in Context: LSTM Language Models and Lexical Ambiguity[C]// The 57th Annual Meeting of the Association for Computational Linguistics. Pennsylvania: ACL, 2019: 3342-3349.
36	Li H, Xu H. Video-based Sentiment Analysis with hvnLBP-TOP Feature and bi-LSTM[C]//The AAAI Conference on Artificial Intelligence. Palo Alto: AAAI, 2019: 9963-9964.
37	Busso C, Bulut M, Lee C C, et al. IEMOCAP: Interactive Emotional Dyadic Motion Capture Database[J]. Language Resources and Evaluation(S1574-020X), 2008, 42(4): 335-339.
38	Zadeh A A B, Liang P P, Poria S, et al. Multimodal Language Analysis in the Wild: Cmu-mosei Dataset and Interpretable Dynamic Fusion Graph[C]//The 56th Annual Meeting of the Association for Computational Linguistics. Pennsylvania: ACL, 2018: 2236-2246.
39	Masuyama N, Loo C K, Seera M. Personality Affected Robotic Emotional Model with Associative Memory for Human-Robot Interaction[J]. Neurocomputing(S0925-2312), 2018, 272(19): 213-225.

分类器	IEMOCAP			CMU-MOSEI
分类器	LSTM	Bi-LSTM	LMFDB-LSTM	LSTM	Bi-LSTM	LMFDB-LSTM
A	0.412 5	0.452 7	0.474 0	0.545 8	0.574 6	0.573 9
L	0.582 1	0.602 2	0.618 3	0.579 5	0. 5 861	0.601 4
V	0.321 5	0.339 7	0.325 2	0.535 7	0.526 2	0.558 1
AL	0.600 5	0.612 4	0.621 6	0.585 8	0.586 6	0.583 7
AV	0.402 2	0.425 3	0.428 9	0.570 2	0.575 4	0.576 6
TV	0.603 2	0.614 3	0.627 6	0.593 6	0.600 3	0.604 9
AVL	0.608 4	0.618 0	0.627 5	0.583 1	0.599 7	0.604 1

情绪	精确率	召回率	F1值
积极	0.887	0.812	0.846
消极	0.690	0.785	0.733
中性	0.793	0.720	0.759

情绪	精确率	召回率	F1值
积极	0.805	0.534	0.647
消极	0.655	0.913	0.752
中性	0.856	0.841	0.850

情绪	精确率	召回率	F1值
积极	0.905	0.872	0.887
消极	0.882	0.880	0.880
中性	0.867	0.900	0.882

情绪	精确率	召回率	F1值
积极	0.899	0.910	0.903
消极	0.966	0.873	0.914
中性	0.861	0.930	0.900