Journal of System Simulation ›› 2024, Vol. 36 ›› Issue (5): 1130-1140.doi: 10.16182/j.issn1004731x.joss.22-1480

Previous Articles     Next Articles

Gradient-based Deep Reinforcement Learning Interpretation Methods

Wang Yuan1,2(), Xu Lin2, Gong Xiaoze3(), Zhang Yongliang4, Wang Yongli1   

  1. 1.School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
    2.Science and Technology on Information Systems Engineering Laboratory, Nanjing 210014, China
    3.PLA 63850 Troops, Baicheng 137001, China
    4.Command and Control Engineering College, Army Engineering University of PLA, Nanjing 210007, China
  • Received:2022-12-11 Revised:2023-03-21 Online:2024-05-15 Published:2024-05-21
  • Contact: Gong Xiaoze;


The learning process and working mechanism of deep reinforcement learning methods such as DQN are not transparent, and their decision basis and reliability cannot be perceived, which makes the decisions made by the model highly questionable and greatly limits the application scenarios of deep reinforcement learning. To explain the decision-making mechanism of intelligent agents, this paper proposes a gradient based saliency map generation algorithm SMGG. It uses the gradient information of feature maps generated by high-level convolutional layers to calculate the importance of different feature maps. With the known structure and internal parameters of the model, starting from the last layer of the model, the weight of different feature maps relative to the saliency map is generated by calculating the gradient of feature maps; it classifies the importance of features in both positive and negative directions, and uses weights with positive influence to weight the features captured in the feature map, forming a positive interpretation of the current decision; it uses weights that have a negative impact on other categories to weight the features captured in the feature map, forming a reverse interpretation of the current decision. The saliency map of the decision is generated by the two together, and the basis for the intelligent agent's decision-making behavior is obtained. The effectiveness of this method has been demonstrated through experiments.

Key words: DRL, saliency map, interpretability, agent, gradient

CLC Number: