Computation Offloading Strategy Based on Stackelberg Game and DRL

doi:10.16182/j.issn1004731x.joss.21-1118

Abstract

Abstract:

To achieve the optimal computation offloading strategy for two kinds of MEC users in 5G hybrid private network, Stackelberg game is used to build the model of the competition for MEC server resources of two kinds of users, andthe strategies of complete information game and partially incomplete information game are researched respectively. It is proved that there is only one Nash equilibrium solution in the complete information scenario. In the incomplete information scenario, the environment is modeled as POMDP, and a two-stage deep reinforcement learning(TSDRL) is proposed to obtain the optimal computation offloading strategy. Simulation results show the proposed algorithm having a total reduction of 20.81% time delay and 3.38 % energy consumption compared with the D-DRL algorithm and can effectively improve the user QoE(quality of experience).

Key words: 5G hybrid private network, computation offloading, Stackelberg game theory, Nash equilibrium, partially observable Markov decision process(POMDP)

CLC Number:

TP393.01

Xianwei Zhou, Qixu Gong, Songsen Yu. Computation Offloading Strategy Based on Stackelberg Game and DRL[J]. Journal of System Simulation, 2023, 35(2): 372-385.

Figures/Tables 8

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Table 1

Experimental parameter setting

参数	值	参数	值
计算总资源 $R$ / $M b p s$	$50$	专网用户2传输速率 $r l, 2 k$ ( $M b / s$ )	$1 / 6$
专网用户任务总量 $C l, i$ / $M b$	$10$	专网用户3传输速率 $r l, 3 k$ ( $M b / s$ )	$1 / 5.6$
公网用户任务总量 $C f, j$ / $M b$	$1$	公网用户1传输速率 $r f, 1 k$ ( $M b / s$ )	$1 / 8.4$
单位传输时间成本 $ρ$ /( $J / s$ )	$1$	公网用户2传输速率 $r f, 2 k$ ( $M b / s$ )	$1 / 8.3$
单位数据消耗功率成本 $ν$ /( $J / M b$ )	$0.1$	公网用户3传输速率 $r f, 3 k$ ( $M b / s$ )	$1 / 8.2$
专网用户本地计算资源 $F l, i k$ ( $M b / s$ )	$0.1$	公网用户4传输速率 $r f, 4 k$ ( $M b / s$ )	$1 / 8.1$
公网用户本地计算资源 $F f, j k$ ( $M b / s$ )	$0.01$	公网用户5传输速率 $r f, 5 k$ ( $M b / s$ )	$1 / 8$
专网用户1传输速率 $r l, 1 k$ ( $M b / s$ )	$1 / 6.4$

Table 1

Fig. 5

Fig. 6

Fig. 7

References 22

1	Strinati E C, Haustein T, Maman M, et al. Beyond 5G Private Networks: the 5G CONNI Perspective[C]//2020 IEEE Globecom Workshops(GC Wkshps. Taiwan, China: IEEE, 2020: 1-6.
2	Li X, Garcia-Saavedra A, Costa-Perez X, et al. 5Growth: an End-to-End Service Platform for Automated Deployment and Management of Vertical Services over 5G Networks[J]. IEEE Communications Magazine(S0163-6804), 2021, 59(3): 84-90.
3	Xu J, Chen L, Zhou P. Joint Service Caching and Task Offloading for Mobile Edge Computing in Dense Networks[C]//IEEE INFOCOM 2018-IEEE Conference on Computer Communications. Honolulu, HI, USA: IEEE, 2018: 207-215.
4	董思岐, 李海龙, 屈毓锛, 等. 移动边缘计算中的计算卸载策略研究综述[J]. 计算机科学, 2019, 46(11): 32-40.
	Dong Siqi, Li Hailong, Qu Yuben, et al. Survey of Research on Computation Unloading Strategy in Mobile Edge Computing[J]. Computer Science, 2019, 46(11): 32-40.
5	Walia J S, Hämmäinen H, Flinck H. Future Scenarios and Value Network Configurations for Industrial 5G[C]//2017 8th International Conference on the Network of the Future(NOF). London, United Kingdom: IEEE, 2017: 79-84.
6	Guo H Z, Zhang J, Liu J J, et al. Energy-Efficient Task Offloading and Transmit Power Allocation for Ultra-Dense Edge Computing[C]//2018 IEEE Global Communications Conference(GLOBECOM). United Arab Emirates: IEEE, 2018: 1-6.
7	Liu J, Mao Y, Zhang J, et al. Delay-Optimal Computation Task Scheduling for Mobile-Edge Computing Systems[C]//2016 IEEE International Symposium on Information Theory(ISIT). Barcelona, Spain: IEEE, 2016: 1451-1455.
8	Kamoun M, Labidi W, Sarkiss M. Joint Resource Allocation and Offloading Strategies in Cloud Enabled Cellular Networks[C]//2015 IEEE International Conference on Communications(ICC). United Kingdom: IEEE, 2015: 5529-5534.
9	You C, Huang K, Chae H, et al. Energy-Efficient Resource Allocation for Mobile-Edge Computation Offloading[J]. IEEE Transactions on Wireless Communications(S1536-1276), 2016, 16(3): 1397-1411.
10	Chen M H, Liang B, Dong M. Joint Offloading Decision and Resource Allocation for Multi-User Multi-Task Mobile Cloud[C]//2016 IEEE International Conference on Communications(ICC). Kuala Lumpur, Malaysia: IEEE, 2016: 1-6.
11	李长云, 黎建波, 徐曦, 等. 具有能量收集设备的移动边缘计算资源分配研究[J]. 系统仿真学报, 2022, 34(11): 2313-2322.
	Li Changyun, Li Jianbo, Xu Xi, et al. Research on Mobile Edge Computing Resource Allocation with Energy Harvesting Device[J]. Journal of System Simulation, 2022, 34(11): 2313-2322.
12	Tran T X, Pompili D. Joint Task Offloading and Resource Allocation for Multi-Server Mobile-Edge Computing Networks[J]. IEEE Transactions on Vehicular Technology(S0018-9545), 2018, 68(1): 856-868.
13	Cao H, Cai J. Distributed Multiuser Computation Offloading for Cloudlet-Based Mobile Cloud Computing: A Game-Theoretic Machine Learning Approach[J]. IEEE Transactions on Vehicular Technology(S0018-9545), 2017, 67(1): 752-764.
14	Zhan Y, Guo S, Li P, et al. A deep Reinforcement Learning Based Offloading Game in Edge Computing[J]. IEEE Transactions on Computers(S0018-9340), 2020, 69(6): 883-893.
15	Asheralieva A, Niyato D. Bayesian Reinforcement Learning and Bayesian Deep Learning for Blockchains with Mobile Edge Computing[J]. IEEE Transactions on Cognitive Communications and Networking(S2332-7731), 2020, 7(1): 319-335.
16	Li F, Yao H, Du J, et al. Stackelberg Game-Based Computation Offloading in Social and Cognitive Industrial Internet of Things[J]. IEEE Transactions on Industrial Informatics(S1551-3203), 2019, 16(8): 5444-5455.
17	Kotulski Z, Niewolski W, Nowak T W, et al. New Security Architecture of Access Control in 5G MEC[C]//International Symposium on Security in Computing and Communication. Singapore: Springer, 2020: 77-91.
18	吴学文, 廖婧贤. 云边协同系统中基于博弈论的资源分配与任务卸载方案[J]. 系统仿真学报, 2022, 34(7): 1468-1481.
	Wu Xuewen, Liao Jingxian. Game-based Resource Allocation and Task Offloading Scheme in Collaborative Cloud-Edge Computing System[J]. Journal of System Simulation, 2022, 34(7): 1468-1481.
19	Wan X, Sheng G, Li Y, et al. Reinforcement Learning Based Mobile Offloading for Cloud-Based Malware Detection[C]//GLOBECOM 2017-2017 IEEE Global Communications Conference. Singapore: IEEE, 2017: 1-6.
20	Bai Y, Chen L, Song L, et al. Risk-Aware Edge Computation Offloading Using Bayesian Stackelberg Game[J]. IEEE Transactions on Network and Service Management(S1932-4537), 2020, 17(2): 1000-1012.
21	Pang J S, Fukushima M. Quasi-Variational Inequalities, Generalized Nash Equilibria, and Multi-Leader-Follower Games[J]. Computational Management Science (S1619-697X), 2005, 2(1): 21-56.
22	He K, Sun J. Convolutional Neural Networks at Constrained Time Cost[C]//IEEE Conference on Computer Vision and Pattern Recognition. United States: IEEE, 2015: 5353-5360.

[1]	Ying Xu, Qinming Liu, Linsen Zhou. Research on Decision-Making of Closed-Loop Supply Chain for Dual-Channel Recovery Based on Game Theory [J]. Journal of System Simulation, 2022, 34(2): 396-408.
[2]	Changyun Li, Jianbo Li, Xi Xu, Tingli Li. Research on Mobile Edge Computing Resource Allocation with Energy Harvesting Device [J]. Journal of System Simulation, 2022, 34(11): 2313-2322.
[3]	You Yichen, Wang Yan, Ji Zhicheng. Research on Flexible Job-shop Dynamic Scheduling Based on Game Theory [J]. Journal of System Simulation, 2021, 33(11): 2579-2588.