A Fast Federated Learning-based Crypto-aggregation Scheme and Its Simulation Analysis

doi:10.16182/j.issn1004731x.joss.24-FZ0817E

Abstract

Abstract:

To solve the problem of increased computation and communication costs caused by using homomorphic encryption (HE) to protect all gradients in traditional cryptographic aggregation (crypto-aggregation) schemes, a fast crypto-aggregation scheme called RandomCrypt was proposed. RandomCrypt performed clipping and quantization to fix the range of gradient values and then added two types of noise on the gradient for encryption and differential privacy (DP) protection. It conducted HE on noise keys to revise the precision loss caused by DP protection. RandomCrypt was implemented based on a FATE framework, and a hacking simulation experiment was conducted. The results show that the proposed scheme can effectively hinder inference attacks while ensuring training accuracy. It only requires 45%~51% communication cost and 5%~23% computation cost compared with traditional schemes.

Key words: federated learning, differential privacy, homomorphic encryption, inference attack, hacking simulation

CLC Number:

TP391

Lü Boshen, Song Xiao. A Fast Federated Learning-based Crypto-aggregation Scheme and Its Simulation Analysis[J]. Journal of System Simulation, 2024, 36(12): 2850-2870.

Figures/Tables 17

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Table 1

Fig. 8

Fig. 9

Fig. 10

Table 2

Fig. 11

Table 3

Fig. 12

Fig. 13

Table 4

References 37

1	Kairouz P, McMahan H B, Avent B, et al. Advances and Open Problems in Federated Learning[J]. Foundations and Trends^® in Machine Learning, 2021, 14(1/2): 1-210.
2	Geyer Robin C, Klein Tassilo, Nabi Moin. Differentially Private Federated Learning: A Client Level Perspective[EB/OL]. (2018-03-01) [2023-11-21]. .
3	Liu Yang, Kang Yan, Xing Chaoping, et al. A Secure Federated Transfer Learning Framework[J]. IEEE Intelligent Systems, 2020, 35(4): 70-82.
4	Truex S, Baracaldo N, Anwar A, et al. A Hybrid Approach to Privacy-preserving Federated Learning[C]//Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security. New York: ACM, 2019: 1-11.
5	Zhang Chengliang, Li Suyi, Xia Junzhe, et al. BatchCrypt: Efficient Homomorphic Encryption for Cross-Silo Federated Learning[C]//2020 USENIX Annual Technical Conference (USENIX ATC 20). Berkeley: USENIX Association, 2020: 493-506.
6	McMahanB, Moore E, Ramage D, et al. Communication-Efficient Learning of Deep Networks from Decentralized Data[C]//Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. Chia Laguna Resort: PMLR, 2017: 1273-1282.
7	Li Li, Fan Yuxi, Tse M, et al. A Review of Applications in Federated Learning[J]. Computers & Industrial Engineering, 2020, 149: 106854.
8	Wang Zhibo, Song Mengkai, Zhang Zhifei, et al. Beyond Inferring Class Representatives: User-Level Privacy Leakage from Federated Learning[C]//IEEE INFOCOM 2019 - IEEE Conference on Computer Communications. Piscataway: IEEE, 2019: 2512-2520.
9	Liu Pengrui, Xu Xiangrui, Wang Wei. Threats, Attacks and Defenses to Federated Learning: Issues, Taxonomy and Perspectives[J]. Cybersecurity, 2022, 5(1): 4.
10	Banner R, Nahshan Y, Soudry Daniel. Post Training 4-Bit Quantization of Convolutional Networks for Rapid-Deployment[C]//Proceedings of the 33rd International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2019: 7950-7958.
11	Banner R, Hubara Itay, Hoffer Elad, et al. Scalable Methods for 8-Bit Training of Neural Networks[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2018: 5151-5159.
12	Dettmers T, Lewis M, Belkada Y, et al. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale[C]//Advances in Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2022: 30318-30332.
13	Tramèr Florian, Terzis A, Steinke T, et al. Debugging Differential Privacy: A Case Study for Privacy Auditing[EB/OL]. (2022-03-28) [2023-11-21]. .
14	Anderson A G, Berg C P. The High-Dimensional Geometry of Binary Neural Networks[EB/OL]. (2017-05-19) [2023-11-25]. .
15	Baskin Chaim, Liss Natan, Schwartz Eli, et al. UNIQ: Uniform Noise Injection for Non-uniform Quantization of Neural Networks[J]. ACM Transactions on Computer Systems, 2021, 37(1/4): 4.
16	Volos C K, Kyprianidis I M, Stouboulos I N. Image Encryption Process Based on Chaotic Synchronization Phenomena[J]. Signal Processing, 2013, 93(5): 1328-1340.
17	Verma Raksha, Kumari Anjali, Anand Adarsh, et al. Revisiting Shift Cipher Technique for Amplified Data Security[J]. Journal of Computational and Cognitive Engineering, 2024, 3(1): 8-14.
18	Pathak M A, Rane S, Raj B. Multiparty Differential Privacy Via Aggregation of Locally Trained Classifiers[C]//Proceedings of the 23rd International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2010: 1876-1884.
19	Li Yong, Song Xiao, Tu Yuchun, et al. GAPBAS: Genetic Algorithm-based Privacy Budget Allocation Strategy in Differential Privacy K-means Clustering Algorithm[J]. Computers & Security, 2024, 139: 103697.
20	Lingjuan Lü, Yu Han, Ma Xingjun, et al. Privacy and Robustness in Federated Learning: Attacks and Defenses[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35(7): 8726-8746.
21	Fang Chen, Guo Yuanbo, Hu Yongjin, et al. Privacy-Preserving and Communication-efficient Federated Learning in Internet of Things[J]. computers & Security, 2021, 103: 102199.
22	Melis L, Song Congzheng, Emiliano De Cristofaro E, et al. Exploiting Unintended Feature Leakage in Collaborative Learning[C]//2019 IEEE Symposium on Security and Privacy (SP). Piscataway: IEEE, 2019: 691-706.
23	Zhao Bo, Mopuri K R, Bilen H. iDLG: Improved Deep Leakage from Gradients[EB/OL]. (2020-01-08) [2023-07-21]. .
24	Geiping Jonas, Bauermeister Hartmut, Dröge Hannah, et al. Inverting Gradients-How Easy is It to Break Privacy in Federated Learning?[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2020: 16937-16947.
25	Mahendran A, Vedaldi A. Visualizing Deep Convolutional Neural Networks Using Natural Pre-Images[J]. International Journal of Computer Vision, 2016, 120(3): 233-255.
26	Xiao Han, Rasul Kashif, Vollgraf Roland. Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms[EB/OL]. (2017-09-15) [2023-11-19]. .
27	Krizhevsky A, Sutskever I, Hinton G E. ImageNet Classification with Deep Convolutional Neural Networks[J]. Communications of the ACM, 2017, 60(6): 84-90.
28	Krizhevsky Alex. Learning Multiple Layers of Features from Tiny Images[EB/OL]. (2009-04-08) [2023-11-25]. .
29	Hochreiter Sepp, Schmidhuber Jürgen. Long Short-term Memory[J]. Neural Computation, 1997, 9(8): 1735-1780.
30	Stefanshipinkoski. Text_Generation-RNN[EB/OL]. [2023-10-26]. .
31	So J, Güler Başak, Avestimehr A S. Turbo-aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning[J]. IEEE Journal on Selected Areas in Information Theory, 2021, 2(1): 479-489.
32	Chen Jingxue, Yan Hang, Liu Zhiyuan, et al. When Federated Learning Meets Privacy-preserving Computation[J]. ACM Computing Surveys, 2024, 56(12): 319.
33	Bell J H, Bonawitz K A, Gascón Adrià, et al. Secure Single-server Aggregation with (Poly)Logarithmic Overhead[C]//Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security. New York: ACM, 2020: 1253-1269.
34	Kadhe S, Rajaraman N, Koyluoglu O O, et al. FastSecAgg: Scalable Secure Aggregation for Privacy-Preserving Federated Learning[EB/OL]. (2020-09-23) [2024-09-25]. .
35	Choi Beongjun, Sohn Jy-yong, Han Dongjun, et al. Communication-computation Efficient Secure Aggregation for Federated Learning[EB/OL]. (2021-07-12) [2024-09-25]. .
36	Fereidooni Hossein, Marchal Samuel, Miettinen Markus, et al. SAFELearn: Secure Aggregation for Private Federated Learning[C]//2021 IEEE Security and Privacy Workshops (SPW). Piscataway: IEEE, 2021: 56-62.
37	Huang Yuxian, Yang Geng, Zhou Hao, et al. VPPFL: A Verifiable Privacy-preserving Federated Learning Scheme Against Poisoning Attacks[J]. Computers & Security, 2024, 136: 103562.

Network	Weights	Dataset	Task	Batch	Hack Batch
3-layer-FC	101.77K	FMNIST	Image class	64	1
AlexNet	1.25M	CIFAR10	Image class	128	1
LSTM	4.02M	Shakespeare	Text gen	–32, 100	(1,1)

metrics	3-layer-FC	ALEX	LSTM
Batch size	64	128	(32,100)
Total epoch	30	300	10
Iteration per epoch	94	40	269
Evaluation step	10	60	13
Total round	282	200	207

Network	Scheme	Epoch	Round	t/h	Traffic/GB	Acc/%
3-layer-FC	Batch	10.10	95.0	0.711	1.197 4	88.1
	Random	11.66	107.0	0.733	0.627 2	88.1
	Plain	11.86	112.0	0.290	0.814 5	88.1
ALEX	Batch	279.4	186.3	83.120	154.190	70.2
	Random	282.0	188.0	18.480	70.720	70.1
	Plain	287.4	191.6	3.832	99.916	69.8
LSTM	Batch	7.732	160.0	47.950	90.809	97.7
	Random	8.360	173.0	3.926	50.888	97.7
	Plain	8.408	174.0	0.878	65.319	97.7

Approach	Computation	Communication
Turbo-Agg	O((n+1)mn log² log n)	O(mn² )
VerifyNet	O(4(2mn²+n³ ))	O(4mn+ 4n² )
Poly-Agg	O(3(n+1)(mn log n+n log²n))	O(3mn+3n log n)
FastSecAgg	O(3(n+1)m log n)	O(3mn+3n²)
CCESA	O(3mn log n)	O(3n(n log n)^0.5+3mn)
SAFELearn	O(4mn)	O(2mn)
RandomCrypt	O(4.75mn+2n+m)	O(1.5mn)